diff --git a/.gitignore b/.gitignore index 225dd761a7b8..ded7f5cd729d 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,4 @@ *.iml *.orig *~ +patchprocess/ diff --git a/CHANGES.txt b/CHANGES.txt index 52d21202fe9c..68b6b2eacf66 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,7 +1,1948 @@ HBase Change Log -Release 0.93.0 - Unreleased - *DO NOT ADD ISSUES HERE ON COMMIT ANY MORE. WE'LL GENERATE THE LIST - FROM JIRA INSTEAD WHEN WE MAKE A RELEASE* +Release 0.94.28 - 01/04/2016 +Sub-task + + [HBASE-14748] - Update 0.94 apidocs and xref on website + +Bug + + [HBASE-12921] - Port HBASE-5356 'region_mover.rb can hang if table region it belongs to is deleted' to 0.94 + [HBASE-13454] - SecureClient#setupIOStreams should handle all Exceptions + [HBASE-13651] - Handle StoreFileScanner FileNotFoundException + [HBASE-14799] - Commons-collections object deserialization remote command execution vulnerability + [HBASE-14830] - Fix broken links in 0.94 generated docs + [HBASE-15054] - Allow 0.94 to compile with JDK8 + [HBASE-15059] - Allow 0.94 to compile against Hadoop 2.7.x + +Improvement + + [HBASE-13344] - Add enforcer rule that matches our JDK support statement + +Task + + [HBASE-14747] - Make it possible to build Javadoc and xref reports for 0.94 again + +Release 0.94.27 - 03/18/2015 +Sub-task + + [HBASE-12776] - SpliTransaction: Log number of files to be split + +Bug + + [HBASE-10528] - DefaultBalancer selects plans to move regions onto draining nodes + [HBASE-12792] - [backport] HBASE-5835: Catch and handle NotServingRegionException when close region attempt fails + [HBASE-12801] - Failed to truncate a table while maintaing binary region boundaries + [HBASE-12968] - [0.94]SecureServer should not ignore CallQueueSize + [HBASE-13039] - Add patchprocess/* to .gitignore to fix builds of branches + [HBASE-13131] - ReplicationAdmin leaks connections if there's an error in the constructor + [HBASE-13229] - Specify bash for local-regionservers.sh and local-master-backup.sh + +Improvement + + [HBASE-11195] - Potentially improve block locality during major compaction for old regions + [HBASE-12223] - MultiTableInputFormatBase.getSplits is too slow + [HBASE-12720] - Make InternalScan LimitedPrivate + +Task + + [HBASE-13020] - Add 'patchprocess/*' to RAT excludes on all branches + + +Release 0.94.26 - 12/16/2014 +Bug + + [HBASE-12279] - Generated thrift files were generated with the wrong parameters + [HBASE-12491] - TableMapReduceUtil.findContainingJar() NPE + [HBASE-12635] - Delete acl notify znode of table after the table is deleted + [HBASE-12657] - The Region is not being split and far exceeds the desired maximum size. + [HBASE-12692] - NPE from SnapshotManager#stop + +Release 0.94.25 - 11/7/2014 +Bug + + [HBASE-12039] - Lower log level for TableNotFoundException log message when throwing + [HBASE-12065] - Import tool is not restoring multiple DeleteFamily markers of a row + [HBASE-12146] - RegionServerTracker should escape data in log messages + [HBASE-12171] - Backport: PerformanceEvaluation: getSplits doesn't provide right splits. + [HBASE-12336] - RegionServer failed to shutdown for NodeFailoverWorker thread + [HBASE-12376] - HBaseAdmin leaks ZK connections if failure starting watchers (ConnectionLossException) + +Improvement + + [HBASE-12272] - Generate Thrift code through maven + +Task + + [HBASE-12235] - Backport to 0.94: HBASE-9002 TestDistributedLogSplitting.testRecoverdEdits should test correct region + [HBASE-12381] - Add maven enforcer rules for build assumptions + + +Release 0.94.24 - 09/29/2014 +Sub-task + + [HBASE-11923] - Potential race condition in RecoverableZookeeper.checkZk() + [HBASE-11963] - Synchronize peer cluster replication connection attempts + [HBASE-12023] - HRegion.applyFamilyMapToMemstore creates too many iterator objects. + [HBASE-12077] - FilterLists create many ArrayList$Itr objects per row. + +Bug + + [HBASE-11405] - Multiple invocations of hbck in parallel disables balancer permanently + [HBASE-11957] - Backport to 0.94 HBASE-5974 Scanner retry behavior with RPC timeout on next() seems incorrect + [HBASE-12019] - hbase-daemon.sh overwrite HBASE_ROOT_LOGGER and HBASE_SECURITY_LOGGER variables + [HBASE-12020] - String formatting on each RPC Invoke + [HBASE-12022] - Payloads on Failure attempt to serialize the byte[] into strings. + [HBASE-12114] - Meta table cache hashing may access the wrong table + +Improvement + + [HBASE-12090] - Bytes: more Unsafe, more Faster + +Task + + [HBASE-12103] - Backport HFileV1Detector to 0.94 + [HBASE-12113] - Backport to 0.94: HBASE-5525 Truncate and preserve region boundaries option + + +Release 0.94.23 - 08/26/2014 +Bug + + [HBASE-9746] - RegionServer can't start when replication tries to replicate to an unknown host + [HBASE-10834] - Better error messaging on issuing grant commands in non-authz mode + [HBASE-11232] - Add MultiRowMutation tests. + [HBASE-11536] - Puts of region location to Meta may be out of order which causes inconsistent of region location + [HBASE-11641] - TestDistributedLogSplitting.testMasterStartsUpWithLogSplittingWork fails frequently + [HBASE-11652] - Port HBASE-3270 and HBASE-11650 to 0.94 - create cluster id and version file in a tmp location and move it into place + [HBASE-11767] - [0.94] Unnecessary garbage produced by schema metrics during scanning + +Improvement + + [HBASE-11667] - Comment ClientScanner logic for NSREs. + [HBASE-11754] - [Shell] Record table property SPLITS_FILE in descriptor + +Task + + [HBASE-11690] - Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94 + [HBASE-11691] - Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94 + [HBASE-11693] - Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94 + + +Release 0.94.22 - 07/31/2014 +Bug + + [HBASE-10645] - Fix wrapping of Requests Counts Regionserver level metrics + [HBASE-11360] - SnapshotFileCache causes too many cache refreshes + [HBASE-11479] - SecureConnection can't be closed when SecureClient is stopping because InterruptedException won't be caught in SecureClient#setupIOstreams() + [HBASE-11496] - HBASE-9745 broke cygwin CLASSPATH translation + [HBASE-11552] - Read/Write requests count metric value is too short + [HBASE-11565] - Stale connection could stay for a while + [HBASE-11633] - [0.94] port HBASE-11217 Race between SplitLogManager task creation + TimeoutMonitor + +Improvement + + [HBASE-2217] - VM OPTS for shell only + [HBASE-7910] - Dont use reflection for security + [HBASE-11444] - Remove use of reflection for User#getShortName + [HBASE-11450] - Improve file size info in SnapshotInfo tool + [HBASE-11480] - ClientScanner might not close the HConnection created in construction + [HBASE-11623] - mutateRowsWithLocks might require updatesLock.readLock with waitTime=0 + + +Release 0.94.21 - 06/27/2014 +Bug + + [HBASE-10692] - The Multi TableMap job don't support the security HBase cluster + [HBASE-11052] - Sending random data crashes thrift service + [HBASE-11096] - stop method of Master and RegionServer coprocessor is not invoked + [HBASE-11234] - FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result + [HBASE-11341] - ZKProcedureCoordinatorRpcs should respond only to members + [HBASE-11414] - Backport to 0.94: HBASE-7711 rowlock release problem with thread interruptions in batchMutate + +Improvement + + [HBASE-8495] - Change ownership of the directory to bulk load + [HBASE-10871] - Indefinite OPEN/CLOSE wait on busy RegionServers + +New Feature + + [HBASE-10935] - support snapshot policy where flush memstore can be skipped to prevent production cluster freeze + + +Release 0.94.20 - 05/23/2014 +Sub-task + + [HBASE-10936] - Add zeroByte encoding test + +Bug + + [HBASE-10958] - [dataloss] Bulk loading with seqids can prevent some log entries from being replayed + [HBASE-11110] - Ability to load FilterList class is dependent on context classloader + [HBASE-11143] - Improve replication metrics + [HBASE-11188] - "Inconsistent configuration" for SchemaMetrics is always shown + [HBASE-11212] - Fix increment index in KeyValueSortReducer + [HBASE-11225] - Backport fix for HBASE-10417 'index is not incremented in PutSortReducer#reduce()' + [HBASE-11247] - [0.94] update maven-site-plugin to 3.3 + +Improvement + + [HBASE-11008] - Align bulk load, flush, and compact to require Action.CREATE + [HBASE-11119] - Update ExportSnapShot to optionally not use a tmp file on external file system + [HBASE-11128] - Add -target option to ExportSnapshot to export with a different name + [HBASE-11134] - Add a -list-snapshots option to SnapshotInfo + + +Release 0.94.19 - 04/21/2014 +Bug + + [HBASE-10118] - Major compact keeps deletes with future timestamps + [HBASE-10312] - Flooding the cluster with administrative actions leads to collapse + [HBASE-10533] - commands.rb is giving wrong error messages on exceptions + [HBASE-10766] - SnapshotCleaner allows to delete referenced files + [HBASE-10805] - Speed up KeyValueHeap.next() a bit + [HBASE-10807] - -ROOT- still stale in table.jsp if it moved + [HBASE-10845] - Memstore snapshot size isn't updated in DefaultMemStore#rollback() + [HBASE-10847] - 0.94: drop non-secure builds, make security the default + [HBASE-10848] - Filter SingleColumnValueFilter combined with NullComparator does not work + [HBASE-10966] - RowCounter misinterprets column names that have colons in their qualifier + [HBASE-10991] - Port HBASE-10639 'Unload script displays wrong counts (off by one) when unloading regions' to 0.94 + [HBASE-11003] - ExportSnapshot is using the wrong fs when staging dir is not in fs.defaultFS + [HBASE-11030] - HBaseTestingUtility.getMiniHBaseCluster should be able to return null + +Task + + [HBASE-10921] - Port HBASE-10323 'Auto detect data block encoding in HFileOutputFormat' to 0.94 / 0.96 + +Test + + [HBASE-10782] - Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf + [HBASE-10969] - TestDistributedLogSplitting fails frequently in 0.94. + [HBASE-10982] - TestZKProcedure.testMultiCohortWithMemberTimeoutDuringPrepare fails frequently in 0.94 + [HBASE-10987] - Increase timeout in TestZKLeaderManager.testLeaderSelection + [HBASE-10988] - Properly wait for server in TestThriftServerCmdLine + [HBASE-10989] - TestAccessController needs better timeout + [HBASE-10996] - TestTableSnapshotInputFormatScan fails frequently on 0.94 + [HBASE-11010] - TestChangingEncoding is unnecessarily slow + [HBASE-11017] - TestHRegionBusyWait.testWritesWhileScanning fails frequently in 0.94 + [HBASE-11022] - Increase timeout for TestHBaseFsck.testSplitDaughtersNotInMeta + [HBASE-11024] - TestSecureLoadIncrementalHFilesSplitRecovery should wait longer for ACL table + [HBASE-11029] - Increase wait in TestSplitTransactionOnCluster.split + [HBASE-11037] - Race condition in TestZKBasedOpenCloseRegion + [HBASE-11040] - TestAccessController, TestAccessControllerFilter, and TestTablePermissions need to wait longer to ACL table + [HBASE-11042] - TestForceCacheImportantBlocks OOMs occasionally in 0.94 + + +Release 0.94.18 - 03/14/2014 +Bug + + [HBASE-9708] - Improve Snapshot Name Error Message + [HBASE-9778] - Add hint to ExplicitColumnTracker to avoid seeking + [HBASE-10514] - Forward port HBASE-10466, possible data loss when failed flushes + [HBASE-10549] - When there is a hole, LoadIncrementalHFiles will hang in an infinite loop. + [HBASE-10575] - ReplicationSource thread can't be terminated if it runs into the loop to contact peer's zk ensemble and fails continuously + [HBASE-10583] - backport HBASE-8402 to 0.94 - ScanMetrics depends on number of rpc calls to the server. + [HBASE-10594] - Speed up TestRestoreSnapshotFromClient + [HBASE-10598] - Written data can not be read out because MemStore#timeRangeTracker might be updated concurrently + [HBASE-10614] - Master could not be stopped + [HBASE-10622] - Improve log and Exceptions in Export Snapshot + [HBASE-10624] - Fix 2 new findbugs warnings introduced by HBASE-10598 + [HBASE-10627] - A logic mistake in HRegionServer isHealthy + [HBASE-10631] - Avoid extra seek on FileLink open + [HBASE-10642] - Add M/R over snapshots to 0.94 + [HBASE-10669] - [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option + [HBASE-10682] - region_mover.rb throws "can't convert nil into String" for regions moved + [HBASE-10712] - Backport HBASE-8304 to 0.94 and 0.96 + [HBASE-10716] - [Configuration]: hbase.regionserver.region.split.policy should be part of hbase-default.xml + [HBASE-10718] - TestHLogSplit fails when it sets a KV size to be negative + [HBASE-10722] - [0.94] HRegion.computeHDFSBlocksDistribution does not account for links and reference files. + [HBASE-10731] - Fix environment variables typos in scripts + [HBASE-10738] - AssignmentManager should shut down executors on stop + [HBASE-10745] - Access ShutdownHook#fsShutdownHooks should be synchronized + [HBASE-10751] - TestHRegion testWritesWhileScanning occasional fail since HBASE-10514 went in + +Improvement + + [HBASE-8604] - improve reporting of incorrect peer address in replication + +Test + + [HBASE-9914] - Port fix for HBASE-9836 'Intermittent TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking failure' to 0.94 + + +Release 0.94.17 - 02/18/2014 +Bug + + [HBASE-7963] - HBase VerifyReplication not working when security enabled + [HBASE-10249] - TestReplicationSyncUpTool fails because failover takes too long + [HBASE-10274] - MiniZookeeperCluster should close ZKDatabase when shutdown ZooKeeperServers + [HBASE-10319] - HLog should roll periodically to allow DN decommission to eventually complete. + [HBASE-10320] - Avoid ArrayList.iterator() ExplicitColumnTracker + [HBASE-10335] - AuthFailedException in zookeeper may block replication forever + [HBASE-10340] - [BACKPORT] HBASE-9892 Add info port to ServerName to support multi instances in a node + [HBASE-10363] - [0.94] TestInputSampler and TestInputSamplerTool fail under hadoop 2.0/23 profiles. + [HBASE-10371] - Compaction creates empty hfile, then selects this file for compaction and creates empty hfile and over again + [HBASE-10383] - Secure Bulk Load for 'completebulkload' fails for version 0.94.15 + [HBASE-10400] - [hbck] Continue if region dir missing on region merge attempt + [HBASE-10401] - [hbck] perform overlap group merges in parallel + [HBASE-10448] - ZKUtil create and watch methods don't set watch in some cases + [HBASE-10470] - Import generates huge log file while importing large amounts of data + [HBASE-10481] - API Compatibility JDiff script does not properly handle arguments in reverse order + [HBASE-10482] - ReplicationSyncUp doesn't clean up its ZK, needed for tests + [HBASE-10485] - PrefixFilter#filterKeyValue() should perform filtering on row key + [HBASE-10489] - TestImportExport fails in 0.94 with Hadoop2 + [HBASE-10493] - InclusiveStopFilter#filterKeyValue() should perform filtering on row key + [HBASE-10501] - Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions + [HBASE-10505] - Import.filterKv does not call Filter.filterRowKey + [HBASE-10506] - Fail-fast if client connection is lost before the real call be executed in RPC layer + [HBASE-10508] - Backport HBASE-10365 'HBaseFsck should clean up connection properly when repair is completed' to 0.94 and 0.96 + [HBASE-10539] - HRegion.addAndGetGlobalMemstoreSize returns previous size + [HBASE-10545] - RS Hangs waiting on region to close on shutdown; has to timeout before can go down + [HBASE-10546] - Two scanner objects are open for each hbase map task but only one scanner object is closed + [HBASE-10551] - Change local mode back to one RS thread by default + [HBASE-10552] - HFilePerformanceEvaluation.GaussianRandomReadBenchmark fails sometimes. + [HBASE-10555] - Backport HBASE-8519 to 0.94, Backup master will never come up if primary master dies during initialization + [HBASE-10562] - Fix TestMultiTableInputFormat for Hadoop 2 in 0.94 + +Improvement + + [HBASE-10212] - New rpc metric: number of active handler + [HBASE-10423] - Report back the message of split or rollback failure to the master + [HBASE-10457] - Print corrupted file information in SnapshotInfo tool without -file option + +Task + + [HBASE-10473] - Add utility for adorning http Context + +Test + + [HBASE-10480] - TestLogRollPeriod#testWithEdits may fail due to insufficient waiting + + +Release 0.94.16 - 01/10/2014 +Sub-task + + [HBASE-10257] - [0.94] Master aborts due to assignment race + +Bug + + [HBASE-7226] - HRegion.checkAndMutate uses incorrect comparison result for <, <=, > and >= + [HBASE-8558] - Add timeout limit for HBaseClient dataOutputStream + [HBASE-8912] - [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE + [HBASE-9346] - HBCK should provide an option to check if regions boundaries are the same in META and in stores. + [HBASE-10078] - Dynamic Filter - Not using DynamicClassLoader when using FilterList + [HBASE-10193] - Cleanup HRegion if one of the store fails to open at region initialization + [HBASE-10214] - Regionserver shutdown improperly and leaves the dir in .old not deleted + [HBASE-10215] - TableNotFoundException should be thrown after removing stale znode in ETH + [HBASE-10225] - Bug in calls to RegionObsever.postScannerFilterRow + [HBASE-10250] - [0.94] TestHLog fails occasionally + [HBASE-10268] - TestSplitLogWorker occasionally fails + [HBASE-10272] - Cluster becomes nonoperational if the node hosting the active Master AND ROOT/META table goes offline + [HBASE-10273] - AssignmentManager.regions and AssignmentManager.servers are not always updated in tandem + [HBASE-10279] - TestStore.testDeleteExpiredStoreFiles is flaky + [HBASE-10281] - TestMultiParallel.testFlushCommitsNoAbort fails frequently in 0.94 + [HBASE-10284] - Build broken with svn 1.8 + [HBASE-10286] - Revert HBASE-9593, breaks RS wildcard addresses + [HBASE-10306] - Backport HBASE-6820 to 0.94, MiniZookeeperCluster should ensure that ZKDatabase is closed upon shutdown() + +Improvement + + [HBASE-10285] - All for configurable policies in ChaosMonkey + +Test + + [HBASE-10259] - [0.94] Upgrade JUnit to 4.11 + + +Release 0.94.15 - 12/17/2013 +Bug + + [HBASE-7886] - [replication] hlog zk node will not be deleted if client roll hlog + [HBASE-9485] - TableOutputCommitter should implement recovery if we don't want jobs to start from 0 on RM restart + [HBASE-9995] - Not stopping ReplicationSink when using custom implementation for the ReplicationSink + [HBASE-10014] - HRegion#doMiniBatchMutation rollbacks the memstore even if there is nothing to rollback. + [HBASE-10015] - Replace intrinsic locking with explicit locks in StoreScanner + [HBASE-10026] - HBaseAdmin#createTable could fail if region splits too fast + [HBASE-10046] - Unmonitored HBase service could accumulate Status objects and OOM + [HBASE-10057] - TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to finish occasionally + [HBASE-10061] - TableMapReduceUtil.findOrCreateJar calls updateMap(null, ) resulting in thrown NPE + [HBASE-10064] - AggregateClient.validateParameters can throw NPE + [HBASE-10089] - Metrics intern table names cause eventual permgen OOM in 0.94 + [HBASE-10111] - Verify that a snapshot is not corrupted before restoring it + [HBASE-10112] - Hbase rest query params for maxVersions and maxValues are not parsed + [HBASE-10117] - Avoid synchronization in HRegionScannerImpl.isFilterDone + [HBASE-10120] - start-hbase.sh doesn't respect --config in non-distributed mode + [HBASE-10179] - HRegionServer underreports readRequestCounts by 1 under certain conditions + [HBASE-10181] - HBaseObjectWritable.readObject catches DoNotRetryIOException and wraps it back in a regular IOException + +Improvement + + [HBASE-9931] - Optional setBatch for CopyTable to copy large rows in batches + [HBASE-10001] - Add a coprocessor to help testing the performances without taking into account the i/o + [HBASE-10007] - PerformanceEvaluation: Add sampling and latency collection to randomRead test + [HBASE-10010] - eliminate the put latency spike on the new log file beginning + [HBASE-10048] - Add hlog number metric in regionserver + [HBASE-10049] - Small improvments in region_mover.rb + [HBASE-10093] - Unregister ReplicationSource metric bean when the replication source thread is terminated + +New Feature + + [HBASE-9047] - Tool to handle finishing replication when the cluster is offline + [HBASE-10119] - Allow HBase coprocessors to clean up when they fail + +Task + + [HBASE-9927] - ReplicationLogCleaner#stop() calls HConnectionManager#deleteConnection() unnecessarily + [HBASE-9986] - Incorporate HTTPS support for HBase (0.94 port) + +Test + + [HBASE-10058] - Test for HBASE-9915 (avoid reading index blocks) + [HBASE-10189] - Intermittent TestReplicationSyncUpTool failure + + +Release 0.94.14 - 11/18/2013 +Sub-task + + [HBASE-9165] - Improvements to addDependencyJars + +Bug + + [HBASE-9138] - getHaseIntegrationTestingUtility() is misspelled + [HBASE-9799] - Change Hadoop 1.2 dependency to 1.2.1 + [HBASE-9809] - RegionTooBusyException should provide region name which was too busy + [HBASE-9834] - Minimize byte[] copies for 'smart' clients + [HBASE-9849] - [REST] Forbidden schema delete in read only mode + [HBASE-9850] - Issues with UI for table compact/split operation completion. After split/compaction operation using UI, the page is not automatically redirecting back using IE8/Firefox. + [HBASE-9865] - Reused WALEdits in replication may cause RegionServers to go OOM + [HBASE-9872] - ModifyTable does not modify the attributes of a newly modified/changed ColumnDescriptor + [HBASE-9890] - MR jobs are not working if started by a delegated user + [HBASE-9902] - Region Server is starting normally even if clock skew is more than default 30 seconds(or any configured). -> Regionserver node time is greater than master node time + [HBASE-9906] - Restore snapshot fails to restore the meta edits sporadically + [HBASE-9915] - Performance: isSeeked() in EncodedScannerV2 always returns false + [HBASE-9952] - Snapshot restore may fail due to NullPointerException + [HBASE-9956] - Remove keyLength cache from KeyValue + [HBASE-9970] - HBase BulkLoad, table is creating with the timestamp key also as a column to the table. + [HBASE-9971] - Port part of HBASE-9958 to 0.94 - change lock scope in locateRegion + [HBASE-9975] - Not starting ReplicationSink when using custom implementation for the ReplicationSink. + [HBASE-9993] - 0.94: HBASE-9865 breaks coprocessor compatibility with WALEdit. + +Improvement + + [HBASE-4654] - [replication] Add a check to make sure we don't replicate to ourselves + [HBASE-8438] - Extend bin/hbase to print a "mapreduce classpath" + [HBASE-9715] - Backport -in_memory option support for LoadTestTool from trunk + [HBASE-9831] - 'hbasefsck.numthreads' property isn't passed to hbck via cmdline -D option + [HBASE-9894] - remove the inappropriate assert statement in Store.getSplitPoint() + [HBASE-9963] - Remove the ReentrantReadWriteLock in the MemStore + +Test + + [HBASE-8397] - improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) + [HBASE-8543] - fix coverage org.apache.hadoop.hbase.rest.client + [HBASE-8552] - fix coverage org.apache.hadoop.hbase.rest.filter + [HBASE-8556] - fix coverage org.apache.hadoop.hbase.metrics.histogram + [HBASE-8557] - fix coverage org.apache.hadoop.hbase.rest.metrics + [HBASE-8559] - increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor + + +Release 0.94.13 - 10/29/2013 +Sub-task + + [HBASE-9711] - Improve HBASE-9428 - avoid copying bytes for RegexFilter unless necessary + +Bug + + [HBASE-7600] - TestAdmin.testCreateBadTables is failing occasionally + [HBASE-8521] - Cells cannot be overwritten with bulk loaded HFiles + [HBASE-9430] - Memstore heapSize calculation - DEEP_OVERHEAD is incorrect + [HBASE-9504] - Backport HBASE-1212 to 0.94 + [HBASE-9548] - Cleanup SnapshotTestingUtils + [HBASE-9607] - Data loss after snapshot restore into cloned table + [HBASE-9649] - HFilePrettyPrinter should not throw a NPE if FirstKey or LastKey is null. + [HBASE-9651] - Backport HBASE-3890 'Scheduled tasks in distributed log splitting not in sync with ZK' to 0.94 + [HBASE-9727] - HBase Rest Server - DELETE scanner operation is a no-op + [HBASE-9731] - updatesBlockedSeconds RegionServer metric should not be a histogram + [HBASE-9732] - Static AtomicLong updated in StoreFileScanner every (re)seek + [HBASE-9737] - Corrupt HFile cause resource leak leading to Region Server OOM + [HBASE-9745] - Append HBASE_CLASSPATH to end of Java classpath and use another env var for prefix + [HBASE-9747] - PrefixFilter with OR condition gives wrong results + [HBASE-9751] - Excessive readpoints checks in StoreFileScanner + [HBASE-9753] - Excessive readpoint checks in MemstoreScanner + [HBASE-9783] - o.a.h.h.r.HRegion.mutateRow() with non-existent CF cause NPE + [HBASE-9789] - Change logging for Coprocessor exec call to trace + [HBASE-9807] - block encoder unnecessarily copies the key for each reseek + [HBASE-9842] - Backport HBASE-9593 and HBASE-8667 to 0.94 + [HBASE-9847] - HConnectionImplementation does not connect to new active master + +Improvement + + [HBASE-9221] - Provide interface for getting a User in the client + [HBASE-9488] - Improve performance for small scan + [HBASE-9716] - LoadTestTool should provide default min and max settings to the data generator + [HBASE-9749] - Custom threadpool for Coprocessor obtained HTables + +Task + + [HBASE-9819] - Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94 + +Test + + [HBASE-8553] - improve unit-test coverage of package org.apache.hadoop.hbase.mapreduce.hadoopbackport + [HBASE-9851] - TestHBaseFsck.testQuarantineMissingHFile is flaky + [HBASE-9852] - TestRpcMetrics.testCustomMetrics is flaky + + +Release 0.94.12 - 9/19/2013 +Sub-task + + [HBASE-9277] - REST should use listTableNames to list tables + [HBASE-9279] - Thrift should use listTableNames to list tables + +Bug + + [HBASE-7709] - Infinite loop possible in Master/Master replication + [HBASE-7954] - Fix the retrying logic of memstore flushes to avoid extra sleep + [HBASE-8760] - possible loss of data in snapshot taken after region split + [HBASE-8930] - Filter evaluates KVs outside requested columns + [HBASE-9167] - ServerCallable retries just once if timeout is not integer.max + [HBASE-9182] - Allow non-admin users to list all table names + [HBASE-9195] - Fix TestFSHDFSUtils against java7 test re-ordering + [HBASE-9207] - An Offline SplitParent Region can be assigned breaking split references + [HBASE-9231] - Multipage book is generated to the wrong location + [HBASE-9233] - isTableAvailable() may be stuck if an offline parent was never assigned + [HBASE-9252] - HConnectionManager#getZooKeeperWatcher() should be deprecated in 0.94 + [HBASE-9256] - HBaseClient#setupIOStreams should handle all exceptions + [HBASE-9266] - Javadoc: Document that HBaseAdmin.flush(...) is synchronous + [HBASE-9286] - [0.94] ageOfLastShippedOp replication metric doesn't update if the slave regionserver is stalled + [HBASE-9301] - Default hbase.dynamic.jars.dir to hbase.rootdir/jars + [HBASE-9303] - Snapshot restore of table which splits after snapshot was taken encounters 'Region is not online' + [HBASE-9326] - ServerName is created using getLocalSocketAddress, breaks binding to the wildcard address. Revert HBASE-8640 + [HBASE-9329] - SnapshotManager should check for directory existance before throwing a warning. + [HBASE-9344] - RegionServer not shutting down upon KeeperException in open region + [HBASE-9356] - [0.94] SecureServer.INSECURE_VERSIONS is declared incorrectly + [HBASE-9397] - Snapshots with the same name are allowed to proceed concurrently + [HBASE-9415] - In rpcServer, replicationQueue is initialized with the max queue size instead of the max queue lenght + [HBASE-9428] - Regex filters are at least an order of magnitude slower since 0.94.3 + [HBASE-9429] - Add back MetaScanner.allTableRegions(Configuration conf,byte[] tablename,boolean offlined) method + [HBASE-9432] - Backport HBASE-8781 to 0.94 + [HBASE-9448] - [0.94] Shell needs to fall back after HBASE-9182 if talking to older servers + [HBASE-9455] - Port HBASE-7113 'TestGzipFilter is flaky with jdk1.7' to 0.94 + [HBASE-9468] - Previous active master can still serves RPC request when it is trying recovering expired zk session + [HBASE-9482] - Do not enforce secure Hadoop for secure HBase + [HBASE-9506] - [0.94] Backport HBASE-9309 The links in the backup masters template are bad + [HBASE-9534] - Short-Circuit Coprocessor HTable access when on the same server + [HBASE-9566] - Add back WALEdit#get/setScopes method + [HBASE-9584] - Short-Circuit Coprocessor doesn't correctly lookup table when on server + +Improvement + + [HBASE-9243] - Add more useful statistics in the HFile tool + [HBASE-9314] - Dropping a table always prints a TableInfoMissingException in the master log + +Task + + [HBASE-9153] - Introduce/update a script to generate jdiff reports + [HBASE-9377] - Backport HBASE- 9208 "ReplicationLogCleaner slow at large scale" + +Test + + [HBASE-9287] - TestCatalogTracker depends on the execution order + + +Release 0.94.11 - 8/13/2013 +Sub-task + + [HBASE-8779] - Add mutateRow method support to Thrift2 + [HBASE-8946] - Add a new function to Thrift 2 to open scanner, get results and close scanner + [HBASE-8947] - Thrift 2 : Replace "bool writeToWAL" with "TDurability durability" + [HBASE-8948] - Bound table pool size in Thrift 2 server + +Bug + + [HBASE-6826] - [WINDOWS] TestFromClientSide failures + [HBASE-8067] - TestHFileArchiving.testArchiveOnTableDelete sometimes fails + [HBASE-8670] - [0.94] Backport HBASE-8449,HBASE-8204 and HBASE-8699 to 0.94 (Refactor recoverLease retries and pauses) + [HBASE-8698] - potential thread creation in MetaScanner.metaScan + [HBASE-8935] - IntegrationTestBigLinkedList fails under load on 0.94 due to some scan issues - add logging + [HBASE-8949] - hbase.mapreduce.hfileoutputformat.blocksize should configure with blocksize of a table + [HBASE-9026] - RestartRsHoldingRoot action in org.apache.hadoop.hbase.util.ChaosMonkey restarting the server holding .META. instead of -ROOT- + [HBASE-9032] - Result.getBytes() returns null if backed by KeyValue array + [HBASE-9048] - HCM throws NullPointerException under load + [HBASE-9050] - HBaseClient#call could hang + [HBASE-9060] - ExportSnapshot job fails if target path contains percentage character + [HBASE-9079] - FilterList getNextKeyHint skips rows that should be included in the results + [HBASE-9080] - Retain assignment should be used when re-enabling table(s) + [HBASE-9085] - Integration Tests fails because of bug in teardown phase where the cluster state is not being restored properly. + [HBASE-9087] - Handlers being blocked during reads + [HBASE-9097] - Set HBASE_CLASSPATH before rest of the classpath + [HBASE-9115] - HTableInterface.append operation may overwrites values + [HBASE-9120] - ClassFinder logs errors that are not + [HBASE-9146] - TestHTablePool hangs when run as part of runMediumTests profile + [HBASE-9158] - Serious bug in cyclic replication + [HBASE-9189] - IncreasingToUpperBoundRegionSplitPolicy.shouldSplit() should check all the stores before returning. + [HBASE-9200] - HFilePrettyPrinter finds incorrect largest row + +Improvement + + [HBASE-6580] - Deprecate HTablePool in favor of HConnection.getTable(...) + [HBASE-8995] - Add hadoop-1.2 profile + [HBASE-9019] - Port HBASE-8690: Reduce unnecessary getFileStatus hdfs calls in TTL hfile and hlog cleanners to 0.94 + [HBASE-9029] - Backport HBASE-8706 Some improvement in snapshot to 0.94 + [HBASE-9132] - Use table dir modtime to avoid scanning table dir to check cached table descriptor in 0.94 + [HBASE-9139] - Independent timeout configuration for rpc channel between cluster nodes + +New Feature + + [HBASE-7826] - Improve Hbase Thrift v1 to return results in sorted order + [HBASE-8663] - a HBase Shell command to list the tables replicated from current cluster + +Test + + [HBASE-8816] - Add support of loading multiple tables into LoadTestTool + [HBASE-9075] - [0.94] Backport HBASE-5760 Unit tests should write only under /target to 0.94 + [HBASE-9090] - cleanup snapshot tests setup/teardown code + [HBASE-9106] - Do not fail TestAcidGuarantees for exceptions on table flush + [HBASE-9107] - [0.94] Backport HBASE-6950 TestAcidGuarantees system test now flushes too aggressively to 0.94 + + +Release 0.94.10 - 7/19/2013 +Sub-task + + [HBASE-8774] - Add BatchSize and Filter to Thrift2 + [HBASE-8819] - Port HBASE-5428 to Thrift 2 + [HBASE-8826] - Ensure HBASE-8695 is covered in Thrift 2 + [HBASE-8832] - Ensure HBASE-4658 is supported by Thrift 2 + [HBASE-8876] - Addendum to HBASE-8774 Add BatchSize and Filter to Thrift2 - Add BatchSize Test + [HBASE-8938] - Thrift2 does not close scanner instance + +Bug + + [HBASE-8432] - a table with unbalanced regions will balance indefinitely with the 'org.apache.hadoop.hbase.master.DefaultLoadBalancer' + [HBASE-8678] - Wrongly delete cells in some case which can not be deleted + [HBASE-8695] - The HBase thrift service ignores XML configuration + [HBASE-8776] - tweak retry settings some more (on trunk and 0.94) + [HBASE-8782] - Thrift2 can not parse values when using framed transport + [HBASE-8794] - DependentColumnFilter.toString() throws NullPointerException + [HBASE-8811] - REST service ignores misspelled "check=" parameter, causing unexpected mutations + [HBASE-8814] - Possible NPE in split if a region has empty store files. + [HBASE-8858] - Backport hbase-6979 "recovered.edits file should not break distributed log splitting" + [HBASE-8910] - HMaster.abortNow shouldn't try to become a master again if it was stopped + [HBASE-8967] - Duplicate call to snapshotManager.stop() in HRegionServer + [HBASE-8968] - Avoid call to zk in HRegionServer#getMaster() to log the master address which already read from the zk. + [HBASE-8971] - Bump 0.94 version to 0.94.10-SNAPSHOT + [HBASE-8988] - Reuse the thread pool in the shell to not run out of threads + +Improvement + + [HBASE-8599] - HLogs in ZK are not cleaned up when replication lag is minimal + [HBASE-8767] - Backport hbase-8001 and hbase-8012, avoid lazy seek + [HBASE-8806] - Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows. + [HBASE-8809] - Include deletes in the scan (setRaw) method does not respect the time range or the filter + [HBASE-8847] - Filter.transform() always applies unconditionally, even when combined in a FilterList + [HBASE-8908] - Backport HBASE-8882 and HBASE-8904 (An Integration Test to Test MTTR) to 0.94 + [HBASE-8921] - [thrift2] Add GenericOptionsParser to Thrift 2 server + [HBASE-8945] - Backport to 0.94: HBASE-7952 Remove update() and Improve ExplicitColumnTracker performance. + +Task + + [HBASE-8829] - Improve failed TestMetaScanner assert message so can see where/why failure + +Test + + [HBASE-7770] - minor integration test framework fixes + [HBASE-8885] - Fix and reenable TestGet failing#testDynamicFilter + [HBASE-8914] - [0.94] TestRegionServerCoprocessorExceptionWithAbort is flaky + [HBASE-8928] - Make ChaosMonkey & LoadTest tools extensible, to allow addition of more actions and policies. + [HBASE-8934] - Fix bunch of flaky tests + [HBASE-8969] - Backport HBASE-8535+HBASE-8586 TestHCM#testDeleteForZKConnLeak enhancement to 0.94 + + +Release 0.94.9 - 6/24/2013 +Sub-task + + [HBASE-8453] - TestImportExport failing again due to configuration issues + +Bug + + [HBASE-8494] - TestRemoteAdmin#testClusterStatus should not assume 'requests' does not change + [HBASE-8522] - Archived hfiles and old hlogs may be deleted immediately by HFileCleaner, LogCleaner in HMaster + [HBASE-8555] - FilterList correctness may be affected by random ordering of sub-filter(list) + [HBASE-8590] - [0.94] BlockingMetaScannerVisitor should check for parent meta entry while waiting for split daughter + [HBASE-8639] - Poor performance of htable#getscanner in multithreaded environment due to DNS.getDefaultHost() being called in ScannerCallable#prepare() + [HBASE-8640] - ServerName in master may not initialize with the configured ipc address of hbase.master.ipc.address + [HBASE-8655] - Backport to 94 - HBASE-8346(Prefetching .META. rows in case only when useCache is set to true) + [HBASE-8656] - Rpc call may not be notified in SecureClient + [HBASE-8671] - Per-region WAL breaks CP backwards compatibility in 0.94 for non-enabled case + [HBASE-8684] - Table Coprocessor can't access external HTable by default + [HBASE-8700] - IntegrationTestBigLinkedList can fail due to random number collision + [HBASE-8724] - [0.94] ExportSnapshot should not use hbase.tmp.dir as a staging dir on hdfs + [HBASE-8742] - HTableDescriptor Properties not preserved when cloning + [HBASE-8743] - upgrade hadoop-23 version to 0.23.7 + [HBASE-8749] - Potential race condition between FSUtils.renameAndSetModifyTime() and HFile/LogCleaner + [HBASE-8762] - Performance/operational penalty when calling HTable.get with a list of one Get + [HBASE-8783] - RSSnapshotManager.ZKProcedureMemberRpcs may be initialized with the wrong server name + +Improvement + + [HBASE-5083] - Backup HMaster should have http infoport open with link to the active master + [HBASE-8609] - Make the CopyTable support startRow, stopRow options + [HBASE-8636] - Backport KeyValue Codec to 0.94 (HBASE-7413) + [HBASE-8683] - Add major compaction support in CompactionTool + [HBASE-8702] - Make WALEditCodec pluggable + +New Feature + + [HBASE-8504] - HTable.getRegionsInRange() should provide a non-cached API + +Task + + [HBASE-8603] - Backport HBASE-6921 to 0.94 + + +Release 0.94.8 - 5/22/2013 +Sub-task + + [HBASE-8381] - TestTableInputFormatScan on Hadoop 2 fails because YARN kills our applications + [HBASE-8399] - TestTableInputFormatScan2#testScanFromConfiguration fails on hadoop2 profile + +Bug + + [HBASE-7122] - Proper warning message when opening a log file with no entries (idle cluster) + [HBASE-7210] - Backport HBASE-6059 to 0.94 + [HBASE-7921] - TestHFileBlock.testGzipCompression should ignore the block checksum + [HBASE-8282] - User triggered flushes does not allow compaction to get triggered even if compaction criteria is met + [HBASE-8327] - Consolidate class loaders + [HBASE-8354] - Backport HBASE-7878 'recoverFileLease does not check return value of recoverLease' to 0.94 + [HBASE-8355] - BaseRegionObserver#pre(Compact|Flush|Store)ScannerOpen returns null + [HBASE-8377] - IntegrationTestBigLinkedList calculates wrap for linked list size incorrectly + [HBASE-8379] - bin/graceful_stop.sh does not return the balancer to original state + [HBASE-8385] - [SNAPSHOTS]: Restore fails to restore snapshot of a deleted table + [HBASE-8389] - HBASE-8354 forces Namenode into loop with lease recovery requests + [HBASE-8413] - Snapshot verify region will always fail if the HFile has been archived + [HBASE-8451] - MetricsMBeanBase has concurrency issues in init + [HBASE-8455] - Update ExportSnapshot to reflect changes in HBASE-7419 + [HBASE-8464] - FastDiffEncoder - valueOffset calculation is incorrect + [HBASE-8483] - HConnectionManager can leak ZooKeeper connections when using deleteStaleConnection + [HBASE-8493] - Backport HBASE-8422, 'Master won't go down', to 0.94 + [HBASE-8503] - Backport hbase-8483 "HConnectionManager can leak ZooKeeper connections when using deleteStaleConnection" to 0.94 + [HBASE-8505] - References to split daughters should not be deleted separately from parent META entry + [HBASE-8509] - ZKUtil#createWithParents won't set data during znode creation when parent folder doesn't exit + [HBASE-8513] - [0.94] Fix class files with CRLF endings + [HBASE-8516] - FSUtils.create() fail with ViewFS + [HBASE-8525] - Use sleep multilier when choosing sinks in ReplicationSource + [HBASE-8530] - Refine error message from ExportSnapshot when there is leftover snapshot in target cluster + [HBASE-8538] - HBaseAdmin#isTableEnabled() should check table existence before checking zk state. + [HBASE-8539] - Double(or tripple ...) ZooKeeper listeners of the same type when Master recovers from ZK SessionExpiredException + [HBASE-8540] - SnapshotFileCache logs too many times if snapshot dir doesn't exists + [HBASE-8547] - Fix java.lang.RuntimeException: Cached an already cached block + [HBASE-8550] - 0.94 ChaosMonkey grep for master is too broad + [HBASE-8563] - Double count of read requests for Gets + [HBASE-8588] - [Documentation]: Add information about adding REST and Thrift API kerberos principals to HBase ACL table + +Improvement + + [HBASE-5930] - Limits the amount of time an edit can live in the memstore. + [HBASE-6870] - HTable#coprocessorExec always scan the whole table + [HBASE-8345] - Add all available resources in o.a.h.h.rest.RootResource and VersionResource to o.a.h.h.rest.client.RemoteAdmin + [HBASE-8350] - enable ChaosMonkey to run commands as different users + [HBASE-8367] - LoadIncrementalHFiles does not return an error code nor throw Exception when failures occur due to timeouts + [HBASE-8383] - Support lib/*jar inside coprocessor jar + [HBASE-8405] - Add more custom options to how ClusterManager runs commands + [HBASE-8446] - Allow parallel snapshot of different tables + +New Feature + + [HBASE-7965] - Port table locking to 0.94 (HBASE-7305, HBASE-7546, HBASE-7933) + [HBASE-8415] - DisabledRegionSplitPolicy + +Task + + [HBASE-8574] - Add how to rename a table in the docbook + +Test + + [HBASE-8508] - improve unit-test coverage of package org.apache.hadoop.hbase.metrics.file + + +Release 0.94.7 - 4/24/2013 +Sub-task + + [HBASE-7615] - Add metrics for snapshots + [HBASE-7801] - Allow a deferred sync option per Mutation. + [HBASE-8210] - Backport the LoadTest portions of HBASE-7383 + [HBASE-8316] - JoinedHeap for non essential column families should reseek instead of seek + +Bug + + [HBASE-7401] - Remove warning message about running 'hbase migrate' + [HBASE-7658] - grant with an empty string as permission should throw an exception + [HBASE-7817] - Suggested JDWP debug options in hbase-env.sh are wrong + [HBASE-7824] - Improve master start up time when there is log splitting work + [HBASE-7925] - Back port HBASE-6881 into 0.94 + [HBASE-7961] - truncate on disabled table should throw TableNotEnabledException. + [HBASE-8014] - Backport HBASE-6915 to 0.94. + [HBASE-8030] - znode path of online region servers is hard coded in rolling_restart.sh + [HBASE-8044] - split/flush/compact/major_compact from hbase shell does not work for region key with \x format + [HBASE-8081] - Backport HBASE-7213 (separate hlog for meta tables) to 0.94 + [HBASE-8092] - bulk assignment in 0.94 doesn't handle ZK errors very well + [HBASE-8096] - [replication] NPE while replicating a log that is acquiring a new block from HDFS + [HBASE-8118] - TestTablePermission depends on the execution order + [HBASE-8125] - HBASE-7435 breaks BuiltInGzipDecompressor on Hadoop < 1.0.x + [HBASE-8127] - Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization + [HBASE-8128] - HTable#put improvements + [HBASE-8131] - Create table handler needs to handle failure cases. + [HBASE-8142] - Sporadic TestZKProcedureControllers failures on trunk + [HBASE-8146] - IntegrationTestBigLinkedList does not work on distributed setup + [HBASE-8150] - server should not produce RAITE for already-opening region in 0.94 (because master retry logic handles this case poorly) + [HBASE-8151] - Decode memstoreTS in HFileReaderV2 only when necessary + [HBASE-8158] - Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job" + [HBASE-8160] - HMaster#move doesn't check if master initialized + [HBASE-8166] - Avoid writing the memstoreTS into HFiles when possible + [HBASE-8169] - TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS may fail due to regions randomly assigned to a RS + [HBASE-8170] - HbaseAdmin.createTable cannot handle creating three regions + [HBASE-8176] - Backport HBASE-5335 "Dynamic Schema Configurations" to 0.94 + [HBASE-8179] - JSON formatting for cluster status is sort of broken + [HBASE-8188] - Avoid unnecessary row compare in StoreScanner + [HBASE-8192] - Logic errror causes infinite loop in HRegion.bulkLoadHFiles(List) + [HBASE-8207] - Replication could have data loss when machine name contains hyphen "-" + [HBASE-8208] - In some situations data is not replicated to slaves when deferredLogSync is enabled + [HBASE-8211] - Support for NN HA for 0.94 + [HBASE-8212] - Introduce a new separator instead of hyphen('-') for renaming recovered queues' znodes + [HBASE-8213] - global authorization may lose efficacy + [HBASE-8215] - Removing existing .regioninfo in writeRegioninfoOnFilesystem + [HBASE-8222] - User class should implement equals() and hashCode() + [HBASE-8225] - [replication] minor code bug when registering ReplicationLogCleaner + [HBASE-8226] - HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts "too many" regions + [HBASE-8229] - Replication code logs like crazy if a target table cannot be found. + [HBASE-8230] - Possible NPE on regionserver abort if replication service has not been started + [HBASE-8231] - delete tests in table_tests.rb(TestShell) always running on empty table. + [HBASE-8232] - TestAccessController occasionally fails with IndexOutOfBoundsException + [HBASE-8246] - Backport HBASE-6318 to 0.94 where SplitLogWorker exits due to ConcurrentModificationException + [HBASE-8259] - Snapshot backport in 0.94.6 breaks rolling restarts + [HBASE-8266] - Master cannot start if TableNotFoundException is thrown while partial table recovery + [HBASE-8270] - Backport HBASE-8097 'MetaServerShutdownHandler may potentially keep bumping up DeadServer.numProcessing' to 0.94 + [HBASE-8274] - Backport to 94: HBASE-7488 Implement HConnectionManager.locateRegions which is currently returning null + [HBASE-8276] - Backport hbase-6738 to 0.94 "Too aggressive task resubmission from the distributed log manager" + [HBASE-8285] - HBaseClient never recovers for single HTable.get() calls with no retries when regions move + [HBASE-8288] - HBaseFileSystem: Refactoring and correct semantics for createPath methods + [HBASE-8303] - Increse the test timeout to 60s when they are less than 20s + [HBASE-8313] - Add Bloom filter testing for HFileOutputFormat + [HBASE-8326] - mapreduce.TestTableInputFormatScan times out frequently + [HBASE-8352] - Rename '.snapshot' directory + [HBASE-8427] - Apache Rat is incorrectly excluding test source files + +Improvement + + [HBASE-7410] - [snapshots] add snapshot/clone/restore/export docs to ref guide + [HBASE-7599] - Port HBASE-6066 (low hanging read path improvements) to 0.94 + [HBASE-8148] - Allow IPC to bind on a specific address + [HBASE-8152] - Avoid creating empty reference file when splitkey is outside the key range of a store file + [HBASE-8174] - Backport HBASE-8161(setting blocking file count on table level doesn't work) to 0.94 + [HBASE-8198] - Backport HBASE-8063(Filter HFiles based on first/last key) into 0.94 + [HBASE-8199] - Eliminate exception for ExportSnapshot against the null table snapshot (with no data in) + [HBASE-8209] - Improve LoadTest extensibility + +New Feature + + [HBASE-1936] - ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services + [HBASE-7415] - [snapshots] Add task information to snapshot operation + +Task + + [HBASE-7929] - Reapply hbase-7507 "Make memstore flush be able to retry after exception" to 0.94 branch. + +Test + + [HBASE-8106] - Test to check replication log znodes move is done correctly + [HBASE-8260] - create generic integration test for trunk and 94 that is more deterministic, can be run for longer and is less aggressive + + +Release 0.94.6.1 - 4/13/2013 +Bug + + [HBASE-8259] - Snapshot backport in 0.94.6 breaks rolling restarts + + +Release 0.94.6 - 3/14/2013 +Sub-task + + [HBASE-7944] - Replication leaks file reader resource & not reset currentNbOperations + +Bug + + [HBASE-6132] - ColumnCountGetFilter & PageFilter not working with FilterList + [HBASE-6347] - -ROOT- and .META. are stale in table.jsp if they moved + [HBASE-6748] - Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback + [HBASE-7111] - hbase zkcli will not start if the zookeeper server chosen to connect to is unavailable + [HBASE-7153] - print gc option in hbase-env.sh affects hbase zkcli + [HBASE-7507] - Make memstore flush be able to retry after exception + [HBASE-7521] - fix HBASE-6060 (regions stuck in opening state) in 0.94 + [HBASE-7624] - Backport HBASE-5359 and HBASE-7596 to 0.94 + [HBASE-7671] - Flushing memstore again after last failure could cause data loss + [HBASE-7700] - TestColumnSeeking is mathematically bound to fail + [HBASE-7723] - Remove NameNode URI from ZK splitlogs + [HBASE-7725] - Add ability to create custom compaction request + [HBASE-7761] - MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true + [HBASE-7763] - Compactions not sorting based on size anymore. + [HBASE-7768] - zkcluster in local mode not seeing configurations in hbase-{site|default}.xml + [HBASE-7777] - HBCK check for lingering split parents should check for child regions + [HBASE-7813] - Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes + [HBASE-7814] - Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94 + [HBASE-7829] - zookeeper kerberos conf keytab and principal parameters interchanged + [HBASE-7832] - Use User.getShortName() in FSUtils + [HBASE-7833] - 0.94 does not compile with Hadoop-0.20.205 and 0.22.0 + [HBASE-7851] - Include the guava classes as a dependency for jobs using mapreduce.TableMapReduceUtil + [HBASE-7866] - TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK failed 3 times in a row + [HBASE-7867] - setPreallocSize is different with COMMENT in setupTestEnv in MiniZooKeeperCluster.java + [HBASE-7869] - Provide way to not start LogSyncer thread + [HBASE-7876] - Got exception when manually triggers a split on an empty region + [HBASE-7883] - Update memstore size when removing the entries in append operation + [HBASE-7884] - ByteBloomFilter's performance can be improved by avoiding multiplication when generating hash + [HBASE-7913] - Secure Rest server should login before getting an instance of Rest servlet + [HBASE-7914] - Port the fix of HBASE-6748 into 0.94 branch + [HBASE-7915] - Secure ThriftServer needs to login before calling HBaseHandler + [HBASE-7916] - HMaster uses wrong InetSocketAddress parameter to throw exception + [HBASE-7919] - Wrong key is used in ServerManager#getServerConnection() to retrieve from Map serverConnections + [HBASE-7920] - Move isFamilyEssential(byte[] name) out of Filter interface in 0.94 + [HBASE-7945] - Remove flaky TestCatalogTrackerOnCluster + [HBASE-7986] - [REST] Make HTablePool size configurable + [HBASE-7991] - Backport HBASE-6479 'HFileReaderV1 caching the same parent META block could cause server abort when splitting' to 0.94 + [HBASE-8007] - Adopt TestLoadAndVerify from BigTop + [HBASE-8019] - Port HBASE-7779 '[snapshot 130201 merge] Fix TestMultiParallel' to 0.94 + [HBASE-8025] - zkcli fails when SERVER_GC_OPTS is enabled + [HBASE-8040] - Race condition in AM after HBASE-7521 (only 0.94) + [HBASE-8055] - Null check missing in StoreFile.Reader.getMaxTimestamp() + [HBASE-8061] - Missing test from TestFlushSnapshotFromClient in 0.94 + [HBASE-8069] - TestHLog is dependent on the execution order + [HBASE-8085] - Backport the fix for Bytes.toStringBinary() into 94 (HBASE-6991) + [HBASE-8099] - ReplicationZookeeper.copyQueuesFromRSUsingMulti should not return any queues if it failed to execute. + [HBASE-8103] - Fix pom so 0.94 can generate site reports + +Improvement + + [HBASE-7818] - add region level metrics readReqeustCount and writeRequestCount + [HBASE-7827] - Improve the speed of Hbase Thirft Batch mutation for deletes + [HBASE-8031] - Adopt goraci as an Integration test + +New Feature + + [HBASE-4210] - Allow coprocessor to interact with batches per region sent from a client + [HBASE-7360] - Snapshot 0.94 Backport + +Task + + [HBASE-8088] - Versioning site: part one, put stake in the ground for 0.94 by copying current versions of book and site + [HBASE-8090] - Versioning site; part two, publish 0.94 site and add link from main site + + +Release 0.94.5 - 2/7/2013 +Sub-task + + [HBASE-2611] - Handle RS that fails while processing the failure of another one + [HBASE-7626] - Backport portions of HBASE-7460 to 0.94 + [HBASE-7687] - TestCatalogTracker.testServerNotRunningIOException fails occasionally + [HBASE-7738] - REST server should publish metrics that are available via HTTP + +Bug + + [HBASE-5458] - Thread safety issues with Compression.Algorithm.GZ and CompressionTest + [HBASE-6513] - Test errors when building on MacOS + [HBASE-6824] - Introduce ${hbase.local.dir} and save coprocessor jars there + [HBASE-7034] - Bad version, failed OPENING to OPENED but master thinks it is open anyways + [HBASE-7293] - [replication] Remove dead sinks from ReplicationSource.currentPeers and pick new ones + [HBASE-7423] - HFileArchiver should not use the configuration from the Filesystem + [HBASE-7468] - TestSplitTransactionOnCluster hangs frequently + [HBASE-7476] - HBase shell count command doesn't escape binary output + [HBASE-7497] - TestDistributedLogSplitting.testDelayedDeleteOnFailure times out occasionally + [HBASE-7498] - Make REST server thread pool size configurable + [HBASE-7499] - TestScannerTimeout timeout is too aggressive. + [HBASE-7502] - TestScannerTimeout fails on snapshot branch + [HBASE-7504] - -ROOT- may be offline forever after FullGC of RS + [HBASE-7505] - Server will hang when stopping cluster, caused by waiting for split threads + [HBASE-7506] - Judgment of carrying ROOT/META will become wrong when expiring server + [HBASE-7513] - HDFSBlocksDistribution shouldn't send NPEs when something goes wrong + [HBASE-7515] - Store.loadStoreFiles should close opened files if there's an exception + [HBASE-7524] - hbase-policy.xml is improperly set thus all rules in it can be by-passed + [HBASE-7530] - [replication] Work around HDFS-4380 else we get NPEs + [HBASE-7531] - [replication] NPE in SequenceFileLogReader because ReplicationSource doesn't nullify the reader + [HBASE-7534] - [replication] TestReplication.queueFailover can fail because HBaseTestingUtility.createMultiRegions is dangerous + [HBASE-7545] - [replication] Break out TestReplication into manageable classes + [HBASE-7549] - Make HTableInterface#batch() javadoc proper + [HBASE-7550] - Synchronization problem in AssignmentManager + [HBASE-7551] - nodeChildrenChange event may happen after the transition to RS_ZK_REGION_SPLITTING in SplitTransaction causing the SPLIT event to be missed in the master side. + [HBASE-7562] - ZKUtil: missing "else condition" in multi processing + [HBASE-7575] - FSUtils#getTableStoreFilePathMap should all ignore non-table folders + [HBASE-7578] - TestCatalogTracker hangs occasionally + [HBASE-7581] - TestAccessController depends on the execution order + [HBASE-7584] - Improve TestAccessController.testAppend + [HBASE-7587] - Fix two findbugs warning in RowResource + [HBASE-7592] - HConnectionManager.getHTableDescriptor() compares too much + [HBASE-7602] - TestFromClientSide.testPoolBehavior is incorrect + [HBASE-7617] - TestHRegionOnCluster.testDataCorrectnessReplayingRecoveredEdits still fails occasionally. + [HBASE-7628] - Port HBASE-6509 fast-forwarding FuzzyRowFilter to 0.94 + [HBASE-7643] - HFileArchiver.resolveAndArchive() race condition may lead to snapshot data loss + [HBASE-7644] - Port HBASE-4802 'Disable show table metrics in bulk loader' to 0.94 + [HBASE-7646] - Make forkedProcessTimeoutInSeconds configurable + [HBASE-7647] - 0.94 hfiles v2.1 are not backwards compatible with HFilev2.0 + [HBASE-7648] - TestAcidGuarantees.testMixedAtomicity hangs sometimes + [HBASE-7654] - Add List getCoprocessors() to HTableDescriptor + [HBASE-7669] - ROOT region wouldn't be handled by PRI-IPC-Handler + [HBASE-7681] - Address some recent random test failures + [HBASE-7684] - NullPointerException in SecureClient when Call is cleaned up due to RPC timeout + [HBASE-7685] - Closing socket connection can't be removed from SecureClient + [HBASE-7693] - Hostname returned by TableInputFormatBase.reverseDNS contains trailing period + [HBASE-7694] - Secure HBase should use replication call queue + [HBASE-7698] - race between RS shutdown thread and openregionhandler causes region to get stuck + [HBASE-7702] - Adding filtering to Import jobs + [HBASE-7715] - FSUtils#waitOnSafeMode can incorrectly loop on standby NN + [HBASE-7717] - Wait until regions are assigned in TestSplitTransactionOnCluster + [HBASE-7728] - deadlock occurs between hlog roller and hlog syncer + [HBASE-7729] - TestCatalogTrackerOnCluster.testbadOriginalRootLocation fails occasionally + [HBASE-7730] - HBaseAdmin#synchronousBalanceSwitch is not compatible with 0.92 + [HBASE-7731] - Append/Increment methods in HRegion don't check whether the table is readonly or not + [HBASE-7740] - Recheck matching row for joined scanners + [HBASE-7771] - Secure HBase Client in MR job causes tasks to wait forever + [HBASE-7772] - clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called + [HBASE-7776] - Use ErrorReporter/Log instead of System.out in hbck + [HBASE-7785] - rolling-restart.sh script unable to check expiration of master znode + [HBASE-7793] - Port HBASE-5564 Bulkload is discarding duplicate records to 0.94 + +Improvement + + [HBASE-3996] - Support multiple tables and scanners as input to the mapper in map/reduce jobs + [HBASE-5416] - Improve performance of scans with some kind of filters. + [HBASE-5498] - Secure Bulk Load + [HBASE-5664] - CP hooks in Scan flow for fast forward when filter filters out a row + [HBASE-7441] - Make ClusterManager in IntegrationTestingUtility pluggable + [HBASE-7540] - Make znode dump to print a dump of replication znodes + [HBASE-7561] - Display the total number of regions for a given table on the master webUI + [HBASE-7757] - Add web UI to REST server and Thrift server + +New Feature + + [HBASE-6669] - Add BigDecimalColumnInterpreter for doing aggregations using AggregationClient + [HBASE-7748] - Add DelimitedKeyPrefixRegionSplitPolicy + +Wish + + [HBASE-7705] - Make the method getCurrentPoolSize of HTablePool public + + +Release 0.94.4 - 1/2/2013 +Sub-task + + [HBASE-3776] - Add Bloom Filter Support to HFileOutputFormat + [HBASE-6206] - Large tests fail with jdk1.7 + [HBASE-7009] - Port HBaseCluster interface/tests to 0.94 + [HBASE-7042] - Master Coprocessor Endpoint + [HBASE-7282] - Backport Compaction Tool to 0.94 + [HBASE-7331] - Add access control for region open and close, row locking, and stopping the regionserver + [HBASE-7336] - HFileBlock.readAtOffset does not work well with multiple threads + [HBASE-7371] - Blocksize in TestHFileBlock is unintentionally small + [HBASE-7399] - Health check chore for HMaster + [HBASE-7406] - Example health checker script + [HBASE-7431] - TestSplitTransactionOnCluster tests still flaky + [HBASE-7438] - TestSplitTransactionOnCluster has too many infinite loops + +Bug + + [HBASE-6175] - TestFSUtils flaky on hdfs getFileStatus method + [HBASE-6317] - Master clean start up and Partially enabled tables make region assignment inconsistent. + [HBASE-6327] - HLog can be null when create table + [HBASE-6423] - Writes should not block reads on blocking updates to memstores + [HBASE-7091] - support custom GC options in hbase-env.sh + [HBASE-7158] - Allow CopyTable to identify the source cluster (for replication scenarios) + [HBASE-7165] - TestSplitLogManager.testUnassignedTimeout is flaky + [HBASE-7166] - TestSplitTransactionOnCluster tests are flaky + [HBASE-7172] - TestSplitLogManager.testVanishingTaskZNode() fails when run individually and is flaky + [HBASE-7177] - TestZooKeeperScanPolicyObserver.testScanPolicyObserver is flaky + [HBASE-7180] - RegionScannerImpl.next() is inefficient. + [HBASE-7205] - Coprocessor classloader is replicated for all regions in the HRegionServer + [HBASE-7214] - CleanerChore logs too much, so much so it obscures all else that is going on + [HBASE-7230] - port HBASE-7109 integration tests on cluster are not getting picked up from distribution to 0.94 + [HBASE-7235] - TestMasterObserver is flaky + [HBASE-7251] - Avoid flood logs during client disconnect during batch get operation + [HBASE-7252] - TestSizeBasedThrottler fails occasionally + [HBASE-7259] - Deadlock in HBaseClient when KeeperException occured + [HBASE-7260] - Upgrade hadoop 1 dependency to hadoop 1.1.1 + [HBASE-7273] - Upgrade zookeeper dependency to 3.4.5 for 0.94 + [HBASE-7279] - Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher + [HBASE-7300] - HbckTestingUtil needs to keep a static executor to lower the number of threads used + [HBASE-7301] - Force ipv4 for unit tests + [HBASE-7307] - MetaReader.tableExists should not return false if the specified table regions has been split + [HBASE-7338] - Fix flaky condition for org.apache.hadoop.hbase.TestRegionRebalancing.testRebalanceOnRegionServerNumberChange + [HBASE-7342] - Split operation without split key incorrectly finds the middle key in off-by-one error + [HBASE-7343] - Fix flaky condition for TestDrainingServer + [HBASE-7357] - HBaseClient and HBaseServer should use hbase.security.authentication when negotiating authentication + [HBASE-7376] - Acquiring readLock does not apply timeout in HRegion#flushcache + [HBASE-7398] - [0.94 UNIT TESTS] TestAssignmentManager fails frequently on CentOS 5 + [HBASE-7412] - Fix how HTableDescriptor handles default max file size and flush size + [HBASE-7417] - TestReplication is flaky + [HBASE-7421] - TestHFileCleaner->testHFileCleaning has an aggressive timeout + [HBASE-7422] - MasterFS doesn't set configuration for internal FileSystem + [HBASE-7432] - TestHBaseFsck prevents testsuite from finishing + [HBASE-7435] - BuiltInGzipDecompressor is only released during full GC + [HBASE-7440] - ReplicationZookeeper#addPeer is racy + [HBASE-7442] - HBase remote CopyTable not working when security enabled + [HBASE-7455] - Increase timeouts in TestReplication and TestSplitLogWorker + [HBASE-7464] - [REST] Sending HTML for errors is unhelpful + [HBASE-7466] - Fix junit dependency typo in 0.94 + [HBASE-7467] - CleanerChore checkAndDeleteDirectory not deleting empty directories + [HBASE-7483] - TestHRegionOnCluster and TestSplitTransactionOnCluster are racy with HBaseAdmin.move() + [HBASE-7485] - TestSplitLogManager is still flaky on windows + +Improvement + + [HBASE-4791] - Allow Secure Zookeeper JAAS configuration to be programmatically set (rather than only by reading JAAS configuration file) + [HBASE-5616] - Make compaction code standalone + [HBASE-5693] - When creating a region, the master initializes it and creates a memstore within the master server + [HBASE-5778] - Fix HLog compression's incompatibilities + [HBASE-5888] - Clover profile in build + [HBASE-6585] - Audit log messages should contain info about the higher level operation being executed + [HBASE-6775] - Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix + [HBASE-7190] - Add an option to hbck to check only meta and assignment + [HBASE-7197] - Add multi get to RemoteHTable + [HBASE-7199] - hbck should check lingering reference hfile and have option to sideline them automatically + [HBASE-7204] - Make hbck ErrorReporter pluggable + [HBASE-7231] - port HBASE-7200 create integration test for balancing regions and killing region servers to 0.94 + [HBASE-7249] - add test name filter to IntegrationTestsDriver + [HBASE-7328] - IntegrationTestRebalanceAndKillServersTargeted supercedes IntegrationTestRebalanceAndKillServers, remove + [HBASE-7351] - Periodic health check chore + [HBASE-7359] - [REST] 'accessToken' in RemoteHTable is vestigial + [HBASE-7374] - Expose master table operations for coprocessors by way of MasterServices + [HBASE-7377] - Clean up TestHBase7051 + [HBASE-7381] - Lightweight data transfer for Class Result + [HBASE-7469] - [REST] Share a HBaseAdmin instance + [HBASE-7472] - [REST] Support MIME type application/protobuf + +Task + + [HBASE-5258] - Move coprocessors set out of RegionLoad + [HBASE-7170] - [0.94 branch] Allow HConnectionImplementation to reconnect to master multiple times + [HBASE-7283] - Backport HBASE-6564 + HBASE-7202 to 0.94 + [HBASE-7341] - Deprecate RowLocks in 0.94 + + +Release 0.94.3 - 11/12/2012 +Sub-task + + [HBASE-4913] - Per-CF compaction Via the Shell + [HBASE-6305] - TestLocalHBaseCluster hangs with hadoop 2.0/0.23 builds. + [HBASE-6925] - Change socket write size from 8K to 64K for HBaseServer + [HBASE-6996] - HRegion.mutateRowsWithLocks should call checkResources/checkReadOnly + [HBASE-7076] - Add test that increment/append properly integrate with MVCC + [HBASE-7077] - Test for: CheckAndPut should properly read MVCC + [HBASE-7078] - Add a test that append is atomic + +Bug + + [HBASE-6389] - Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments + [HBASE-6583] - Enhance Hbase load test tool to automatically create column families if not present + [HBASE-6665] - ROOT region should not be splitted even with META row as explicit split key + [HBASE-6700] - [replication] empty znodes created during queue failovers aren't deleted + [HBASE-6728] - [89-fb] prevent OOM possibility due to per connection responseQueue being unbounded + [HBASE-6733] - [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-2] + [HBASE-6796] - Backport HBASE-5547, Don't delete HFiles in backup mode. + [HBASE-6843] - loading lzo error when using coprocessor + [HBASE-6846] - BitComparator bug - ArrayIndexOutOfBoundsException + [HBASE-6904] - In the HBase shell, an error is thrown that states replication-related znodes already exist + [HBASE-6958] - TestAssignmentManager sometimes fails + [HBASE-6974] - Metric for blocked updates + [HBASE-6978] - Minor typo in ReplicationSource SocketTimeoutException error handling + [HBASE-7017] - Backport "[replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file" to 0.94 + [HBASE-7018] - Fix and Improve TableDescriptor caching for bulk assignment + [HBASE-7021] - Default to Hadoop 1.0.4 in 0.94 and add Hadoop 1.1 profile + [HBASE-7037] - ReplicationPeer logs at WARN level aborting server instead of at FATAL + [HBASE-7048] - Regionsplitter requires the hadoop config path to be in hbase classpath + [HBASE-7051] - CheckAndPut should properly read MVCC + [HBASE-7060] - Region load balancing by table does not handle the case where a table's region count is lower than the number of the RS in the cluster + [HBASE-7069] - HTable.batch does not have to be synchronized + [HBASE-7086] - Enhance ResourceChecker to log stack trace for potentially hanging threads + [HBASE-7095] - Cannot set 'lenAsVal' for KeyOnlyFilter from shell + [HBASE-7103] - Need to fail split if SPLIT znode is deleted even before the split is completed. + [HBASE-7143] - TestMetaMigrationRemovingHTD fails when used with Hadoop 0.23/2.x + +Improvement + + [HBASE-5257] - Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter + [HBASE-5314] - Gracefully rolling restart region servers in rolling-restart.sh + [HBASE-5898] - Consider double-checked locking for block cache lock + [HBASE-6852] - SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields + [HBASE-6942] - Endpoint implementation for bulk deletion of data + [HBASE-6951] - Allow the master info server to be started in a read only mode. + [HBASE-7073] - OperationMetrics needs to cache the value of hbase.metrics.exposeOperationTimes + [HBASE-7089] - Allow filter to be specified for Get from HBase shell + [HBASE-7097] - Log message in SecureServer.class uses wrong class name + [HBASE-7151] - Better log message for Per-CF compactions + +Task + + [HBASE-6032] - Port HFileBlockIndex improvement from HBASE-5987 + [HBASE-7016] - port HBASE-6518 'Bytes.toBytesBinary() incorrect trailing backslash escape' to 0.94 + [HBASE-7020] - Backport HBASE-6336 Split point should not be equal to start row or end row + [HBASE-7038] - Port HBASE-5970 Improve the AssignmentManager#updateTimer and speed up handling opened event to 0.94 + [HBASE-7040] - Port HBASE-5867 Improve Compaction Throttle Default to 0.94 + [HBASE-7053] - port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 + [HBASE-7087] - Add to NOTICE.txt a note on jamon being MPL + +Test + + [HBASE-5984] - TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0 + [HBASE-7142] - TestSplitLogManager#testDeadWorker may fail because of hard limit on the TimeoutMonitor's timeout period + + +Release 0.94.2 - 10/08/2012 +Sub-task + + [HBASE-6257] - Avoid unnecessary flush & compact on Meta in admin.rb. + [HBASE-6496] - Example ZK based scan policy + [HBASE-6792] - Remove interface audience annotations in 0.94/0.92 introduced by HBASE-6516 + +Bug + + [HBASE-4565] - Maven HBase build broken on cygwin with copynativelib.sh call. + [HBASE-5292] - getsize per-CF metric incorrectly counts compaction related reads as well + [HBASE-5549] - Master can fail if ZooKeeper session expires + [HBASE-5997] - Fix concerns raised in HBASE-5922 related to HalfStoreFileReader + [HBASE-6165] - Replication can overrun .META. scans on cluster re-start + [HBASE-6211] - Put latencies in jmx + [HBASE-6263] - Use default mode for HBase Thrift gateway if not specified + [HBASE-6268] - Can't enable a table on a 0.94 cluster from a 0.92 client + [HBASE-6299] - RS starting region open while failing ack to HMaster.sendRegionOpen() causes inconsistency in HMaster's region state and a series of successive problems + [HBASE-6321] - ReplicationSource dies reading the peer's id + [HBASE-6340] - HBase RPC should allow protocol extension with common interfaces. + [HBASE-6359] - KeyValue may return incorrect values after readFields() + [HBASE-6364] - Powering down the server host holding the .META. table causes HBase Client to take excessively long to recover and connect to reassigned .META. table + [HBASE-6378] - the javadoc of setEnabledTable maybe not describe accurately + [HBASE-6432] - HRegionServer doesn't properly set clusterId in conf + [HBASE-6437] - Avoid admin.balance during master initialize + [HBASE-6438] - RegionAlreadyInTransitionException needs to give more info to avoid assignment inconsistencies + [HBASE-6447] - Common TestZooKeeper failures on jenkins: testMasterSessionExpired and testCreateSilentIsReallySilent + [HBASE-6450] - HBase startup should be with MALLOC_MAX_ARENA set + [HBASE-6460] - hbck "-repairHoles" usage inconsistent with "-fixHdfsOrphans" + [HBASE-6471] - Performance regression caused by HBASE-4054 + [HBASE-6478] - TestClassLoading.testClassLoadingFromLibDirInJar occasionally fails + [HBASE-6488] - HBase wont run on IPv6 on OSes that use zone-indexes + [HBASE-6503] - HBase Shell Documentation For DROP Is Outdated + [HBASE-6504] - Adding GC details prevents HBase from starting in non-distributed mode + [HBASE-6512] - Incorrect OfflineMetaRepair log class name + [HBASE-6514] - unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram + [HBASE-6516] - hbck cannot detect any IOException while ".tableinfo" file is missing + [HBASE-6520] - MSLab May cause the Bytes.toLong not work correctly for increment + [HBASE-6525] - bin/replication/copy_tables_desc.rb references non-existent class + [HBASE-6529] - With HFile v2, the region server will always perform an extra copy of source files + [HBASE-6537] - Race between balancer and disable table can lead to inconsistent cluster + [HBASE-6552] - TestAcidGuarantees system test should flush more aggressively + [HBASE-6561] - Gets/Puts with many columns send the RegionServer into an "endless" loop + [HBASE-6565] - Coprocessor exec result Map is not thread safe + [HBASE-6576] - HBaseAdmin.createTable should wait until the table is enabled + [HBASE-6579] - Unnecessary KV order check in StoreScanner + [HBASE-6587] - Region would be assigned twice in the case of all RS offline + [HBASE-6596] - Revert HBASE-5022; it undoes HBC.create + [HBASE-6602] - Region Server Dynamic Metrics can cause high cpu usage. + [HBASE-6603] - RegionMetricsStorage.incrNumericMetric is called too often + [HBASE-6608] - Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's + [HBASE-6615] - hbase.rs.evictblocksonclose seems to be ineffective + [HBASE-6616] - test failure in TestDelayedRpc#testTooManyDelayedRpcs + [HBASE-6621] - Reduce calls to Bytes.toInt + [HBASE-6623] - [replication] replication metrics value AgeOfLastShippedOp is not set correctly + [HBASE-6631] - TestHMasterRPCException in 0.92 failed twice on socket timeout + [HBASE-6632] - [0.92 UNIT TESTS] testCreateTableRPCTimeOut sets rpc timeout to 1500ms and leaves it (testHundredsOfTable fails w/ 1500ms timeout) + [HBASE-6638] - Move DaemonThreadFactory into Threads (0.94) + [HBASE-6641] - more message with DoNotRetryIOException in client + [HBASE-6647] - [performance regression] appendNoSync/HBASE-4528 doesn't take deferred log flush into account + [HBASE-6648] - [0.92 UNIT TESTS] TestMasterObserver.testRegionTransitionOperations fails occasionally + [HBASE-6649] - [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1] + [HBASE-6662] - Region server incorrectly reports its own address as master's address + [HBASE-6663] - NPE race in HConnection if zookeeper is reset + [HBASE-6671] - Kerberos authenticated super user should be able to retrieve proxied delegation tokens + [HBASE-6679] - RegionServer aborts due to race between compaction and split + [HBASE-6685] - Thrift DemoClient.pl got NullPointerException + [HBASE-6686] - HFile Quarantine fails with missing dirs in hadoop 2.0 + [HBASE-6688] - folder referred by thrift demo app instructions is outdated + [HBASE-6710] - 0.92/0.94 compatibility issues due to HBASE-5206 + [HBASE-6711] - Avoid local results copy in StoreScanner + [HBASE-6713] - Stopping META/ROOT RS may take 50mins when some region is splitting + [HBASE-6714] - TestMultiSlaveReplication#testMultiSlaveReplication may fail + [HBASE-6734] - Code duplication in LoadIncrementalHFiles + [HBASE-6757] - Very inefficient behaviour of scan using FilterList + [HBASE-6762] - HBASE-6340 broke SecureRPCEngine + [HBASE-6769] - HRS.multi eats NoSuchColumnFamilyException since HBASE-5021 + [HBASE-6784] - TestCoprocessorScanPolicy is sometimes flaky when run locally + [HBASE-6803] - script hbase should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH + [HBASE-6839] - Operations may be executed without holding rowLock + [HBASE-6842] - the jar used in coprocessor is not deleted in local which will exhaust the space of /tmp + [HBASE-6844] - upgrade 0.23 version dependency in 0.94 + [HBASE-6847] - HBASE-6649 broke replication + [HBASE-6851] - Race condition in TableAuthManager.updateGlobalCache() + [HBASE-6853] - IllegalArgument Exception is thrown when an empty region is spliitted. + [HBASE-6854] - Deletion of SPLITTING node on split rollback should clear the region from RIT + [HBASE-6868] - Skip checksum is broke; are we double-checksumming by default? + [HBASE-6871] - HFileBlockIndex Write Error in HFile V2 due to incorrect split into intermediate index blocks + [HBASE-6888] - HBase scripts ignore any HBASE_OPTS set in the environment + [HBASE-6889] - Ignore source control files with apache-rat + [HBASE-6900] - RegionScanner.reseek() creates NPE when a flush or compaction happens before the reseek. + [HBASE-6901] - Store file compactSelection throws ArrayIndexOutOfBoundsException + [HBASE-6906] - TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException + [HBASE-6912] - Filters are not properly applied in certain cases + [HBASE-6916] - HBA logs at info level errors that won't show in the shell + [HBASE-6920] - On timeout connecting to master, client can get stuck and never make progress + [HBASE-6927] - WrongFS using HRegionInfo.getTableDesc() and different fs for hbase.root and fs.defaultFS + [HBASE-6946] - JavaDoc missing from release tarballs + +Improvement + + [HBASE-3271] - Allow .META. table to be exported + [HBASE-5582] - "No HServerInfo found for" should be a WARNING message + [HBASE-5631] - hbck should handle case where .tableinfo file is missing. + [HBASE-5714] - Add write permissions check before any hbck run that modifies hdfs. + [HBASE-5728] - Methods Missing in HTableInterface + [HBASE-6286] - Upgrade maven-compiler-plugin to 2.5.1 + [HBASE-6291] - Don't retry increments on an invalid cell + [HBASE-6308] - Coprocessors should be loaded in a custom ClassLoader to prevent dependency conflicts with HBase + [HBASE-6373] - Add more context information to audit log messages + [HBASE-6444] - Expose the ability to set custom HTTP Request Headers for the REST client used by RemoteHTable + [HBASE-6458] - new comparator twice in checkAndPut, just reuse the first one + [HBASE-6522] - Expose locks and leases to Coprocessors + [HBASE-6586] - Quarantine Corrupted HFiles with hbck + [HBASE-6643] - Accept encoded region name in compacting/spliting region from shell + [HBASE-6644] - HBaseAdmin.createTable should wait more till table is enabled. + [HBASE-6860] - [replication] HBASE-6550 is too aggressive, DDOSes .META. + [HBASE-6914] - Scans/Gets/Mutations don't give a good error if the table is disabled. + +New Feature + + [HBASE-6427] - Pluggable compaction and scan policies via coprocessors + [HBASE-6505] - Allow shared RegionObserver state + [HBASE-6550] - Refactoring ReplicationSink to make it more responsive of cluster health + +Task + + [HBASE-5042] - TestReadWriteConsistencyControl should be renamed + [HBASE-6288] - In hbase-daemons.sh, description of the default backup-master file path is wrong + [HBASE-6538] - Remove copy_table.rb script + +Test + + [HBASE-6507] - [hbck] TestHBaseFsck ran into TableNotEnabledException + [HBASE-6593] - TestAdmin times out sometimes + + +Release 0.94.1 - 7/24/2012 +Sub-task + + [HBASE-5342] - Grant/Revoke global permissions + [HBASE-5372] - Table mutation operations should check table level rights, not global rights + [HBASE-5385] - Delete table/column should delete stored permissions on -acl- table + [HBASE-5659] - TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally + [HBASE-6061] - Fix ACL "Admin" Table inconsistent permission check + [HBASE-6062] - preCheckAndPut/Delete() checks for READ when also a WRITE is performed + [HBASE-6092] - Authorize flush, split, compact operations in AccessController + [HBASE-6157] - Revoke of Global permission is not taking effect without restart. + [HBASE-6181] - TestStoreFile fails with jdk1.7 + [HBASE-6188] - Remove the concept of table owner + [HBASE-6209] - ACL Corrections for AccessControllerProtocol apis + [HBASE-6224] - add Pre and Post coprocessor hooks for BulkLoad + [HBASE-6238] - Grant on META not taking effect + [HBASE-6252] - TABLE ADMIN should be allowed to relocate regions + [HBASE-6253] - Do not allow user to disable or drop ACL table + [HBASE-6292] - Compact can skip the security access control + [HBASE-6355] - Allow HBase to compile against JDK7 + +Bug + + [HBASE-4379] - [hbck] Does not complain about tables with no end region [Z,] + [HBASE-4470] - ServerNotRunningException coming out of assignRootAndMeta kills the Master + [HBASE-4891] - HTable.ClientScanner needs to clone the Scan object + [HBASE-5546] - Master assigns region in the original region server when opening region failed + [HBASE-5722] - NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW down. + [HBASE-5733] - AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE. + [HBASE-5741] - ImportTsv does not check for table existence + [HBASE-5757] - TableInputFormat should handle as many errors as possible + [HBASE-5806] - Handle split region related failures on master restart and RS restart + [HBASE-5840] - Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing the old status + [HBASE-5853] - java.lang.RuntimeException: readObject can't find class org.apache.hadoop.hdfs.protocol.HdfsFileStatus + [HBASE-5874] - When 'fs.default.name' not configured, the hbck tool and Merge tool throw IllegalArgumentException. + [HBASE-5875] - Process RIT and Master restart may remove an online server considering it as a dead server + [HBASE-5876] - TestImportExport has been failing against hadoop 0.23 profile + [HBASE-5883] - Backup master is going down due to connection refused exception + [HBASE-5894] - Table deletion failed but HBaseAdmin#deletetable reports it as success + [HBASE-5902] - Some scripts are not executable + [HBASE-5909] - SlabStats should be a daemon thread + [HBASE-5916] - RS restart just before master intialization we make the cluster non operative + [HBASE-5918] - Master will block forever at startup if root server dies between assigning root and assigning meta + [HBASE-5922] - HalfStoreFileReader seekBefore causes StackOverflowError + [HBASE-5927] - SSH and DisableTableHandler happening together does not clear the znode of the region and RIT map. + [HBASE-5928] - Hbck shouldn't npe when there are no tables. + [HBASE-5955] - Guava 11 drops MapEvictionListener and Hadoop 2.0.0-alpha requires it + [HBASE-5963] - ClassCastException: FileSystem$Cache$ClientFinalizer cannot be cast to Thread + [HBASE-5964] - HFileSystem: "No FileSystem for scheme: hdfs" + [HBASE-5966] - MapReduce based tests broken on Hadoop 2.0.0-alpha + [HBASE-5975] - Failed suppression of fs shutdown hook with Hadoop 2.0.0 + [HBASE-5986] - Clients can see holes in the META table when regions are being split + [HBASE-6002] - Possible chance of resource leak in HlogSplitter + [HBASE-6011] - Unable to start master in local mode + [HBASE-6016] - ServerShutdownHandler#processDeadRegion could return false for disabling table regions + [HBASE-6018] - hbck fails with a RejectedExecutionException when >50 regions present + [HBASE-6021] - NullPointerException when running LoadTestTool without specifying compression type + [HBASE-6029] - HBCK doesn't recover Balance switch if exception occurs in onlineHbck() + [HBASE-6046] - Master retry on ZK session expiry causes inconsistent region assignments. + [HBASE-6047] - Put.has() can't determine result correctly + [HBASE-6049] - Serializing "List" containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject() + [HBASE-6050] - HLogSplitter renaming recovered.edits and CJ removing the parent directory race, making the HBCK think cluster is inconsistent. + [HBASE-6056] - Restore hbase-default version check + [HBASE-6065] - Log for flush would append a non-sequential edit in the hlog, leading to possible data loss + [HBASE-6068] - Secure HBase cluster : Client not able to call some admin APIs + [HBASE-6069] - TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE + [HBASE-6070] - AM.nodeDeleted and SSH races creating problems for regions under SPLIT + [HBASE-6088] - Region splitting not happened for long time due to ZK exception while creating RS_ZK_SPLITTING node + [HBASE-6089] - SSH and AM.joinCluster causes Concurrent Modification exception. + [HBASE-6095] - ActiveMasterManager NullPointerException + [HBASE-6115] - NullPointerException is thrown when root and meta table regions are assigning to another RS. + [HBASE-6122] - Backup master does not become Active master after ZK exception + [HBASE-6126] - Fix broke TestLocalHBaseCluster in 0.92/0.94 + [HBASE-6133] - TestRestartCluster failing in 0.92 + [HBASE-6141] - InterfaceAudience breaks 0.94 on older versions of hadoop + [HBASE-6146] - Disabling of Catalog tables should not be allowed + [HBASE-6158] - Data loss if the words 'merges' or 'splits' are used as Column Family name + [HBASE-6160] - META entries from daughters can be deleted before parent entries + [HBASE-6164] - Correct the bug in block encoding usage in bulkload + [HBASE-6185] - Update javadoc for ConstantSizeRegionSplitPolicy class + [HBASE-6195] - Increment data will be lost when the memstore is flushed + [HBASE-6200] - KeyComparator.compareWithoutRow can be wrong when families have the same prefix + [HBASE-6210] - Backport HBASE-6197 to 0.94 + [HBASE-6227] - SSH and cluster startup causes data loss + [HBASE-6229] - AM.assign() should not set table state to ENABLED directly. + [HBASE-6236] - Offline meta repair fails if the HBase base mount point is on a different cluster/volume than its parent in a ViewFS or similar FS + [HBASE-6237] - Fix race on ACL table creation in TestTablePermissions + [HBASE-6240] - Race in HCM.getMaster stalls clients + [HBASE-6246] - Admin.move without specifying destination does not go through AccessController + [HBASE-6248] - Jetty init may fail if directory name contains "master" + [HBASE-6265] - Calling getTimestamp() on a KV in cp.prePut() causes KV not to be flushed + [HBASE-6269] - Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue + [HBASE-6281] - Assignment need not be called for disabling table regions during clean cluster start up. + [HBASE-6284] - Introduce HRegion#doMiniBatchMutation() + [HBASE-6293] - HMaster does not go down while splitting logs even if explicit shutdown is called. + [HBASE-6303] - HCD.setCompressionType should use Enum support for storing compression types as strings + [HBASE-6311] - Data error after majorCompaction caused by keeping MVCC for opened scanners + [HBASE-6313] - Client hangs because the client is not notified + [HBASE-6319] - ReplicationSource can call terminate on itself and deadlock + [HBASE-6325] - [replication] Race in ReplicationSourceManager.init can initiate a failover even if the node is alive + [HBASE-6326] - Avoid nested retry loops in HConnectionManager + [HBASE-6328] - FSHDFSUtils#recoverFileLease tries to rethrow InterruptedException but actually shallows it + [HBASE-6329] - Stopping META regionserver when splitting region could cause daughter region to be assigned twice + [HBASE-6337] - [MTTR] Remove renaming tmp log file in SplitLogManager + [HBASE-6357] - Failed distributed log splitting stuck on master web UI + [HBASE-6369] - HTable is not closed in AggregationClient + [HBASE-6375] - Master may be using a stale list of region servers for creating assignment plan during startup + [HBASE-6377] - HBASE-5533 metrics miss all operations submitted via MultiAction + [HBASE-6380] - bulkload should update the store.storeSize + [HBASE-6392] - UnknownRegionException blocks hbck from sideline big overlap regions + [HBASE-6394] - verifyrep MR job map tasks throws NullPointerException + [HBASE-6397] - [hbck] print out bulk load commands for sidelined regions if necessary + [HBASE-6406] - TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently + [HBASE-6420] - Gracefully shutdown logsyncer + [HBASE-6426] - Add Hadoop 2.0.x profile to 0.92+ + [HBASE-6440] - SplitLogManager - log the exception when failed to finish split log file + [HBASE-6443] - HLogSplitter should ignore 0 length files + [HBASE-6445] - rat check fails if hs_err_pid26514.log dropped in tests + +Improvement + + [HBASE-4720] - Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server + [HBASE-5360] - [uberhbck] Add options for how to handle offline split parents. + [HBASE-5630] - hbck should disable the balancer using synchronousBalanceSwitch. + [HBASE-5802] - Change the default metrics class to NullContextWithUpdateThread + [HBASE-5838] - Add an LZ4 compression option to HFile + [HBASE-5887] - Make TestAcidGuarantees usable for system testing. + [HBASE-5892] - [hbck] Refactor parallel WorkItem* to Futures. + [HBASE-5913] - Speed up the full scan of META + [HBASE-5973] - Add ability for potentially long-running IPC calls to abort if client disconnects + [HBASE-6010] - Security audit logger configuration for log4j + [HBASE-6013] - Polish sharp edges from CopyTable + [HBASE-6022] - Include Junit in the libs when packaging so that TestAcidGaurntee can run + [HBASE-6023] - Normalize security audit logging level with Hadoop + [HBASE-6040] - Use block encoding and HBase handled checksum verification in bulk loading using HFileOutputFormat + [HBASE-6067] - HBase won't start when hbase.rootdir uses ViewFileSystem + [HBASE-6114] - CacheControl flags should be tunable per table schema per CF + [HBASE-6124] - Backport HBASE-6033 to 0.90, 0.92 and 0.94 + [HBASE-6161] - Log Error when thrift server fails to start up. + [HBASE-6173] - hbck check specified tables only + [HBASE-6207] - Add jitter to client retry timer + [HBASE-6214] - Backport HBASE-5998 to 94.1 + [HBASE-6244] - [REST] Result generators do not need to query table schema + [HBASE-6247] - [REST] HTablePool.putTable is deprecated + [HBASE-6267] - hbase.store.delete.expired.storefile should be true by default + [HBASE-6283] - [region_mover.rb] Add option to exclude list of hosts on unload instead of just assuming the source node. + [HBASE-6314] - Fast fail behavior for unauthenticated user + [HBASE-6332] - Improve POM for better integration with downstream ivy projects + [HBASE-6334] - TestImprovement for TestHRegion.testWritesWhileGetting + [HBASE-6341] - Publicly expose HConnectionKey + [HBASE-6363] - HBaseConfiguration can carry a main method that dumps XML output for debug purposes + [HBASE-6382] - Upgrade Jersey to 1.8 to match Hadoop 1 and 2 + [HBASE-6384] - hbck should group together those sidelined regions need to be bulk loaded later + [HBASE-6433] - Improve HBaseServer#getRemoteAddress by utilizing HBaseServer.Connection.hostAddress + +New Feature + + [HBASE-2730] - Expose RS work queue contents on web UI + [HBASE-4956] - Control direct memory buffer consumption by HBaseClient + [HBASE-5609] - Add the ability to pass additional information for slow query logging + [HBASE-5886] - Add new metric for possible data loss due to puts without WAL + [HBASE-6044] - copytable: remove rs.* parameters + +Task + + [HBASE-6001] - Upgrade slf4j to 1.6.1 + [HBASE-6034] - Upgrade Hadoop dependencies + [HBASE-6077] - Document the most common secure RPC troubleshooting resolutions + [HBASE-6129] - Backport of Add Increment Coalescing in thrift. + [HBASE-6131] - Add attribution for code added by HBASE-5533 metrics + +Test + + [HBASE-5985] - TestMetaMigrationRemovingHTD failed with HADOOP 2.0.0 + + +Release 0.94.0 - 5/1/2012 +Sub-task + + [HBASE-4343] - Get the TestAcidGuarantee unit test to fail consistently + [HBASE-4345] - Ensure that Scanners that read from the storefiles respect MVCC + [HBASE-4346] - Optimise the storage that we use for storing MVCC information. + [HBASE-4485] - Eliminate window of missing Data + [HBASE-4517] - Document new replication features in 0.92 + [HBASE-4544] - Rename RWCC to MVCC + [HBASE-4594] - Ensure that KV's newer than the oldest-living-scanner is not accounted for the maxVersions during flush/compaction. + [HBASE-4661] - Ability to export the list of files for a some or all column families for a given region + [HBASE-4682] - Support deleted rows using Import/Export + [HBASE-4908] - HBase cluster test tool (port from 0.89-fb) + [HBASE-4911] - Clean shutdown + [HBASE-4979] - Setting KEEP_DELETE_CELLS fails in shell + [HBASE-4981] - add raw scan support to shell + [HBASE-4998] - Support deleted rows in CopyTable + [HBASE-5005] - Add DEFAULT_MIN_VERSIONS to HColumnDescriptor.DEFAULT_VALUES + [HBASE-5058] - Allow HBaseAdmin to use an existing connection + [HBASE-5096] - Replication does not handle deletes correctly. + [HBASE-5118] - Fix Scan documentation + [HBASE-5143] - Fix config typo in pluggable load balancer factory + [HBASE-5203] - Group atomic put/delete operation into a single WALEdit to handle region server failures. + [HBASE-5266] - Add documentation for ColumnRangeFilter + [HBASE-5346] - Fix testColumnFamilyCompression and test_TIMERANGE in TestHFileOutputFormat + [HBASE-5368] - Move PrefixSplitKeyPolicy out of the src/test into src, so it is accessible in HBase installs + [HBASE-5371] - Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API + [HBASE-5413] - Rename RowMutation to RowMutations + [HBASE-5431] - Improve delete marker handling in Import M/R jobs + [HBASE-5460] - Add protobuf as M/R dependency jar + [HBASE-5497] - Add protobuf as M/R dependency jar (mapred) + [HBASE-5523] - Fix Delete Timerange logic for KEEP_DELETED_CELLS + [HBASE-5541] - Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks + [HBASE-5638] - Backport to 0.90 and 0.92 - NPE reading ZK config in HBase + [HBASE-5641] - decayingSampleTick1 prevents HBase from shutting down. + [HBASE-5793] - TestHBaseFsck#TestNoHdfsTable test hangs after client retries increased + +Bug + + [HBASE-2856] - TestAcidGuarantee broken on trunk + [HBASE-3443] - ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix + [HBASE-3690] - Option to Exclude Bulk Import Files from Minor Compaction + [HBASE-3987] - Fix a NullPointerException on a failure to load Bloom filter data + [HBASE-4065] - TableOutputFormat ignores failure to create table instance + [HBASE-4078] - Silent Data Offlining During HDFS Flakiness + [HBASE-4105] - Stargate does not support Content-Type: application/json and Content-Encoding: gzip in parallel + [HBASE-4116] - [stargate] StringIndexOutOfBoundsException in row spec parse + [HBASE-4326] - Tests that use HBaseTestingUtility.startMiniCluster(n) should shutdown with HBaseTestingUtility.shutdownMiniCluster. + [HBASE-4397] - -ROOT-, .META. tables stay offline for too long in recovery phase after all RSs are shutdown at the same time + [HBASE-4398] - If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml. + [HBASE-4476] - Compactions must fail if column tracker gets columns out of order + [HBASE-4496] - HFile V2 does not honor setCacheBlocks when scanning. + [HBASE-4607] - Split log worker should terminate properly when waiting for znode + [HBASE-4609] - ThriftServer.getRegionInfo() is expecting old ServerName format, need to use new Addressing class instead + [HBASE-4610] - Port HBASE-3380 (Master failover can split logs of live servers) to 92/trunk (definitely bring in config params, decide if we need to do more to fix the bug) + [HBASE-4626] - Filters unnecessarily copy byte arrays... + [HBASE-4645] - Edits Log recovery losing data across column families + [HBASE-4648] - Bytes.toBigDecimal() doesn't use offset + [HBASE-4658] - Put attributes are not exposed via the ThriftServer + [HBASE-4673] - NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 + [HBASE-4679] - Thrift null mutation error + [HBASE-4691] - Remove more unnecessary byte[] copies from KeyValues + [HBASE-4729] - Clash between region unassign and splitting kills the master + [HBASE-4745] - LRU Statistics thread should be daemon + [HBASE-4769] - Abort RegionServer Immediately on OOME + [HBASE-4776] - HLog.closed should be checked inside of updateLock + [HBASE-4778] - Don't ignore corrupt StoreFiles when opening a region + [HBASE-4790] - Occasional TestDistributedLogSplitting failure + [HBASE-4792] - SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline + [HBASE-4795] - Fix TestHFileBlock when running on a 32-bit JVM + [HBASE-4797] - [availability] Skip recovered.edits files with edits we know older than what region currently has + [HBASE-4805] - Allow better control of resource consumption in HTable + [HBASE-4819] - TestShell broke in trunk; typo + [HBASE-4825] - TestRegionServersMetrics and TestZKLeaderManager are not categorized (small/medium/large) + [HBASE-4826] - Modify hbasetests.sh to take into account the new pom.xml with surefire + [HBASE-4832] - TestRegionServerCoprocessorExceptionWithAbort fails if the region server stops too fast + [HBASE-4853] - HBASE-4789 does overzealous pruning of seqids + [HBASE-4874] - Run tests with non-secure random, some tests hang otherwise + [HBASE-4878] - Master crash when splitting hlog may cause data loss + [HBASE-4886] - truncate fails in HBase shell + [HBASE-4890] - fix possible NPE in HConnectionManager + [HBASE-4932] - Block cache can be mistakenly instantiated by tools + [HBASE-4936] - Cached HRegionInterface connections crash when getting UnknownHost exceptions + [HBASE-4937] - Error in Quick Start Shell Exercises + [HBASE-4942] - HMaster is unable to start of HFile V1 is used + [HBASE-4946] - HTable.coprocessorExec (and possibly coprocessorProxy) does not work with dynamically loaded coprocessors (from hdfs or local system), because the RPC system tries to deserialize an unknown class. + [HBASE-4993] - Performance regression in minicluster creation + [HBASE-5003] - If the master is started with a wrong root dir, it gets stuck and can't be killed + [HBASE-5010] - Filter HFiles based on TTL + [HBASE-5015] - Remove some leaks in tests due to lack of HTable.close() + [HBASE-5026] - Add coprocessor hook to HRegionServer.ScannerListener.leaseExpired() + [HBASE-5027] - HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge. + [HBASE-5038] - Some tests leak connections + [HBASE-5041] - Major compaction on non existing table does not throw error + [HBASE-5051] - HBaseTestingUtility#getHBaseAdmin() creates a new HBaseAdmin instance at each call + [HBASE-5053] - HCM Tests leak connections + [HBASE-5055] - Build against hadoop 0.22 broken + [HBASE-5068] - RC1 can not build its hadoop-0.23 profile + [HBASE-5085] - fix test-patch script from setting the ulimit + [HBASE-5088] - A concurrency issue on SoftValueSortedMap + [HBASE-5091] - [replication] Update replication doc to reflect current znode structure + [HBASE-5097] - RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE + [HBASE-5099] - ZK event thread waiting for root region assignment may block server shutdown handler for the region sever the root region was on + [HBASE-5100] - Rollback of split could cause closed region to be opened again + [HBASE-5103] - Fix improper master znode deserialization + [HBASE-5120] - Timeout monitor races with table disable handler + [HBASE-5121] - MajorCompaction may affect scan's correctness + [HBASE-5141] - Memory leak in MonitoredRPCHandlerImpl + [HBASE-5152] - Region is on service before completing initialization when doing rollback of split, it will affect read correctness + [HBASE-5163] - TestLogRolling#testLogRollOnDatanodeDeath fails sometimes on Jenkins or hadoop QA ("The directory is already locked.") + [HBASE-5172] - HTableInterface should extend java.io.Closeable + [HBASE-5176] - AssignmentManager#getRegion: logging nit adds a redundant '+' + [HBASE-5182] - TBoundedThreadPoolServer threadKeepAliveTimeSec is not configured properly + [HBASE-5195] - [Coprocessors] preGet hook does not allow overriding or wrapping filter on incoming Get + [HBASE-5196] - Failure in region split after PONR could cause region hole + [HBASE-5200] - AM.ProcessRegionInTransition() and AM.handleRegion() race thus leaving the region assignment inconsistent + [HBASE-5206] - Port HBASE-5155 to 0.92, 0.94, and TRUNK + [HBASE-5212] - Fix test TestTableMapReduce against 0.23. + [HBASE-5213] - "hbase master stop" does not bring down backup masters + [HBASE-5221] - bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout + [HBASE-5228] - [REST] Rip out "transform" feature + [HBASE-5267] - Add a configuration to disable the slab cache by default + [HBASE-5271] - Result.getValue and Result.getColumnLatest return the wrong column. + [HBASE-5278] - HBase shell script refers to removed "migrate" functionality + [HBASE-5281] - Should a failure in creating an unassigned node abort the master? + [HBASE-5282] - Possible file handle leak with truncated HLog file. + [HBASE-5283] - Request counters may become negative for heavily loaded regions + [HBASE-5286] - bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation + [HBASE-5288] - Security source code dirs missing from 0.92.0 release tarballs. + [HBASE-5290] - [FindBugs] Synchronization on boxed primitive + [HBASE-5292] - getsize per-CF metric incorrectly counts compaction related reads as well + [HBASE-5317] - Fix TestHFileOutputFormat to work against hadoop 0.23 + [HBASE-5327] - Print a message when an invalid hbase.rootdir is passed + [HBASE-5331] - Off by one bug in util.HMerge + [HBASE-5345] - CheckAndPut doesn't work when value is empty byte[] + [HBASE-5348] - Constraint configuration loaded with bloat + [HBASE-5350] - Fix jamon generated package names + [HBASE-5351] - hbase completebulkload to a new table fails in a race + [HBASE-5364] - Fix source files missing licenses in 0.92 and trunk + [HBASE-5384] - Up heap used by hadoopqa + [HBASE-5387] - Reuse compression streams in HFileBlock.Writer + [HBASE-5398] - HBase shell disable_all/enable_all/drop_all promp wrong tables for confirmation + [HBASE-5415] - FSTableDescriptors should handle random folders in hbase.root.dir better + [HBASE-5420] - TestImportTsv does not shut down MR Cluster correctly (fails against 0.23 hadoop) + [HBASE-5423] - Regionserver may block forever on waitOnAllRegionsToClose when aborting + [HBASE-5425] - Punt on the timeout doesn't work in BulkEnabler#waitUntilDone (master's EnableTableHandler) + [HBASE-5437] - HRegionThriftServer does not start because of a bug in HbaseHandlerMetricsProxy + [HBASE-5466] - Opening a table also opens the metatable and never closes it. + [HBASE-5470] - Make DataBlockEncodingTool work correctly with no native compression codecs loaded + [HBASE-5473] - Metrics does not push pread time + [HBASE-5477] - Cannot build RPM for hbase-0.92.0 + [HBASE-5480] - Fixups to MultithreadedTableMapper for Hadoop 0.23.2+ + [HBASE-5481] - Uncaught UnknownHostException prevents HBase from starting + [HBASE-5484] - Spelling mistake in error message in HMasterCommandLine + [HBASE-5485] - LogCleaner refers to non-existant SnapshotLogCleaner + [HBASE-5488] - OfflineMetaRepair doesn't support hadoop 0.20's fs.default.name property + [HBASE-5499] - dev-support/test-patch.sh does not have execute perms + [HBASE-5502] - region_mover.rb fails to load regions back to original server for regions only containing empty tables. + [HBASE-5507] - ThriftServerRunner.HbaseHandler.getRegionInfo() and getTableRegions() do not use ByteBuffer correctly + [HBASE-5514] - Compile against hadoop 0.24-SNAPSHOT + [HBASE-5522] - hbase 0.92 test artifacts are missing from Maven central + [HBASE-5524] - Add a couple of more filters to our rat exclusion set + [HBASE-5529] - MR test failures becuase MALLOC_ARENA_MAX is not set + [HBASE-5531] - Maven hadoop profile (version 23) needs to be updated with latest 23 snapshot + [HBASE-5535] - Make the functions in task monitor synchronized + [HBASE-5537] - MXBean shouldn't have a dependence on InterfaceStability until 0.96 + [HBASE-5545] - region can't be opened for a long time. Because the creating File failed. + [HBASE-5552] - Clean up our jmx view; its a bit of a mess + [HBASE-5562] - test-patch.sh reports a javadoc warning when there are no new javadoc warnings + [HBASE-5563] - HRegionInfo#compareTo should compare regionId as well + [HBASE-5567] - test-patch.sh has logic error in findbugs check + [HBASE-5568] - Multi concurrent flushcache() for one region could cause data loss + [HBASE-5569] - Do not collect deleted KVs when they are still in use by a scanner. + [HBASE-5574] - DEFAULT_MAX_FILE_SIZE defaults to a negative value + [HBASE-5579] - A Delete Version could mask other values + [HBASE-5581] - Creating a table with invalid syntax does not give an error message when it fails + [HBASE-5586] - [replication] NPE in ReplicationSource when creating a stream to an inexistent cluster + [HBASE-5596] - Few minor bugs from HBASE-5209 + [HBASE-5597] - Findbugs check in test-patch.sh always fails + [HBASE-5603] - rolling-restart.sh script hangs when attempting to detect expiration of /hbase/master znode. + [HBASE-5606] - SplitLogManger async delete node hangs log splitting when ZK connection is lost + [HBASE-5611] - Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size + [HBASE-5613] - ThriftServer getTableRegions does not return serverName and port + [HBASE-5623] - Race condition when rolling the HLog and hlogFlush + [HBASE-5624] - Aborting regionserver when splitting region, may cause daughter region not assigned by ServerShutdownHandler. + [HBASE-5633] - NPE reading ZK config in HBase + [HBASE-5635] - If getTaskList() returns null, splitlogWorker would go down and it won't serve any requests + [HBASE-5636] - TestTableMapReduce doesn't work properly. + [HBASE-5639] - The logic used in waiting for region servers during startup is broken + [HBASE-5656] - LoadIncrementalHFiles createTable should detect and set compression algorithm + [HBASE-5663] - MultithreadedTableMapper doesn't work. + [HBASE-5665] - Repeated split causes HRegionServer failures and breaks table + [HBASE-5669] - AggregationClient fails validation for open stoprow scan + [HBASE-5680] - Improve compatibility warning about HBase with Hadoop 0.23.x + [HBASE-5689] - Skipping RecoveredEdits may cause data loss + [HBASE-5690] - compression does not work in Store.java of 0.94 + [HBASE-5694] - getRowsWithColumnsTs() in Thrift service handles timestamps incorrectly + [HBASE-5701] - Put RegionServerDynamicStatistics under RegionServer in MBean hierarchy rather than have it as a peer. + [HBASE-5717] - Scanner metrics are only reported if you get to the end of a scanner + [HBASE-5720] - HFileDataBlockEncoderImpl uses wrong header size when reading HFiles with no checksums + [HBASE-5722] - NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW down. + [HBASE-5724] - Row cache of KeyValue should be cleared in readFields(). + [HBASE-5736] - ThriftServerRunner.HbaseHandler.mutateRow() does not use ByteBuffer correctly + [HBASE-5743] - Support GIT patches + [HBASE-5773] - HtablePool constructor not reading config files in certain cases + [HBASE-5780] - Fix race in HBase regionserver startup vs ZK SASL authentication + [HBASE-5781] - Zookeeper session got closed while trying to assign the region to RS using hbck -fix + [HBASE-5782] - Edits can be appended out of seqid order since HBASE-4487 + [HBASE-5787] - Table owner can't disable/delete his/her own table + [HBASE-5795] - HServerLoad$RegionLoad breaks 0.92<->0.94 compatibility + [HBASE-5825] - TestHLog not running any tests; fix + [HBASE-5833] - 0.92 build has been failing pretty consistently on TestMasterFailover.... + [HBASE-5848] - Create table with EMPTY_START_ROW passed as splitKey causes the HMaster to abort + [HBASE-5849] - On first cluster startup, RS aborts if root znode is not available + [HBASE-5850] - Refuse operations from Admin before master is initialized - fix for all branches. + [HBASE-5857] - RIT map in RS not getting cleared while region opening + [HBASE-5861] - Hadoop 23 compilation broken due to tests introduced in HBASE-5604 + [HBASE-5864] - Error while reading from hfile in 0.94 + [HBASE-5865] - test-util.sh broken with unittest updates + [HBASE-5866] - Canary in tool package but says its in tools. + [HBASE-5871] - Usability regression, we don't parse compression algos anymore + [HBASE-5873] - TimeOut Monitor thread should be started after atleast one region server registers. + [HBASE-5884] - MapReduce package info has broken link to bulk-loads + [HBASE-5885] - Invalid HFile block magic on Local file System + [HBASE-5893] - Allow spaces in coprocessor conf (aka trim() className) + [HBASE-5897] - prePut coprocessor hook causing substantial CPU usage + [HBASE-5908] - TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog + [HBASE-6265] - Calling getTimestamp() on a KV in cp.prePut() causes KV not to be flushed + [HBASE-6357] - Failed distributed log splitting stuck on master web UI + +Improvement + + [HBASE-1744] - Thrift server to match the new java api. + [HBASE-2418] - add support for ZooKeeper authentication + [HBASE-3373] - Allow regions to be load-balanced by table + [HBASE-3433] - Remove the KV copy of every KV in Scan; introduced by HBASE-3232 + [HBASE-3512] - Coprocessors: Shell support for listing currently loaded coprocessor set + [HBASE-3565] - Add metrics to keep track of slow HLog appends + [HBASE-3763] - Add Bloom Block Index Support + [HBASE-3850] - Log more details when a scanner lease expires + [HBASE-3924] - Improve Shell's CLI help + [HBASE-3949] - Add "Master" link to RegionServer pages + [HBASE-4058] - Extend TestHBaseFsck with a complete .META. recovery scenario + [HBASE-4062] - Multi-column scanner unit test + [HBASE-4070] - [Coprocessors] Improve region server metrics to report loaded coprocessors to master + [HBASE-4076] - hbase should pick up HADOOP_CONF_DIR on its classpath + [HBASE-4131] - Make the Replication Service pluggable via a standard interface definition + [HBASE-4132] - Extend the WALActionsListener API to accomodate log archival + [HBASE-4145] - Provide metrics for hbase client + [HBASE-4213] - Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. + [HBASE-4218] - Data Block Encoding of KeyValues (aka delta encoding / prefix compression) + [HBASE-4365] - Add a decent heuristic for region size + [HBASE-4418] - Show all the hbase configuration in the web ui + [HBASE-4439] - Move ClientScanner out of HTable + [HBASE-4440] - add an option to presplit table to PerformanceEvaluation + [HBASE-4461] - Expose getRowOrBefore via Thrift + [HBASE-4463] - Run more aggressive compactions during off peak hours + [HBASE-4465] - Lazy-seek optimization for StoreFile scanners + [HBASE-4469] - Avoid top row seek by looking up ROWCOL bloomfilter + [HBASE-4480] - Testing script to simplify local testing + [HBASE-4487] - The increment operation can release the rowlock before sync-ing the Hlog + [HBASE-4489] - Better key splitting in RegionSplitter + [HBASE-4519] - 25s sleep when expiring sessions in tests + [HBASE-4522] - Make hbase-site-custom.xml override the hbase-site.xml + [HBASE-4528] - The put operation can release the rowlock before sync-ing the Hlog + [HBASE-4532] - Avoid top row seek by dedicated bloom filter for delete family bloom filter + [HBASE-4542] - add filter info to slow query logging + [HBASE-4554] - Allow set/unset coprocessor table attributes from shell. + [HBASE-4568] - Make zk dump jsp response more quickly + [HBASE-4585] - Avoid next operations (and instead reseek) when current kv is deleted + [HBASE-4591] - TTL for old HLogs should be calculated from last modification time. + [HBASE-4612] - Allow ColumnPrefixFilter to support multiple prefixes + [HBASE-4627] - Ability to specify a custom start/end to RegionSplitter + [HBASE-4628] - Enhance Table Create Presplit Functionality within the HBase Shell + [HBASE-4640] - Catch ClosedChannelException and document it + [HBASE-4657] - Improve the efficiency of our MR jobs with a few configurations + [HBASE-4669] - Add an option of using round-robin assignment for enabling table + [HBASE-4696] - HRegionThriftServer' might have to indefinitely do redirtects + [HBASE-4704] - A JRuby script for identifying active master + [HBASE-4737] - Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM + [HBASE-4746] - Use a random ZK client port in unit tests so we can run them in parallel + [HBASE-4752] - Don't create an unnecessary LinkedList when evicting from the BlockCache + [HBASE-4760] - Add Developer Debug Options to HBase Config + [HBASE-4761] - Add Developer Debug Options to HBase Config + [HBASE-4764] - naming errors for TestHLogUtils and SoftValueSortedMapTest + [HBASE-4779] - TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite + [HBASE-4780] - Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown + [HBASE-4781] - Pom update to use the new versions of surefire & junit + [HBASE-4783] - Improve RowCounter to count rows in a specific key range. + [HBASE-4787] - Make corePool as a configurable parameter in HTable + [HBASE-4798] - Sleeps and synchronisation improvements for tests + [HBASE-4809] - Per-CF set RPC metrics + [HBASE-4820] - Distributed log splitting coding enhancement to make it easier to understand, no semantics change + [HBASE-4847] - Activate single jvm for small tests on jenkins + [HBASE-4863] - Make Thrift server thread pool bounded and add a command-line UI test + [HBASE-4884] - Allow environment overrides for various HBase processes + [HBASE-4933] - Ability to calculate the blockcache hit ratio for the last few minutes + [HBASE-4938] - Create a HRegion.getScanner public method that allows reading from a specified readPoint + [HBASE-4940] - hadoop-metrics.properties can include configuration of the "rest" context for ganglia + [HBASE-4957] - Clean up some log messages, code in RecoverableZooKeeper + [HBASE-4964] - Add builddate, make less sections in toc, and add header and footer customizations + [HBASE-4965] - Monitor the open file descriptors and the threads counters during the unit tests + [HBASE-4970] - Add a parameter so that keepAliveTime of Htable thread pool can be changed + [HBASE-4971] - Useless sleeps in TestTimestampsFilter and TestMultipleTimestamps + [HBASE-4973] - On failure, HBaseAdmin sleeps one time too many + [HBASE-4989] - Metrics to measure sequential reads and random reads separately + [HBASE-4995] - Increase zk maxClientCnxns to give us some head room + [HBASE-5014] - PutSortReducer should adhere to memory limits + [HBASE-5017] - Bump the default hfile.block.cache.size because of HFileV2 + [HBASE-5021] - Enforce upper bound on timestamp + [HBASE-5033] - Opening/Closing store in parallel to reduce region open/close time + [HBASE-5064] - utilize surefire tests parallelization + [HBASE-5072] - Support Max Value for Per-Store Metrics + [HBASE-5074] - support checksums in HBase block cache + [HBASE-5134] - Remove getRegionServerWithoutRetries and getRegionServerWithRetries from HConnection Interface + [HBASE-5166] - MultiThreaded Table Mapper analogous to MultiThreaded Mapper in hadoop + [HBASE-5167] - We shouldn't be injecting 'Killing [daemon]' into logs, when we aren't doing that. + [HBASE-5186] - Add metrics to ThriftServer + [HBASE-5189] - Add metrics to keep track of region-splits in RS + [HBASE-5190] - Limit the IPC queue size based on calls' payload size + [HBASE-5193] - Use TBoundedThreadPoolServer in HRegionThriftServer + [HBASE-5197] - [replication] Handle socket timeouts in ReplicationSource to prevent DDOS + [HBASE-5199] - Delete out of TTL store files before compaction selection + [HBASE-5201] - Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer + [HBASE-5209] - HConnection/HMasterInterface should allow for way to get hostname of currently active master in multi-master HBase setup + [HBASE-5246] - Regenerate code with thrift 0.8.0 + [HBASE-5255] - Use singletons for OperationStatus to save memory + [HBASE-5259] - Normalize the RegionLocation in TableInputFormat by the reverse DNS lookup. + [HBASE-5297] - Update metrics numOpenConnections and callQueueLen directly in HBaseServer + [HBASE-5298] - Add thrift metrics to thrift2 + [HBASE-5304] - Pluggable split key policy + [HBASE-5310] - HConnectionManager server cache key enhancement + [HBASE-5325] - Expose basic information about the master-status through jmx beans + [HBASE-5332] - Deterministic Compaction Jitter + [HBASE-5358] - HBaseObjectWritable should be able to serialize/deserialize generic arrays + [HBASE-5363] - Automatically run rat check on mvn release builds + [HBASE-5388] - Tune HConnectionManager#getCachedLocation method + [HBASE-5393] - Consider splitting after flushing + [HBASE-5394] - Add ability to include Protobufs in HbaseObjectWritable + [HBASE-5395] - CopyTable needs to use GenericOptionsParser + [HBASE-5411] - Add more metrics for ThriftMetrics + [HBASE-5421] - use hadoop-client/hadoop-minicluster artifacts for Hadoop 0.23 build + [HBASE-5428] - Allow for custom filters to be registered within the Thrift interface + [HBASE-5433] - [REST] Add metrics to keep track of success/failure count + [HBASE-5434] - [REST] Include more metrics in cluster status request + [HBASE-5436] - Right-size the map when reading attributes. + [HBASE-5439] - Fix some performance findbugs issues + [HBASE-5440] - Allow Import to optionally use HFileOutputFormat + [HBASE-5442] - Use builder pattern in StoreFile and HFile + [HBASE-5454] - Refuse operations from Admin before master is initialized + [HBASE-5464] - Log warning message when thrift calls throw exceptions + [HBASE-5483] - Allow configurable host to bind to for starting REST server from commandline + [HBASE-5489] - Add HTable accessor to get regions for a key range + [HBASE-5508] - Add an option to allow test output to show on the terminal + [HBASE-5520] - Support reseek() at RegionScanner + [HBASE-5533] - Add more metrics to HBase + [HBASE-5551] - Some functions should not be used by customer code and must be deprecated in 0.94 + [HBASE-5560] - Avoid RegionServer GC caused by timed-out calls + [HBASE-5588] - Deprecate/remove AssignmentManager#clearRegionFromTransition + [HBASE-5589] - Add of the offline call to the Master Interface + [HBASE-5592] - Make it easier to get a table from shell + [HBASE-5618] - SplitLogManager - prevent unnecessary attempts to resubmits + [HBASE-5670] - Have Mutation implement the Row interface. + [HBASE-5671] - hbase.metrics.showTableName should be true by default + [HBASE-5682] - Allow HConnectionImplementation to recover from ZK connection loss (for 0.94 only) + [HBASE-5706] - "Dropping fs latency stats since buffer is full" spam + [HBASE-5712] - Parallelize load of .regioninfo files in diagnostic/repair portion of hbck. + [HBASE-5734] - Change hbck sideline root + [HBASE-5735] - Clearer warning message when connecting a non-secure HBase client to a secure HBase server + [HBASE-5737] - Minor Improvements related to balancer. + [HBASE-5748] - Enable lib directory in jar file for coprocessor + [HBASE-5770] - Add a clock skew warning threshold + [HBASE-5775] - ZKUtil doesn't handle deleteRecurisively cleanly + [HBASE-5823] - Hbck should be able to print help + [HBASE-5862] - After Region Close remove the Operation Metrics. + [HBASE-5863] - Improve the graceful_stop.sh CLI help (especially about reloads) + [HBASE-6173] - hbck check specified tables only + [HBASE-5360] - [uberhbck] Add options for how to handle offline split parents. + +New Feature + + [HBASE-2947] - MultiIncrement/MultiAppend (MultiGet functionality for increments and appends) + [HBASE-3134] - [replication] Add the ability to enable/disable streams + [HBASE-3584] - Allow atomic put/delete in one call + [HBASE-3856] - Build a tree structure data block index inside of the HFile + [HBASE-4102] - atomicAppend: A put that appends to the latest version of a cell; i.e. reads current value then adds the bytes offered by the client to the tail and writes out a new entry + [HBASE-4219] - Add Per-Column Family Metrics + [HBASE-4393] - Implement a canary monitoring program + [HBASE-4460] - Support running an embedded ThriftServer within a RegionServer + [HBASE-4536] - Allow CF to retain deleted rows + [HBASE-4608] - HLog Compression + [HBASE-4629] - enable automated patch testing for hbase + [HBASE-4683] - Always cache index and bloom blocks + [HBASE-4698] - Let the HFile Pretty Printer print all the key values for a specific row. + [HBASE-4768] - Per-(table, columnFamily) metrics with configurable table name inclusion + [HBASE-5128] - [uber hbck] Online automated repair of table integrity and region consistency problems + [HBASE-5177] - HTable needs a non cached version of getRegionLocation + [HBASE-5229] - Provide basic building blocks for "multi-row" local transactions. + [HBASE-5526] - Configurable file and directory based umask + [HBASE-5599] - [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies + [HBASE-5604] - M/R tool to replay WAL files + [HBASE-5719] - Enhance hbck to sideline overlapped mega regions + +Task + + [HBASE-4256] - Intra-row scanning (part deux) + [HBASE-4429] - Provide synchronous balanceSwitch() + [HBASE-4611] - Add support for Phabricator/Differential as an alternative code review tool + [HBASE-4712] - Document rules for writing tests + [HBASE-4751] - Make TestAdmin#testEnableTableRoundRobinAssignment friendly to concurrent tests + [HBASE-4968] - Add to troubleshooting workaround for direct buffer oome's. + [HBASE-5011] - Move test-util.sh from src/test/bin to dev-tools + [HBASE-5084] - Allow different HTable instances to share one ExecutorService + [HBASE-5111] - Upgrade zookeeper to 3.4.2 release + [HBASE-5173] - Commit hbase-4480 findHangingTest.sh script under dev-support + [HBASE-5256] - Use WritableUtils.readVInt() in RegionLoad.readFields() + [HBASE-5264] - Add 0.92.0 upgrade guide + [HBASE-5294] - Make sure javadoc is included in tarball bundle when we release + [HBASE-5400] - Some tests does not have annotations for (Small|Medium|Large)Tests + [HBASE-5427] - Upgrade our zk to 3.4.3 + [HBASE-5511] - More doc on maven release process + [HBASE-5715] - Revert 'Instant schema alter' for now, HBASE-4213 + [HBASE-5721] - Update bundled hadoop to be 1.0.2 (it was just released) + [HBASE-5758] - Forward port "HBASE-4109 Hostname returned via reverse dns lookup contains trailing period if configured interface is not 'default'" + [HBASE-5836] - Backport per region metrics from HBASE-3614 to 0.94.1 + +Test + + [HBASE-4516] - HFile-level load tester with compaction and random-read workloads + [HBASE-4534] - A new unit test for lazy seek and StoreScanner in general + [HBASE-4545] - TestHLog doesn't clean up after itself + [HBASE-4772] - Utility to Create StoreFiles + [HBASE-4808] - Test to Ensure Expired Deletes Don't Override Puts + [HBASE-4864] - TestMasterObserver#testRegionTransitionOperations occasionally fails + [HBASE-4868] - TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails + [HBASE-5150] - Failure in a thread may not fail a test, clean up log splitting test + [HBASE-5223] - TestMetaReaderEditor is missing call to CatalogTracker.stop() + [HBASE-5455] - Add test to avoid unintentional reordering of items in HbaseObjectWritable + [HBASE-5792] - HLog Performance Evaluation Tool + Release 0.92.1 - Unreleased BUG FIXES diff --git a/LICENSE.txt b/LICENSE.txt index d64569567334..b94f0c03adf9 100644 --- a/LICENSE.txt +++ b/LICENSE.txt @@ -200,3 +200,72 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. + +---- +This project incorporates portions of the 'Protocol Buffers' project avaialble +under a '3-clause BSD' license. + + Copyright 2008, Google Inc. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following disclaimer + in the documentation and/or other materials provided with the + distribution. + * Neither the name of Google Inc. nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + Code generated by the Protocol Buffer compiler is owned by the owner + of the input file used when generating it. This code is not + standalone and requires a support library to be linked with it. This + support library is itself covered by the above license. + +-- + +This project incorporates part of the 'FreeBSD Documentation Project' +available under a BSD-style license. + + * Copyright (c) 2001, 2003, 2010 The FreeBSD Documentation Project + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD: doc/share/misc/docbook.css,v 1.15 2010/03/20 04:15:01 hrs Exp $ diff --git a/NOTICE.txt b/NOTICE.txt index 3ae710800cd6..dcaa169338be 100644 --- a/NOTICE.txt +++ b/NOTICE.txt @@ -1,16 +1,12 @@ -This product includes software developed by The Apache Software -Foundation (http://www.apache.org/). +Apache HBase +Copyright 2007-2015 The Apache Software Foundation -In addition, this product includes software developed by: +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). -JUnit (http://www.junit.org/) included under the Common Public License v1.0. See -the full text here: http://junit.sourceforge.net/cpl-v10.html +-- +This product incorporates portions of the 'Hadoop' project -JRuby (http://jruby.org) is tri-licensed. We include it under terms of the -Common Public License v1.0. +Copyright 2007-2009 The Apache Software Foundation -JRuby itself includes libraries variously licensed. See its COPYING document -for details: https://github.com/jruby/jruby/blob/master/COPYING - -The JRuby community went out of their way to make JRuby compatible with Apache -projects: See https://issues.apache.org/jira/browse/HBASE-3374) +Licensed under the Apache License v2.0 diff --git a/bin/copy_table.rb b/bin/copy_table.rb deleted file mode 100644 index ad6e70321b82..000000000000 --- a/bin/copy_table.rb +++ /dev/null @@ -1,166 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# Script that copies table in hbase. As written, will not work for rare -# case where there is more than one region in .META. table. Does the -# update of the hbase .META. and copies the directories in filesystem. -# HBase MUST be shutdown when you run this script. -# -# To see usage for this script, run: -# -# ${HBASE_HOME}/bin/hbase org.jruby.Main rename_table.rb -# -include Java -import org.apache.hadoop.hbase.util.MetaUtils -import org.apache.hadoop.hbase.util.FSUtils -import org.apache.hadoop.hbase.util.Bytes -import org.apache.hadoop.hbase.util.Writables -import org.apache.hadoop.hbase.HConstants -import org.apache.hadoop.hbase.HBaseConfiguration -import org.apache.hadoop.hbase.HStoreKey -import org.apache.hadoop.hbase.HRegionInfo -import org.apache.hadoop.hbase.HTableDescriptor -import org.apache.hadoop.hbase.io.ImmutableBytesWritable -import org.apache.hadoop.hbase.regionserver.HLogEdit -import org.apache.hadoop.hbase.regionserver.HRegion -import org.apache.hadoop.fs.Path -import org.apache.hadoop.fs.FileSystem -import org.apache.hadoop.fs.FileUtil -import org.apache.commons.logging.Log -import org.apache.commons.logging.LogFactory -import java.util.TreeMap - -# Name of this script -NAME = "copy_table" - -# Print usage for this script -def usage - puts 'Usage: %s.rb ' % NAME - exit! -end - -# Passed 'dir' exists and is a directory else exception -def isDirExists(fs, dir) - raise IOError.new("Does not exit: " + dir.toString()) unless fs.exists(dir) - raise IOError.new("Not a directory: " + dir.toString()) unless fs.isDirectory(dir) -end - -# Returns true if the region belongs to passed table -def isTableRegion(tableName, hri) - return Bytes.equals(hri.getTableDesc().getName(), tableName) -end - -# Create new HRI based off passed 'oldHRI' -def createHRI(tableName, oldHRI) - htd = oldHRI.getTableDesc() - newHtd = HTableDescriptor.new(tableName) - for family in htd.getFamilies() - newHtd.addFamily(family) - end - return HRegionInfo.new(newHtd, oldHRI.getStartKey(), oldHRI.getEndKey(), - oldHRI.isSplit()) -end - -# Check arguments -if ARGV.size != 2 - usage -end - -# Check good table names were passed. -oldTableName = HTableDescriptor.isLegalTableName(ARGV[0].to_java_bytes) -newTableName = HTableDescriptor.isLegalTableName(ARGV[1].to_java_bytes) - -# Get configuration to use. -c = HBaseConfiguration.new() - -# Set hadoop filesystem configuration using the hbase.rootdir. -# Otherwise, we'll always use localhost though the hbase.rootdir -# might be pointing at hdfs location. -c.set("fs.default.name", c.get(HConstants::HBASE_DIR)) -fs = FileSystem.get(c) - -# If new table directory does not exit, create it. Keep going if already -# exists because maybe we are rerunning script because it failed first -# time. -rootdir = FSUtils.getRootDir(c) -oldTableDir = Path.new(rootdir, Path.new(Bytes.toString(oldTableName))) -isDirExists(fs, oldTableDir) -newTableDir = Path.new(rootdir, Bytes.toString(newTableName)) -if !fs.exists(newTableDir) - fs.mkdirs(newTableDir) -end - -# Get a logger and a metautils instance. -LOG = LogFactory.getLog(NAME) -utils = MetaUtils.new(c) - -# Start. Get all meta rows. -begin - # Get list of all .META. regions that contain old table name - metas = utils.getMETARows(oldTableName) - index = 0 - for meta in metas - # For each row we find, move its region from old to new table. - # Need to update the encoded name in the hri as we move. - # After move, delete old entry and create a new. - LOG.info("Scanning " + meta.getRegionNameAsString()) - metaRegion = utils.getMetaRegion(meta) - scanner = metaRegion.getScanner(HConstants::COL_REGIONINFO_ARRAY, oldTableName, - HConstants::LATEST_TIMESTAMP, nil) - begin - key = HStoreKey.new() - value = TreeMap.new(Bytes.BYTES_COMPARATOR) - while scanner.next(key, value) - index = index + 1 - keyStr = key.toString() - oldHRI = Writables.getHRegionInfo(value.get(HConstants::COL_REGIONINFO)) - if !oldHRI - raise IOError.new(index.to_s + " HRegionInfo is null for " + keyStr) - end - unless isTableRegion(oldTableName, oldHRI) - # If here, we passed out the table. Break. - break - end - oldRDir = Path.new(oldTableDir, Path.new(oldHRI.getEncodedName().to_s)) - if !fs.exists(oldRDir) - LOG.warn(oldRDir.toString() + " does not exist -- region " + - oldHRI.getRegionNameAsString()) - else - # Now make a new HRegionInfo to add to .META. for the new region. - newHRI = createHRI(newTableName, oldHRI) - newRDir = Path.new(newTableDir, Path.new(newHRI.getEncodedName().to_s)) - # Move the region in filesystem - LOG.info("Copying " + oldRDir.toString() + " as " + newRDir.toString()) - FileUtil.copy(fs, oldRDir, fs, newRDir, false, true, c) - # Create 'new' region - newR = HRegion.new(rootdir, utils.getLog(), fs, c, newHRI, nil) - # Add new row. NOTE: Presumption is that only one .META. region. If not, - # need to do the work to figure proper region to add this new region to. - LOG.info("Adding to meta: " + newR.toString()) - HRegion.addRegionToMETA(metaRegion, newR) - LOG.info("Done copying: " + Bytes.toString(key.getRow())) - end - # Need to clear value else we keep appending values. - value.clear() - end - ensure - scanner.close() - end - end -ensure - utils.shutdown() -end diff --git a/bin/get-active-master.rb b/bin/get-active-master.rb index 8887a4574c14..6855cfcd7c40 100644 --- a/bin/get-active-master.rb +++ b/bin/get-active-master.rb @@ -1,6 +1,4 @@ #!/usr/bin/env hbase-jruby -# Copyright 2011 The Apache Software Foundation -# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with this # work for additional information regarding copyright ownership. The ASF diff --git a/bin/graceful_stop.sh b/bin/graceful_stop.sh old mode 100644 new mode 100755 index cf7bee86ff1d..80461a593851 --- a/bin/graceful_stop.sh +++ b/bin/graceful_stop.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2011 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -19,16 +17,16 @@ # * See the License for the specific language governing permissions and # * limitations under the License. # */ - + # Move regions off a server then stop it. Optionally restart and reload. # Turn off the balancer before running this script. function usage { - echo "Usage: graceful_stop.sh [--config ] [--restart] [--reload] [--thrift] [--rest] " + echo "Usage: graceful_stop.sh [--config ] [--restart [--reload]] [--thrift] [--rest] " echo " thrift If we should stop/start thrift before/after the hbase stop/start" echo " rest If we should stop/start rest before/after the hbase stop/start" echo " restart If we should restart after graceful stop" - echo " reload Move offloaded regions back on to the stopped server" - echo " debug Move offloaded regions back on to the stopped server" + echo " reload Move offloaded regions back on to the restarted server" + echo " debug Print helpful debug information" echo " hostname Hostname of server we are to stop" exit 1 } @@ -69,8 +67,9 @@ fi hostname=$1 filename="/tmp/$hostname" # Run the region mover script. -echo "Disabling balancer!" -echo 'balance_switch false' | "$bin"/hbase --config ${HBASE_CONF_DIR} shell +echo "Disabling balancer! (if required)" +HBASE_BALANCER_STATE=`echo 'balance_switch false' | "$bin"/hbase --config ${HBASE_CONF_DIR} shell | tail -3 | head -1` +echo "Previous balancer state was $HBASE_BALANCER_STATE" echo "Unloading $hostname region(s)" HBASE_NOEXEC=true "$bin"/hbase --config ${HBASE_CONF_DIR} org.jruby.Main "$bin"/region_mover.rb --file=$filename $debug unload $hostname echo "Unloaded $hostname region(s)" @@ -100,5 +99,10 @@ if [ "$restart" != "" ]; then fi fi +if [ $HBASE_BALANCER_STATE != "false" ]; then + echo "Restoring balancer state to" $HBASE_BALANCER_STATE + echo "balance_switch $HBASE_BALANCER_STATE" | "$bin"/hbase --config ${HBASE_CONF_DIR} shell &> /dev/null +fi + # Cleanup tmp files. trap "rm -f "/tmp/$(basename $0).*.tmp" &> /dev/null" EXIT diff --git a/bin/hbase b/bin/hbase index c5692ffb746b..158c7c8fd475 100755 --- a/bin/hbase +++ b/bin/hbase @@ -1,8 +1,6 @@ #! /usr/bin/env bash # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -31,11 +29,14 @@ # # HBASE_CLASSPATH Extra Java CLASSPATH entries. # +# HBASE_CLASSPATH_PREFIX Extra Java CLASSPATH entries that should be +# prefixed to the system classpath. +# # HBASE_HEAPSIZE The maximum amount of heap to use, in MB. # Default is 1000. # # HBASE_LIBRARY_PATH HBase additions to JAVA_LIBRARY_PATH for adding -# native libaries. +# native libraries. # # HBASE_OPTS Extra Java runtime options. # @@ -91,6 +92,7 @@ if [ $# = 0 ]; then echo "" echo "PACKAGE MANAGEMENT" echo " classpath dump hbase CLASSPATH" + echo " mapredcp dump CLASSPATH entries required by mapreduce" echo " version print the version" echo "" echo " or" @@ -186,11 +188,6 @@ for f in $HBASE_HOME/lib/*.jar; do CLASSPATH=${CLASSPATH}:$f; done -# Add user-specified CLASSPATH last -if [ "$HBASE_CLASSPATH" != "" ]; then - CLASSPATH=${CLASSPATH}:${HBASE_CLASSPATH} -fi - # default log directory & file if [ "$HBASE_LOG_DIR" = "" ]; then HBASE_LOG_DIR="$HBASE_HOME/logs" @@ -199,13 +196,6 @@ if [ "$HBASE_LOGFILE" = "" ]; then HBASE_LOGFILE='hbase.log' fi -# cygwin path translation -if $cygwin; then - CLASSPATH=`cygpath -p -w "$CLASSPATH"` - HBASE_HOME=`cygpath -d "$HBASE_HOME"` - HBASE_LOG_DIR=`cygpath -d "$HBASE_LOG_DIR"` -fi - function append_path() { if [ -z "$1" ]; then echo $2 @@ -217,31 +207,31 @@ function append_path() { JAVA_PLATFORM="" #If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH -if [ ! -z $HADOOP_HOME ]; then - HADOOPCPPATH="" - if [ -z $HADOOP_CONF_DIR ]; then - HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_HOME}/conf") - else - HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_CONF_DIR}") - fi - if [ "`echo ${HADOOP_HOME}/hadoop-core*.jar`" != "${HADOOP_HOME}/hadoop-core*.jar" ] ; then - HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-core*.jar | head -1`) - else - HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-common*.jar | head -1`) - HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-hdfs*.jar | head -1`) - HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-mapred*.jar | head -1`) - fi - for i in "${HADOOP_HOME}/lib/"*.jar; do - HADOOPCPPATH="${HADOOPCPPATH}:$i" - done - CLASSPATH=$(append_path "${CLASSPATH}" "${HADOOPCPPATH}") - - if [ -d "${HADOOP_HOME}/lib/native" ]; then - JAVA_PLATFORM=`CLASSPATH=${HADOOPCPPATH} ${JAVA} org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"` - if [ -d "${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}" ]; then - JAVA_LIBRARY_PATH=$(append_path "${JAVA_LIBRARY_PATH}" "${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}") - fi +HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH" which hadoop 2>/dev/null) +if [ -f ${HADOOP_IN_PATH} ]; then + HADOOP_JAVA_LIBRARY_PATH=$(HADOOP_CLASSPATH="$CLASSPATH" ${HADOOP_IN_PATH} \ + org.apache.hadoop.hbase.util.GetJavaProperty java.library.path 2>/dev/null) + if [ -n "$HADOOP_JAVA_LIBRARY_PATH" ]; then + JAVA_LIBRARY_PATH=$(append_path "${JAVA_LIBRARY_PATH}" "$HADOOP_JAVA_LIBRARY_PATH") fi + CLASSPATH=$(append_path "${CLASSPATH}" `${HADOOP_IN_PATH} classpath 2>/dev/null`) +fi + +# Add user-specified CLASSPATH last +if [ "$HBASE_CLASSPATH" != "" ]; then + CLASSPATH=${CLASSPATH}:${HBASE_CLASSPATH} +fi + +# Add user-specified CLASSPATH prefix first +if [ "$HBASE_CLASSPATH_PREFIX" != "" ]; then + CLASSPATH=${HBASE_CLASSPATH_PREFIX}:${CLASSPATH} +fi + +# cygwin path translation +if $cygwin; then + CLASSPATH=`cygpath -p -w "$CLASSPATH"` + HBASE_HOME=`cygpath -d "$HBASE_HOME"` + HBASE_LOG_DIR=`cygpath -d "$HBASE_LOG_DIR"` fi if [ -d "${HBASE_HOME}/build/native" -o -d "${HBASE_HOME}/lib/native" ]; then @@ -265,6 +255,21 @@ fi # restore ordinary behaviour unset IFS +#Set the right GC options based on the what we are running +declare -a server_cmds=("master" "regionserver" "thrift" "thrift2" "rest" "avro" "zookeeper") +for cmd in ${server_cmds[@]}; do + if [[ $cmd == $COMMAND ]]; then + server=true + break + fi +done + +if [[ $server ]]; then + HBASE_OPTS="$HBASE_OPTS $SERVER_GC_OPTS" +else + HBASE_OPTS="$HBASE_OPTS $CLIENT_GC_OPTS" +fi + # figure out which class to run if [ "$COMMAND" = "shell" ] ; then # eg export JRUBY_HOME=/usr/local/share/jruby @@ -272,6 +277,7 @@ if [ "$COMMAND" = "shell" ] ; then CLASSPATH="$JRUBY_HOME/lib/jruby.jar:$CLASSPATH" HBASE_OPTS="$HBASE_OPTS -Djruby.home=$JRUBY_HOME -Djruby.lib=$JRUBY_HOME/lib" fi + HBASE_OPTS="$HBASE_OPTS $HBASE_SHELL_OPTS" CLASS="org.jruby.Main -X+O ${JRUBY_OPTS} ${HBASE_HOME}/bin/hirb.rb" elif [ "$COMMAND" = "hbck" ] ; then CLASS='org.apache.hadoop.hbase.util.HBaseFsck' @@ -319,7 +325,8 @@ elif [ "$COMMAND" = "zookeeper" ] ; then if [ "$1" != "stop" ] ; then HBASE_OPTS="$HBASE_OPTS $HBASE_ZOOKEEPER_OPTS" fi - +elif [ "$COMMAND" = "mapredcp" ] ; then + CLASS='org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool' elif [ "$COMMAND" = "classpath" ] ; then echo $CLASSPATH exit 0 @@ -340,6 +347,14 @@ HBASE_OPTS="$HBASE_OPTS -Dhbase.id.str=$HBASE_IDENT_STRING" HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger=${HBASE_ROOT_LOGGER:-INFO,console}" if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then HBASE_OPTS="$HBASE_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH" + export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH" +fi + +# Enable security logging on the master and regionserver only +if [ "$COMMAND" = "master" ] || [ "$COMMAND" = "regionserver" ]; then + HBASE_OPTS="$HBASE_OPTS -Dhbase.security.logger=${HBASE_SECURITY_LOGGER:-INFO,DRFAS}" +else + HBASE_OPTS="$HBASE_OPTS -Dhbase.security.logger=${HBASE_SECURITY_LOGGER:-INFO,NullAppender}" fi # Exec unless HBASE_NOEXEC is set. diff --git a/bin/hbase-config.sh b/bin/hbase-config.sh index 63b4f058eb68..0137db9219dd 100644 --- a/bin/hbase-config.sh +++ b/bin/hbase-config.sh @@ -1,7 +1,5 @@ # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -77,10 +75,16 @@ HBASE_REGIONSERVERS="${HBASE_REGIONSERVERS:-$HBASE_CONF_DIR/regionservers}" HBASE_BACKUP_MASTERS="${HBASE_BACKUP_MASTERS:-$HBASE_CONF_DIR/backup-masters}" # Source the hbase-env.sh. Will have JAVA_HOME defined. -if [ -f "${HBASE_CONF_DIR}/hbase-env.sh" ]; then +# HBASE-7817 - Source the hbase-env.sh only if it has not already been done. HBASE_ENV_INIT keeps track of it. +if [ -z "$HBASE_ENV_INIT" ] && [ -f "${HBASE_CONF_DIR}/hbase-env.sh" ]; then . "${HBASE_CONF_DIR}/hbase-env.sh" + export HBASE_ENV_INIT="true" fi +# Newer versions of glibc use an arena memory allocator that causes virtual +# memory usage to explode. Tune the variable down to prevent vmem explosion. +export MALLOC_ARENA_MAX=${MALLOC_ARENA_MAX:-4} + if [ -z "$JAVA_HOME" ]; then for candidate in \ /usr/lib/jvm/java-6-sun \ diff --git a/bin/hbase-daemon.sh b/bin/hbase-daemon.sh index ffae30a49dae..201548a286eb 100755 --- a/bin/hbase-daemon.sh +++ b/bin/hbase-daemon.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -116,14 +114,26 @@ fi JAVA=$JAVA_HOME/bin/java export HBASE_LOG_PREFIX=hbase-$HBASE_IDENT_STRING-$command-$HOSTNAME export HBASE_LOGFILE=$HBASE_LOG_PREFIX.log -export HBASE_ROOT_LOGGER="INFO,DRFA" -logout=$HBASE_LOG_DIR/$HBASE_LOG_PREFIX.out + +if [ -z "${HBASE_ROOT_LOGGER}" ]; then +export HBASE_ROOT_LOGGER=${HBASE_ROOT_LOGGER:-"INFO,DRFA"} +fi + +if [ -z "${HBASE_SECURITY_LOGGER}" ]; then +export HBASE_SECURITY_LOGGER=${HBASE_SECURITY_LOGGER:-"INFO,DRFAS"} +fi + +logout=$HBASE_LOG_DIR/$HBASE_LOG_PREFIX.out + loggc=$HBASE_LOG_DIR/$HBASE_LOG_PREFIX.gc loglog="${HBASE_LOG_DIR}/${HBASE_LOGFILE}" pid=$HBASE_PID_DIR/hbase-$HBASE_IDENT_STRING-$command.pid -if [ "$HBASE_USE_GC_LOGFILE" = "true" ]; then - export HBASE_GC_OPTS=" -Xloggc:${loggc}" +if [ -n "$SERVER_GC_OPTS" ]; then + export SERVER_GC_OPTS=${SERVER_GC_OPTS/"-Xloggc:"/"-Xloggc:${loggc}"} +fi +if [ -n "$CLIENT_GC_OPTS" ]; then + export CLIENT_GC_OPTS=${CLIENT_GC_OPTS/"-Xloggc:"/"-Xloggc:${loggc}"} fi # Set default scheduling priority diff --git a/bin/hbase-daemons.sh b/bin/hbase-daemons.sh index 843eaaa74ffe..21ce635777d9 100755 --- a/bin/hbase-daemons.sh +++ b/bin/hbase-daemons.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information diff --git a/bin/hirb.rb b/bin/hirb.rb index 32a51b3bf2f7..c8f13b5703d4 100644 --- a/bin/hirb.rb +++ b/bin/hirb.rb @@ -1,6 +1,4 @@ # -# Copyright 2009 The Apache Software Foundation -# # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information diff --git a/bin/local-master-backup.sh b/bin/local-master-backup.sh old mode 100644 new mode 100755 index 2c0a4c02c76d..c945e2b39646 --- a/bin/local-master-backup.sh +++ b/bin/local-master-backup.sh @@ -1,7 +1,5 @@ -#!/bin/sh +#!/usr/bin/env bash #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information diff --git a/bin/local-regionservers.sh b/bin/local-regionservers.sh old mode 100644 new mode 100755 index a4d5a1d93211..29adcf396a33 --- a/bin/local-regionservers.sh +++ b/bin/local-regionservers.sh @@ -1,7 +1,5 @@ -#!/bin/sh +#!/usr/bin/env bash #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information diff --git a/bin/master-backup.sh b/bin/master-backup.sh index d20f5793e094..feca4ab86572 100755 --- a/bin/master-backup.sh +++ b/bin/master-backup.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2010 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -25,11 +23,11 @@ # Environment Variables # # HBASE_BACKUP_MASTERS File naming remote hosts. -# Default is ${HADOOP_CONF_DIR}/backup-masters +# Default is ${HBASE_CONF_DIR}/backup-masters # HADOOP_CONF_DIR Alternate conf dir. Default is ${HADOOP_HOME}/conf. # HBASE_CONF_DIR Alternate hbase conf dir. Default is ${HBASE_HOME}/conf. -# HADOOP_SLAVE_SLEEP Seconds to sleep between spawning remote commands. -# HADOOP_SSH_OPTS Options passed to ssh when running remote commands. +# HBASE_SLAVE_SLEEP Seconds to sleep between spawning remote commands. +# HBASE_SSH_OPTS Options passed to ssh when running remote commands. # # Modelled after $HADOOP_HOME/bin/slaves.sh. diff --git a/bin/region_mover.rb b/bin/region_mover.rb index 028f9a89cbe7..1ea785d4f1b3 100644 --- a/bin/region_mover.rb +++ b/bin/region_mover.rb @@ -1,5 +1,3 @@ -# Copyright 2011 The Apache Software Foundation -# # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information @@ -75,12 +73,22 @@ def getTable(config, name) return $TABLES[key] end +def closeTables() + if not $TABLES + return + end + + $LOG.info("Close all tables") + $TABLES.each do |name, table| + $TABLES.delete(name) + table.close() + end +end # Returns true if passed region is still on 'original' when we look at .META. def isSameServer(admin, r, original) server = getServerNameForRegion(admin, r) - return false unless server - return true unless original + return false unless server and original return server == original end @@ -94,6 +102,7 @@ def abort(why, e) # Get servername that is up in .META.; this is hostname + port + startcode comma-delimited. # Can return nil def getServerNameForRegion(admin, r) + return nil unless admin.isTableEnabled(r.getTableName) if r.isRootRegion() # Hack tracker = org.apache.hadoop.hbase.zookeeper.RootRegionTracker.new(admin.getConnection().getZooKeeperWatcher(), RubyAbortable.new()) @@ -116,6 +125,7 @@ def getServerNameForRegion(admin, r) g.addColumn(HConstants::CATALOG_FAMILY, HConstants::SERVER_QUALIFIER) g.addColumn(HConstants::CATALOG_FAMILY, HConstants::STARTCODE_QUALIFIER) result = table.get(g) + return nil unless result server = result.getValue(HConstants::CATALOG_FAMILY, HConstants::SERVER_QUALIFIER) startcode = result.getValue(HConstants::CATALOG_FAMILY, HConstants::STARTCODE_QUALIFIER) return nil unless server @@ -129,15 +139,24 @@ def isSuccessfulScan(admin, r) scan.setBatch(1) scan.setCaching(1) scan.setFilter(FirstKeyOnlyFilter.new()) - table = getTable(admin.getConfiguration(), r.getTableDesc().getName()) - scanner = table.getScanner(scan) + begin + table = getTable(admin.getConfiguration(), r.getTableName()) + scanner = table.getScanner(scan) + rescue org.apache.hadoop.hbase.TableNotFoundException, + org.apache.hadoop.hbase.TableNotEnabledException => e + $LOG.warn("Region " + r.getEncodedName() + " belongs to recently " + + "deleted/disabled table. Skipping... " + e.message) + return + end begin results = scanner.next() # We might scan into next region, this might be an empty table. # But if no exception, presume scanning is working. ensure scanner.close() - table.close() + # Do not close the htable. It is cached in $TABLES and + # may be reused in moving another region of same table. + # table.close() end end @@ -150,6 +169,7 @@ def move(admin, r, newServer, original) retries = admin.getConfiguration.getInt("hbase.move.retries.max", 5) count = 0 same = true + start = Time.now while count < retries and same if count > 0 $LOG.info("Retry " + count.to_s + " of maximum " + retries.to_s) @@ -157,7 +177,8 @@ def move(admin, r, newServer, original) count = count + 1 begin admin.move(Bytes.toBytes(r.getEncodedName()), Bytes.toBytes(newServer)) - rescue java.lang.reflect.UndeclaredThrowableException => e + rescue java.lang.reflect.UndeclaredThrowableException, + org.apache.hadoop.hbase.UnknownRegionException => e $LOG.info("Exception moving " + r.getEncodedName() + "; split/moved? Continuing: " + e) return @@ -174,6 +195,8 @@ def move(admin, r, newServer, original) raise RuntimeError, "Region stuck on #{original}, newserver=#{newServer}" if same # Assert can Scan from new location. isSuccessfulScan(admin, r) + $LOG.info("Moved region " + r.getRegionNameAsString() + " cost: " + + java.lang.String.format("%.3f", (Time.now - start))) end # Return the hostname portion of a servername (all up to first ',') @@ -213,6 +236,16 @@ def stripServer(servers, hostname) return servername end +# Returns a new serverlist that excludes the servername whose hostname portion +# matches from the passed array of servers. +def stripExcludes(servers, excludefile) + excludes = readExcludes(excludefile) + servers = servers.find_all{|server| !excludes.contains(getHostnameFromServerName(server)) } + # return updated servers list + return servers +end + + # Return servername that matches passed hostname def getServerName(servers, hostname) servername = nil @@ -309,9 +342,15 @@ def unloadRegions(options, hostname) # Remove the server we are unloading from from list of servers. # Side-effect is the servername that matches this hostname servername = stripServer(servers, hostname) + + # Remove the servers in our exclude list from list of servers. + servers = stripExcludes(servers, options[:excludesFile]) + puts "Valid region move targets: ", servers movedRegions = java.util.ArrayList.new() while true rs = getRegions(config, servername) + # Remove those already tried to move + rs.removeAll(movedRegions) break if rs.length == 0 count = 0 $LOG.info("Moving " + rs.length.to_s + " region(s) from " + servername + @@ -319,8 +358,9 @@ def unloadRegions(options, hostname) for r in rs # Get a random server to move the region to. server = servers[rand(servers.length)] - $LOG.info("Moving region " + r.getEncodedName() + " (" + count.to_s + - " of " + rs.length.to_s + ") to server=" + server); + $LOG.info("Moving region " + r.getRegionNameAsString() + " (" + + (count + 1).to_s + " of " + rs.length.to_s + ") from server=" + + servername + " to server=" + server); count = count + 1 # Assert we can scan region in its current location isSuccessfulScan(admin, r) @@ -361,10 +401,13 @@ def loadRegions(options, hostname) end $LOG.info("Moving " + regions.size().to_s + " regions to " + servername) count = 0 + # sleep 20s to make sure the rs finished initialization. + sleep 20 for r in regions exists = false begin - exists = isSuccessfulScan(admin, r) + isSuccessfulScan(admin, r) + exists = true rescue org.apache.hadoop.hbase.NotServingRegionException => e $LOG.info("Failed scan of " + e.message) end @@ -376,12 +419,36 @@ def loadRegions(options, hostname) " of " + regions.length.to_s + ") already on target server=" + servername) next end - $LOG.info("Moving region " + r.getEncodedName() + " (" + count.to_s + - " of " + regions.length.to_s + ") to server=" + servername); + $LOG.info("Moving region " + r.getRegionNameAsString() + " (" + + (count + 1).to_s + " of " + regions.length.to_s + ") from server=" + + currentServer.to_s + " to server=" + servername.to_s); move(admin, r, servername, currentServer) end end +# Returns an array of hosts to exclude as region move targets +def readExcludes(filename) + if filename == nil + return java.util.ArrayList.new() + end + if ! File.exist?(filename) + puts "Error: Unable to read host exclude file: ", filename + raise RuntimeError + end + + f = File.new(filename, "r") + # Read excluded hosts list + excludes = java.util.ArrayList.new() + while (line = f.gets) + line.strip! # do an inplace drop of pre and post whitespaces + excludes.add(line) unless line.empty? # exclude empty lines + end + puts "Excluding hosts as region move targets: ", excludes + f.close + + return excludes +end + def getFilename(options, targetServer) filename = options[:file] if not filename @@ -408,6 +475,9 @@ def getFilename(options, targetServer) opts.on('-d', '--debug', 'Display extra debug logging') do options[:debug] = true end + opts.on('-x', '--excludefile=FILE', 'File with hosts-per-line to exclude as unload targets; default excludes only target host; useful for rack decommisioning.') do |file| + options[:excludesFile] = file + end end optparse.parse! @@ -432,3 +502,5 @@ def getFilename(options, targetServer) puts optparse exit 3 end + +closeTables() diff --git a/bin/regionservers.sh b/bin/regionservers.sh index 9759f2b00c41..fc96563b733a 100755 --- a/bin/regionservers.sh +++ b/bin/regionservers.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -28,8 +26,8 @@ # Default is ${HADOOP_CONF_DIR}/regionservers # HADOOP_CONF_DIR Alternate conf dir. Default is ${HADOOP_HOME}/conf. # HBASE_CONF_DIR Alternate hbase conf dir. Default is ${HBASE_HOME}/conf. -# HADOOP_SLAVE_SLEEP Seconds to sleep between spawning remote commands. -# HADOOP_SSH_OPTS Options passed to ssh when running remote commands. +# HBASE_SLAVE_SLEEP Seconds to sleep between spawning remote commands. +# HBASE_SSH_OPTS Options passed to ssh when running remote commands. # # Modelled after $HADOOP_HOME/bin/slaves.sh. diff --git a/bin/replication/copy_tables_desc.rb b/bin/replication/copy_tables_desc.rb index ed85655933fe..6d4e9c8fc452 100644 --- a/bin/replication/copy_tables_desc.rb +++ b/bin/replication/copy_tables_desc.rb @@ -1,6 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation -# # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information @@ -32,7 +30,6 @@ import org.apache.hadoop.hbase.client.HBaseAdmin import org.apache.hadoop.hbase.HTableDescriptor import org.apache.hadoop.conf.Configuration -import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper # Name of this script NAME = "copy_tables_desc" diff --git a/bin/rolling-restart.sh b/bin/rolling-restart.sh index 8c3cc2bf5731..362b29cd0c1c 100755 --- a/bin/rolling-restart.sh +++ b/bin/rolling-restart.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -28,13 +26,13 @@ # Default is ${HADOOP_CONF_DIR}/regionservers # HADOOP_CONF_DIR Alternate conf dir. Default is ${HADOOP_HOME}/conf. # HBASE_CONF_DIR Alternate hbase conf dir. Default is ${HBASE_HOME}/conf. -# HADOOP_SLAVE_SLEEP Seconds to sleep between spawning remote commands. -# HADOOP_SLAVE_TIMEOUT Seconds to wait for timing out a remote command. -# HADOOP_SSH_OPTS Options passed to ssh when running remote commands. +# HBASE_SLAVE_SLEEP Seconds to sleep between spawning remote commands. +# HBASE_SLAVE_TIMEOUT Seconds to wait for timing out a remote command. +# HBASE_SSH_OPTS Options passed to ssh when running remote commands. # # Modelled after $HADOOP_HOME/bin/slaves.sh. -usage="Usage: $0 [--config ] [--rs-only] [--master-only]" +usage="Usage: $0 [--config ] [--rs-only] [--master-only] [--graceful]" bin=`dirname "$0"` bin=`cd "$bin">/dev/null; pwd` @@ -56,16 +54,24 @@ function usage() { RR_RS=1 RR_MASTER=1 +RR_GRACEFUL=0 for x in "$@" ; do case "$x" in --rs-only|-r) RR_RS=1 RR_MASTER=0 + RR_GRACEFUL=0 ;; --master-only) RR_RS=0 RR_MASTER=1 + RR_GRACEFUL=0 + ;; + --graceful) + RR_RS=0 + RR_MASTER=0 + RR_GRACEFUL=1 ;; *) echo Bad argument: $x @@ -76,7 +82,8 @@ for x in "$@" ; do done # quick function to get a value from the HBase config file -distMode=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed` +# HBASE-6504 - only take the first line of the output in case verbose gc is on +distMode=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed | head -n 1` if [ "$distMode" == 'false' ]; then if [ $RR_RS -ne 1 ] || [ $RR_MASTER -ne 1 ]; then echo Cant do selective rolling restart if not running distributed @@ -84,6 +91,9 @@ if [ "$distMode" == 'false' ]; then fi "$bin"/hbase-daemon.sh restart master else + zparent=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool zookeeper.znode.parent` + if [ "$zparent" == "null" ]; then zparent="/hbase"; fi + if [ $RR_MASTER -eq 1 ]; then # stop all masters before re-start to avoid races for master znode "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}" stop master @@ -91,13 +101,11 @@ else --hosts "${HBASE_BACKUP_MASTERS}" stop master-backup # make sure the master znode has been deleted before continuing - zparent=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool zookeeper.znode.parent` - if [ "$zparent" == "null" ]; then zparent="/hbase"; fi zmaster=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool zookeeper.znode.master` if [ "$zmaster" == "null" ]; then zmaster="master"; fi zmaster=$zparent/$zmaster echo -n "Waiting for Master ZNode ${zmaster} to expire" - while bin/hbase zkcli stat $zmaster >/dev/null 2>&1; do + while ! "$bin"/hbase zkcli stat $zmaster 2>&1 | grep "Node does not exist"; do echo -n "." sleep 1 done @@ -136,4 +144,20 @@ else "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" \ --hosts "${HBASE_REGIONSERVERS}" restart regionserver fi + + if [ $RR_GRACEFUL -eq 1 ]; then + # gracefully restart all online regionservers + zkrs=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool zookeeper.znode.rs` + if [ "$zkrs" == "null" ]; then zkrs="rs"; fi + zkrs="$zparent/$zkrs" + online_regionservers=`$bin/hbase zkcli ls $zkrs 2>&1 | tail -1 | sed "s/\[//" | sed "s/\]//"` + for rs in $online_regionservers + do + rs_parts=(${rs//,/ }) + hostname=${rs_parts[0]} + echo "Gracefully restarting: $hostname" + "$bin"/graceful_stop.sh --config "${HBASE_CONF_DIR}" --restart --reload --debug "$hostname" + sleep 1 + done + fi fi diff --git a/bin/start-hbase.sh b/bin/start-hbase.sh index 6240ee649abf..aed729a808bf 100755 --- a/bin/start-hbase.sh +++ b/bin/start-hbase.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -38,12 +36,13 @@ then exit $errCode fi -distMode=`$bin/hbase --config "$HBASE_CONF_DIR" org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed` +# HBASE-6504 - only take the first line of the output in case verbose gc is on +distMode=`$bin/hbase --config "$HBASE_CONF_DIR" org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed | head -n 1` if [ "$distMode" == 'false' ] then - "$bin"/hbase-daemon.sh start master + "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}" start master else "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" start zookeeper "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}" start master diff --git a/bin/stop-hbase.sh b/bin/stop-hbase.sh index b3828345d51d..5b2e69d6f21e 100755 --- a/bin/stop-hbase.sh +++ b/bin/stop-hbase.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -60,7 +58,8 @@ done echo # distributed == false means that the HMaster will kill ZK when it exits -distMode=`$bin/hbase --config "$HBASE_CONF_DIR" org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed` +# HBASE-6504 - only take the first line of the output in case verbose gc is on +distMode=`$bin/hbase --config "$HBASE_CONF_DIR" org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed | head -n 1` if [ "$distMode" == 'true' ] then # TODO: store backup masters in ZooKeeper and have the primary send them a shutdown message diff --git a/bin/zookeepers.sh b/bin/zookeepers.sh index 89a214e5a809..97bf41b60528 100755 --- a/bin/zookeepers.sh +++ b/bin/zookeepers.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2009 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information diff --git a/conf/hadoop-metrics.properties b/conf/hadoop-metrics.properties index 046a369524da..4eb70a6911d3 100644 --- a/conf/hadoop-metrics.properties +++ b/conf/hadoop-metrics.properties @@ -1,3 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + # See http://wiki.apache.org/hadoop/GangliaMetrics # Make sure you know whether you are using ganglia 3.0 or 3.1. # If 3.1, you will have to patch your hadoop instance with HADOOP-4675 @@ -7,13 +23,18 @@ # for the moment. # # See also http://hadoop.apache.org/hbase/docs/current/metrics.html +# GMETADHOST_IP is the hostname (or) IP address of the server on which the ganglia +# meta daemon (gmetad) service is running -# Configuration of the "hbase" context for null -hbase.class=org.apache.hadoop.metrics.spi.NullContext +# Configuration of the "hbase" context for NullContextWithUpdateThread +# NullContextWithUpdateThread is a null context which has a thread calling +# periodically when monitoring is started. This keeps the data sampled +# correctly. +hbase.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread +hbase.period=10 # Configuration of the "hbase" context for file # hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext -# hbase.period=10 # hbase.fileName=/tmp/metrics_hbase.log # HBase-specific configuration to reset long-running stats (e.g. compactions) @@ -28,11 +49,11 @@ hbase.extendedperiod = 3600 # hbase.servers=GMETADHOST_IP:8649 # Configuration of the "jvm" context for null -jvm.class=org.apache.hadoop.metrics.spi.NullContext +jvm.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread +jvm.period=10 # Configuration of the "jvm" context for file # jvm.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext -# jvm.period=10 # jvm.fileName=/tmp/metrics_jvm.log # Configuration of the "jvm" context for ganglia @@ -43,11 +64,11 @@ jvm.class=org.apache.hadoop.metrics.spi.NullContext # jvm.servers=GMETADHOST_IP:8649 # Configuration of the "rpc" context for null -rpc.class=org.apache.hadoop.metrics.spi.NullContext +rpc.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread +rpc.period=10 # Configuration of the "rpc" context for file # rpc.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext -# rpc.period=10 # rpc.fileName=/tmp/metrics_rpc.log # Configuration of the "rpc" context for ganglia @@ -56,3 +77,10 @@ rpc.class=org.apache.hadoop.metrics.spi.NullContext # rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 # rpc.period=10 # rpc.servers=GMETADHOST_IP:8649 + +# Configuration of the "rest" context for ganglia +# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter) +# rest.class=org.apache.hadoop.metrics.ganglia.GangliaContext +# rest.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 +# rest.period=10 +# rest.servers=GMETADHOST_IP:8649 diff --git a/conf/hbase-env.sh b/conf/hbase-env.sh index bc293b4b52f0..39f3b74e54bb 100644 --- a/conf/hbase-env.sh +++ b/conf/hbase-env.sh @@ -1,7 +1,5 @@ # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -21,6 +19,10 @@ # Set environment variables here. +# This script sets variables multiple times over the course of starting an hbase process, +# so try to keep things idempotent unless you want to take an even deeper look +# into the startup scripts (bin/hbase, etc.) + # The java implementation to use. Java 1.6 required. # export JAVA_HOME=/usr/java/jdk1.6.0/ @@ -36,12 +38,31 @@ # see http://wiki.apache.org/hadoop/PerformanceTuning export HBASE_OPTS="-XX:+UseConcMarkSweepGC" -# Uncomment below to enable java garbage collection logging in the .out file. -# export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps $HBASE_GC_OPTS" +# Uncomment one of the below three options to enable java garbage collection logging for the server-side processes. + +# This enables basic gc logging to the .out file. +# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps" + +# This enables basic gc logging to its own file. +# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . +# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:" -# Uncomment below (along with above GC logging) to put GC information in its own logfile (will set HBASE_GC_OPTS) -# export HBASE_USE_GC_LOGFILE=true +# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+. +# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . +# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc: -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M" +# Uncomment one of the below three options to enable java garbage collection logging for the client processes. + +# This enables basic gc logging to the .out file. +# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps" + +# This enables basic gc logging to its own file. +# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . +# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:" + +# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+. +# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . +# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc: -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M" # Uncomment below if you intend to use the EXPERIMENTAL off heap cache. # export HBASE_OPTS="$HBASE_OPTS -XX:MaxDirectMemorySize=" @@ -61,6 +82,9 @@ export HBASE_OPTS="-XX:+UseConcMarkSweepGC" # File naming hosts on which HRegionServers will run. $HBASE_HOME/conf/regionservers by default. # export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers +# File naming hosts on which backup HMaster will run. $HBASE_HOME/conf/backup-masters by default. +# export HBASE_BACKUP_MASTERS=${HBASE_HOME}/conf/backup-masters + # Extra ssh options. Empty by default. # export HBASE_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR" diff --git a/conf/hbase-site.xml b/conf/hbase-site.xml index af4c30095216..3ecd24c2cf4d 100644 --- a/conf/hbase-site.xml +++ b/conf/hbase-site.xml @@ -2,8 +2,6 @@ + + + + +Creates a report in the directory "hbase_jdiff_report-p-PREVIOUS_BRANCH-c-CURRENT_BRANCH" of the default jdiff report folder. +This defaults to /tmp/jdiff but can optionally be specified by export $JDIFF_WORKING_DIRECTORY. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev-support/hbase_jdiff_afterSingularityTemplate.xml b/dev-support/hbase_jdiff_afterSingularityTemplate.xml new file mode 100644 index 000000000000..6c4cd9355202 --- /dev/null +++ b/dev-support/hbase_jdiff_afterSingularityTemplate.xml @@ -0,0 +1,66 @@ + + + + + + +Creates a report in the directory "hbase_jdiff_report-p-PREVIOUS_BRANCH-c-CURRENT_BRANCH" of the default jdiff report folder. +This defaults to /tmp/jdiff but can optionally be specified by export $JDIFF_WORKING_DIRECTORY. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev-support/hbase_jdiff_template.xml b/dev-support/hbase_jdiff_template.xml new file mode 100644 index 000000000000..21fe8ed1b299 --- /dev/null +++ b/dev-support/hbase_jdiff_template.xml @@ -0,0 +1,53 @@ + + + + + + +Creates a report in the directory "hbase_jdiff_report-p-PREVIOUS_BRANCH-c-CURRENT_BRANCH" of the default jdiff report folder. +This defaults to /tmp/jdiff but can optionally be specified by export $JDIFF_WORKING_DIRECTORY. + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev-support/jdiffHBasePublicAPI.sh b/dev-support/jdiffHBasePublicAPI.sh new file mode 100644 index 000000000000..2000d2abd945 --- /dev/null +++ b/dev-support/jdiffHBasePublicAPI.sh @@ -0,0 +1,249 @@ +#!/bin/bash +set -e + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +################################################ ABOUT JDIFF ####################################################### +# +# What is JDiff? JDiff is a tool for comparing the public APIs of two separate Java codebases. Like diff, it will +# give additions, changes, and removals. It will output an HTML report with the information. +# To learn more, visit http://javadiff.sourceforge.net/. +# JDiff is licensed under LGPL. + +############################################# QUICK-START EXAMPLE ################################################## +# +# Suppose we wanted to see the API diffs between HBase 0.92 and HBase 0.94. We could use this tool like so: +# > ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.92 https://github.com/apache/hbase.git 0.94 +# +# This would generate a report in the local folder /tmp/jdiff/hbase_jdiff_report-p-0.92-c-0.94/ +# To view the report, simply examine /tmp/jdiff/hbase_jdiff_report-p-0.92-c-0.94/changes.html in your choice of +# browser. +# +# Note that this works because 0.92 and 0.94 have the source directory structure that is specified in the +# hbase_jdiff_template.xml file. To compare 0.95 to 0.96, which have the post-singularity structure, two other +# template files (included) are used. The formats are autoated and is all taken care of automatically by the script. +# +# On a local machine, JDiff reports have taken ~20-30 minutes to run. On Jenkins, it has taken over 35 minutes +# in some cases. Your mileage may vary. Trunk and 0.95 take more time than 0.92 and 0.94. +# +# +############################################ SPECIFYING A LOCAL REPO ############################################### +# +# The JDiff tool also works with local code. Instead of specifying a repo and a branch, you can specifying the +# absolute path of the ./hbase folder and a name for code (e.g. experimental_94). +# +# A local repo can be specified for none, one, or both of the sources. +# +############################################### EXAMPLE USE CASES ################################################## +# +# Example 1: Generate a report to check if potential change doesn't break API compatibility with Apache HBase 0.94 +# +# In this case, you could compare the version you are using against a repo branch where your changes are. +# > ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.94 https://github.com/MY_REPO/hbase.git 0.94 +# +# Example 2: Generate a report to check if two branches of the same repo have any public API incompatibilities +# > ./jdiffHBasePublicAPI.sh https://github.com/MY_REPO/hbase.git $BRANCH_1 \ +# > https://github.com/MY_REPO/hbase.git $BRANCH_2 +# +# Example 3: Have Example 1 done in a special directory in the user's home folder +# +# > export JDIFF_WORKING_DIRECTORY=~/jdiff_reports +# > ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.94 https://github.com/MY_REPO/hbase.git 0.94 +# +# Example 4: Check the API diff of a local change against an existing repo branch. +# > ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.95 /home/aleks/exp_hbase/hbase experiment_95 +# +# Example 5: Compare two local repos for public API changes +# > ./jdiffHBasePublicAPI.sh /home/aleks/stable_hbase/hbase stable_95 /home/aleks/exp_hbase/hbase experiment_95 +# +# +################################################## NOTE ON USAGE ################################################### +# +# 1. When using this tool, please specify the initial version first and the current version second. The semantics +# do not make sense otherwise. For example: jdiff 94 95 is good. jdiff 95 94 is bad +# +############################################# READING A JDIFF REPORT ############################################### +# +# The purpose of the JDiff report is show things that have changed between two versions of the public API. A user +# would use this report to determine if committing a change would cause existing API clients to break. To do so, +# there are specific things that one should look for in the report. +# +# 1. Identify the classes that constitute the public API. An example in 0.94 might be all classes in +# org.apache.hadoop.hbase.client.* +# 2. After identifying those classes, go through each one and look for offending changes. +# Those may include, but are not limited to: +# 1. Removed methods +# 2. Changed methods (including changes in return type and exception types) +# 3. Methods added to interfaces +# 4. Changed class inheritence information (may in innocuous but definitely worth validating) +# 5. Removed or renamed public static member variables and constants +# 6. Removed or renamed packages +# 7. Class moved to a different package + +########################################### SETTING THE JDIFF WORKING DIRECTORY #################################### +# +# By default, the working environment of jdiff is /tmp/jdiff. However, sometimes it is nice to have it place reports +# and temp files elsewhere. In that case, please export JDIFF_WORKING_DIRECTORY into the bash environment and this +# script will pick that up and use it. +# + +scriptDirectory=$(dirname ${BASH_SOURCE[0]}) +x=`echo $scriptDirectory | sed "s{\.{{g"` +DEV_SUPPORT_HOME="`pwd`$x" +. $scriptDirectory/jdiffHBasePublicAPI_common.sh + +EXPECTED_ARGS=4 + +if [[ "$#" -ne "$EXPECTED_ARGS" ]]; then + echo "This tool expects $EXPECTED_ARGS arguments, but received $#. Please check your command and try again."; + echo "Usage: $0 " + exit 1; +fi + +echo "JDiff evaluation beginning:"; +isGitRepo $1 +FIRST_SOURCE_TYPE=$INPUT_FORMAT; +isGitRepo $3 +SECOND_SOURCE_TYPE=$INPUT_FORMAT; + +PREVIOUS_BRANCH=$2 ## We will still call it a branch even if it's not from a git repo. +CURRENT_BRANCH=$4 + +echo "We are going to compare source 1 which is a $FIRST_SOURCE_TYPE and source 2, which is a $SECOND_SOURCE_TYPE" + + +# Check that if either source is from a git repo, that the name is reasonable. +if [[ "$FIRST_SOURCE_TYPE" = "git_repo" ]]; then + + git check-ref-format --branch $2 +fi + +if [[ "$SECOND_SOURCE_TYPE" = "git_repo" ]]; then + + git check-ref-format --branch $4 +fi + +#If the JDIFF_WORKING_DIRECTORY is set, then we will output the report there. Otherwise, to the default location +if [[ "$JDIFF_WORKING_DIRECTORY" = "" ]]; then + + JDIFF_WORKING_DIRECTORY=/tmp/jdiff + echo "JDIFF_WORKING_DIRECTORY not set. That's not an issue. We will default it to $JDIFF_WORKING_DIRECTORY." +else + echo "JDIFF_WORKING_DIRECTORY set to $JDIFF_WORKING_DIRECTORY"; +fi +mkdir -p $JDIFF_WORKING_DIRECTORY + +# We will need this to reference the template we want to use +cd $JDIFF_WORKING_DIRECTORY +scenario_template_name=hbase_jdiff_p-$PREVIOUS_BRANCH-c-$CURRENT_BRANCH.xml + + +# Pull down JDiff tool and unpack it +if [ ! -d jdiff-1.1.1-with-incompatible-option ]; then + curl -O http://cloud.github.com/downloads/tomwhite/jdiff/jdiff-1.1.1-with-incompatible-option.zip + unzip jdiff-1.1.1-with-incompatible-option.zip +fi + +JDIFF_HOME=`pwd`/jdiff-1.1.1-with-incompatible-option +cd $JDIFF_WORKING_DIRECTORY + +# Pull down sources if necessary. Note that references to previous change are prefaced with p- in order to avoid collission of branch names +if [[ "$FIRST_SOURCE_TYPE" = "git_repo" ]]; then + + PREVIOUS_REPO=$1 + rm -rf p-$PREVIOUS_BRANCH + mkdir -p p-$PREVIOUS_BRANCH + cd p-$PREVIOUS_BRANCH + git clone --depth 1 $PREVIOUS_REPO && cd hbase && git checkout origin/$PREVIOUS_BRANCH + cd $JDIFF_WORKING_DIRECTORY + HBASE_1_HOME=`pwd`/p-$PREVIOUS_BRANCH/hbase +else + HBASE_1_HOME=$1 +fi + +echo "HBASE_1_HOME set to $HBASE_1_HOME" +echo "In HBASE_1_HOME, we have" +ls -la $HBASE_1_HOME + +if [[ "$SECOND_SOURCE_TYPE" = "git_repo" ]]; then + CURRENT_REPO=$3 + rm -rf $JDIFF_WORKING_DIRECTORY/c-$CURRENT_BRANCH + mkdir -p $JDIFF_WORKING_DIRECTORY/c-$CURRENT_BRANCH + cd $JDIFF_WORKING_DIRECTORY/c-$CURRENT_BRANCH + git clone --depth 1 $CURRENT_REPO && cd hbase && git checkout origin/$CURRENT_BRANCH + cd $JDIFF_WORKING_DIRECTORY + HBASE_2_HOME=`pwd`/c-$CURRENT_BRANCH/hbase +else + HBASE_2_HOME=$3 +fi + +echo "HBASE_2_HOME set to $HBASE_2_HOME" +echo "In HBASE_2_HOME, we have" +ls -la $HBASE_2_HOME + +# Next step is to pull down the proper template based on the directory structure +isNewFormat $HBASE_1_HOME +export P_FORMAT=$BRANCH_FORMAT + +isNewFormat $HBASE_2_HOME +export C_FORMAT=$BRANCH_FORMAT + +if [[ "$C_FORMAT" = "new" ]]; then + + if [[ "$P_FORMAT" = "new" ]]; then + templateFile=$DEV_SUPPORT_HOME/hbase_jdiff_afterSingularityTemplate.xml + echo "Previous format is of the new style. We'll be using template $templateFile"; + else + templateFile=$DEV_SUPPORT_HOME/hbase_jdiff_acrossSingularityTemplate.xml + echo "Previous format is of the old style. We'll be using template $templateFile"; + fi + +else + + if [[ "P_FORMAT" != "old" ]]; then + echo "When using this tool, please specify the initial version first and the current version second. They should be in ascending chronological order. + The semantics do not make sense otherwise. For example: jdiff 94 95 is good. jdiff 95 94 is bad." + echo "Exiting the script." + exit 5; + fi + templateFile=$DEV_SUPPORT_HOME/hbase_jdiff_template.xml + echo "Both formats are using the 94 and earlier style directory format. We'll be using template $templateFile" +fi + +cp $templateFile $JDIFF_WORKING_DIRECTORY/$scenario_template_name + +### Configure the jdiff script + +### Note that PREVIOUS_BRANCH and CURRENT_BRANCH will be the absolute locations of the source. +echo "Configuring the jdiff script" +sed -i "s]hbase_jdiff_report]hbase_jdiff_report-p-$PREVIOUS_BRANCH-c-$CURRENT_BRANCH]g" $JDIFF_WORKING_DIRECTORY/$scenario_template_name +sed -i "s]JDIFF_HOME_NAME]$JDIFF_HOME]g" $JDIFF_WORKING_DIRECTORY/$scenario_template_name +sed -i "s]OLD_BRANCH_NAME]$HBASE_1_HOME]g" $JDIFF_WORKING_DIRECTORY/$scenario_template_name +sed -i "s]NEW_BRANCH_NAME]$HBASE_2_HOME]g" $JDIFF_WORKING_DIRECTORY/$scenario_template_name + +sed -i "s]V1]$PREVIOUS_BRANCH]g" $JDIFF_WORKING_DIRECTORY/$scenario_template_name +sed -i "s]V2]$CURRENT_BRANCH]g" $JDIFF_WORKING_DIRECTORY/$scenario_template_name + +sed -i "s]JDIFF_FOLDER]$JDIFF_WORKING_DIRECTORY]g" $JDIFF_WORKING_DIRECTORY/$scenario_template_name + +echo "Running jdiff"; +ls -la $JDIFF_WORKING_DIRECTORY; +ant -f $JDIFF_WORKING_DIRECTORY/$scenario_template_name; + +echo "jdiff operation complete. Report placed into $JDIFF_WORKING_DIRECTORY/hbase_jdiff_report-p-$PREVIOUS_BRANCH-c-$CURRENT_BRANCH/changes.html"; + diff --git a/dev-support/jdiffHBasePublicAPI_common.sh b/dev-support/jdiffHBasePublicAPI_common.sh new file mode 100644 index 000000000000..1cc99549b585 --- /dev/null +++ b/dev-support/jdiffHBasePublicAPI_common.sh @@ -0,0 +1,76 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +########################################################################################################################## +# +### Purpose: To describe whether the directory specified has the old directory format or the new directory format +### Usage: This function takes one argument: The directory in question +### It will set the temporary variable BRANCH_FORMAT. This variable can change with every call, so it is up to the user to +### store it into something else as soon as the function exists +### Example: +### > isNewFormat ./myDevDir/testing/branch/hbase +isNewFormat() { + + echo "Determining if directory $1 is of the 0.94 and before OR 0.95 and after versions"; + if [[ "$1" = "" ]]; then + echo "Directory not specified. Exiting"; + fi + echo "First, check that $1 exists"; + if [[ -d $1 ]]; then + echo "Directory $1 exists" + else + echo "Directory $1 does not exist. Exiting"; + exit 1; + fi + + if [[ -d "$1/hbase-server" ]]; then + + echo "The directory $1/hbase-server exists so this is of the new format"; + export BRANCH_FORMAT=new; + + else + echo "The directory $1/hbase-server does not exist. Therefore, this is of the old format"; + export BRANCH_FORMAT=old; + fi +} + +### Purpose: To describe whether the argument specified is a git repo or a local directory +### Usage: This function takes one argument: The directory in question +### It will set the temporary variable INPUT_FORMAT. This variable can change with every call, so it is up to the user to +### store it into something else as soon as the function exists +### Example: +### > isGitRepo ./myDevDir/testing/branch/hbase + +isGitRepo() { + + echo "Determining if this is a local directory or a git repo."; + if [[ "$1" = "" ]]; then + echo "No value specified for repo or directory. Exiting." + exit 1; + fi + + if [[ `echo $1 | grep 'http://'` || `echo $1 | grep 'https://'` || `echo $1 | grep 'git://'` ]]; then + echo "Looks like $1 is a git repo"; + export INPUT_FORMAT=git_repo + else + echo "$1 is a local directory"; + export INPUT_FORMAT=local_directory + fi + + +} diff --git a/dev-support/smart-apply-patch.sh b/dev-support/smart-apply-patch.sh new file mode 100755 index 000000000000..9200e3ba921c --- /dev/null +++ b/dev-support/smart-apply-patch.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -e + +PATCH_FILE=$1 +if [ -z "$PATCH_FILE" ]; then + echo usage: $0 patch-file + exit 1 +fi + +PATCH=${PATCH:-patch} # allow overriding patch binary + +# Cleanup handler for temporary files +TOCLEAN="" +cleanup() { + rm $TOCLEAN + exit $1 +} +trap "cleanup 1" HUP INT QUIT TERM + +# Allow passing "-" for stdin patches +if [ "$PATCH_FILE" == "-" ]; then + PATCH_FILE=/tmp/tmp.in.$$ + cat /dev/fd/0 > $PATCH_FILE + TOCLEAN="$TOCLEAN $PATCH_FILE" +fi + +# Come up with a list of changed files into $TMP +TMP=/tmp/tmp.paths.$$ +TOCLEAN="$TOCLEAN $TMP" + +if $PATCH -p0 -E --dry-run < $PATCH_FILE 2>&1 > $TMP; then + PLEVEL=0 + #if the patch applied at P0 there is the possability that all we are doing + # is adding new files and they would apply anywhere. So try to guess the + # correct place to put those files. + +# NOTE 2014/07/17: +# Temporarily disabling below check since our jenkins boxes seems to be not defaulting to bash +# causing below checks to fail. Once it is fixed, we can revert the commit and enable this again. + +# TMP2=/tmp/tmp.paths.2.$$ +# TOCLEAN="$TOCLEAN $TMP2" +# +# grep '^patching file ' $TMP | awk '{print $3}' | grep -v /dev/null | sort | uniq > $TMP2 +# +# #first off check that all of the files do not exist +# FOUND_ANY=0 +# for CHECK_FILE in $(cat $TMP2) +# do +# if [[ -f $CHECK_FILE ]]; then +# FOUND_ANY=1 +# fi +# done +# +# if [[ "$FOUND_ANY" = "0" ]]; then +# #all of the files are new files so we have to guess where the correct place to put it is. +# +# # if all of the lines start with a/ or b/, then this is a git patch that +# # was generated without --no-prefix +# if ! grep -qv '^a/\|^b/' $TMP2 ; then +# echo Looks like this is a git patch. Stripping a/ and b/ prefixes +# echo and incrementing PLEVEL +# PLEVEL=$[$PLEVEL + 1] +# sed -i -e 's,^[ab]/,,' $TMP2 +# fi +# fi +elif $PATCH -p1 -E --dry-run < $PATCH_FILE 2>&1 > /dev/null; then + PLEVEL=1 +elif $PATCH -p2 -E --dry-run < $PATCH_FILE 2>&1 > /dev/null; then + PLEVEL=2 +else + echo "The patch does not appear to apply with p0 to p2"; + cleanup 1; +fi + +echo Going to apply patch with: $PATCH -p$PLEVEL +$PATCH -p$PLEVEL -E < $PATCH_FILE + +cleanup $? diff --git a/dev-support/test-patch.properties b/dev-support/test-patch.properties index 6c3f5e37aaad..afe21d455dad 100644 --- a/dev-support/test-patch.properties +++ b/dev-support/test-patch.properties @@ -19,5 +19,5 @@ MAVEN_OPTS="-Xmx3g" # Please update the per-module test-patch.properties if you update this file. OK_RELEASEAUDIT_WARNINGS=84 -OK_FINDBUGS_WARNINGS=607 +OK_FINDBUGS_WARNINGS=768 OK_JAVADOC_WARNINGS=169 diff --git a/dev-support/test-patch.sh b/dev-support/test-patch.sh old mode 100644 new mode 100755 index efa1fc795ed6..3c728454d035 --- a/dev-support/test-patch.sh +++ b/dev-support/test-patch.sh @@ -1,15 +1,20 @@ #!/usr/bin/env bash -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. #set -x @@ -366,7 +371,7 @@ checkJavadocWarnings () { echo "There appear to be $javadocWarnings javadoc warnings generated by the patched build." ### if current warnings greater than OK_JAVADOC_WARNINGS - if [[ $javadocWarnings > $OK_JAVADOC_WARNINGS ]] ; then + if [[ $javadocWarnings -gt $OK_JAVADOC_WARNINGS ]] ; then JIRA_COMMENT="$JIRA_COMMENT -1 javadoc. The javadoc tool appears to have generated `expr $(($javadocWarnings-$OK_JAVADOC_WARNINGS))` warning messages." @@ -545,7 +550,7 @@ $JIRA_COMMENT_FOOTER" done ### if current warnings greater than OK_FINDBUGS_WARNINGS - if [[ $findbugsWarnings > $OK_FINDBUGS_WARNINGS ]] ; then + if [[ $findbugsWarnings -gt $OK_FINDBUGS_WARNINGS ]] ; then JIRA_COMMENT="$JIRA_COMMENT -1 findbugs. The patch appears to introduce `expr $(($findbugsWarnings-$OK_FINDBUGS_WARNINGS))` new Findbugs (version ${findbugs_version}) warnings." diff --git a/dev-support/test-util.sh b/dev-support/test-util.sh index c37ab485acb9..9219bb96606c 100755 --- a/dev-support/test-util.sh +++ b/dev-support/test-util.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash # #/** -# * Copyright 2007 The Apache Software Foundation -# * # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information @@ -39,9 +37,13 @@ options: -n N Run each test N times. Default = 1. -s N Print N slowest tests -H Print which tests are hanging (if any) + -e Echo the maven call before running. Default: not enabled + -r Runs remotely, on the build server. Default: not enabled EOF } +echoUsage=0 +server=0 testFile= doClean="" testType=verify @@ -59,7 +61,7 @@ else fi testDir=$scriptDir/../../../target/surefire-reports -while getopts "hcHun:s:f:" OPTION +while getopts "hcerHun:s:f:" OPTION do case $OPTION in h) @@ -84,6 +86,12 @@ do f) testFile=$OPTARG ;; + e) + echoUsage=1 + ;; + r) + server=1 + ;; ?) usage exit 1 @@ -124,15 +132,44 @@ do #Now loop through each test for (( j = 0; j < $numTests; j++ )) do - nice -10 mvn $doClean $testType -Dtest=${test[$j]} - if [ $? -ne 0 ]; then + # Create the general command + cmd="nice -10 mvn $doClean $testType -Dtest=${test[$j]}" + + # Add that is should run locally, if not on the server + if [ ${server} -eq 0 ]; then + cmd="${cmd} -P localTests" + fi + + # Print the command, if we should + if [ ${echoUsage} -eq 1 ]; then + echo "${cmd}" + fi + + # Run the command + $cmd + + if [ $? -ne 0 ]; then echo "${test[$j]} failed, iteration: $i" exit 1 fi done else echo "EXECUTING ALL TESTS" - nice -10 mvn $doClean $testType + # Create the general command + cmd="nice -10 mvn $doClean $testType" + + # Add that is should run locally, if not on the server + if [ ${server} -eq 0 ]; then + cmd="${cmd} -P localTests" + fi + + # Print the command, if we should + if [ ${echoUsage} -eq 1 ]; then + echo "${cmd}" + fi + + #now run the command + $cmd fi done diff --git a/pom.xml b/pom.xml index 4460d5d22461..ccbcfbbea81d 100644 --- a/pom.xml +++ b/pom.xml @@ -1,4 +1,4 @@ - + + + org.apache.maven.plugins + maven-remote-resources-plugin + 1.5 + + + org.apache.maven.plugins + maven-shade-plugin + 2.3 + + + org.apache.maven.plugins + maven-release-plugin + + + + apache-release + + -Dmaven.test.skip.exec + + maven-compiler-plugin + 2.5.1 ${compileSource} ${compileSource} true false + -Xlint:-options @@ -349,15 +374,15 @@ - 900 + ${surefire.timeout} -enableassertions -Xmx1900m -Djava.security.egd=file:/dev/./urandom - true + ${test.output.tofile} org.apache.maven.plugins maven-site-plugin - 2.0.1 + 3.3 org.apache.maven.plugins @@ -375,14 +400,14 @@ ${integrationtest.include} - ${unittest.include} + ${unittest.include} **/*$* - ${test.exclude.pattern} - true + ${test.output.tofile} ${env.LD_LIBRARY_PATH}:${project.build.directory}/nativelib ${env.DYLD_LIBRARY_PATH}:${project.build.directory}/nativelib + 4 @@ -421,14 +446,19 @@ avro-maven-plugin ${avro.version} + + org.codehaus.mojo + buildnumber-maven-plugin + 1.3 + org.codehaus.mojo build-helper-maven-plugin 1.5 - org.eclipse.m2e @@ -450,7 +480,7 @@ - + org.apache.maven.plugins @@ -477,30 +507,71 @@ + + + org.apache.maven.plugins + maven-remote-resources-plugin + [1.5,) + + process + + + + + + + + + org.codehaus.mojo + buildnumber-maven-plugin + [1.3,) + + create-timestamp + + + + + true + true + + + org.apache.rat - apache-rat-plugin + apache-rat-plugin 0.8 + **/*.log **/.* + **/*.tgz + **/*.orig + test/** + **/8e8ab58dcf39412da19833fcd8f687ac **/.git/** **/target/** **/CHANGES.txt **/generated/** - **/conf/* + + conf/regionservers **/*.avpr - **/*.svg + **/*.svg **/*.vm **/control **/conffile docs/* **/src/site/resources/css/freebsd_docbook.css + + **/src/main/resources/META-INF/LEGAL + + .git/** + .svn/** + **/patchprocess/** @@ -515,6 +586,16 @@ hbase-default.xml + ${project.build.directory} @@ -529,9 +610,26 @@ hbase-site.xml + + + maven-site-plugin + + UTF-8 + UTF-8 + src/site/site.vm + + org.apache.avro avro-maven-plugin @@ -549,6 +647,52 @@ ${project.build.directory}/generated-sources/java + + org.apache.maven.plugins + maven-enforcer-plugin + 1.0.1 + + + org.codehaus.mojo + extra-enforcer-rules + ${extra.enforcer.version} + + + + + + + + [${maven.min.version},) + Maven is out of date. + HBase requires at least version ${maven.min.version} of Maven to properly build from source. + You appear to be using an older version. You can use either "mvn -version" or + "mvn enforcer:display-info" to verify what version is active. + See the reference guide on building for more information: http://hbase.apache.org/book.html#build + + + + + [${java.min.version},) + Java is out of date. + HBase requirs at least version ${java.min.version} of the JDK to properly build from source. + You appear to be using an older version. You can use either "mvn -version" or + "mvn enforcer:display-info" to verify what version is active. + See the reference guide on building for more information: http://hbase.apache.org/book.html#build + + + + + + + enforce + + enforce + + + + + org.codehaus.mojo xml-maven-plugin @@ -590,7 +734,7 @@ 100 true true - ${basedir}/target/site/book/ + ${basedir}/target/site/ ../css/freebsd_docbook.css src/docbkx/customization.xsl ../images/ @@ -689,6 +833,16 @@ jar-no-fork + @@ -729,6 +883,12 @@ false always + + 1800 + -enableassertions -Xmx1900m + -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true + false @@ -802,6 +962,18 @@ package="org.apache.hadoop.hbase.generated.regionserver" webxml="${build.webapps}/regionserver/WEB-INF/web.xml"/> + + + + + + @@ -839,8 +1011,14 @@ - if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0 ]; then - cp -PR ${project.build.directory}/nativelib/lib* ${project.build.directory}/${project.build.finalName}/${project.build.finalName}/lib/native/${build.platform} + which cygpath 2> /dev/null + if [ $? = 1 ]; then + BUILD_DIR="${project.build.directory}" + else + BUILD_DIR=`cygpath --unix '${project.build.directory}'` + fi + if [ `ls $BUILD_DIR/nativelib | wc -l` -ne 0 ]; then + cp -PR $BUILD_DIR/nativelib/lib* $BUILD_DIR/${project.build.finalName}/${project.build.finalName}/lib/native/${build.platform} fi @@ -848,11 +1026,19 @@ - - - - + + which cygpath 2> /dev/null + if [ $? = 1 ]; then + BUILD_DIR="${project.build.directory}" + else + BUILD_DIR=`cygpath --unix '${project.build.directory}'` + fi + + cd $BUILD_DIR/${project.build.finalName} + tar czf $BUILD_DIR/${project.build.finalName}.tar.gz ${project.build.finalName} + + + @@ -868,6 +1054,28 @@ build-helper-maven-plugin 1.5 + + add-source + + add-source + + + + ${project.basedir}/security/src/main/java + + + + + add-test-source + + add-test-source + + + + ${project.basedir}/security/src/test/java + + + jspcSource-packageInfo-Avro-source generate-sources @@ -925,45 +1133,172 @@ + + org.codehaus.mojo + buildnumber-maven-plugin + + + validate + + create-timestamp + + + + + yyyy + build.year + + + + org.apache.maven.plugins + maven-remote-resources-plugin + + + + build-legal-for-assembly + + + process-sources + + process + + + src/assembly + ${project.build.directory}/maven-shared-archive-resources-for-assembly + + src/assembly/resource + + + src/assembly/resources/supplemental-models.xml + + + true + ${license.debug.print.included} + + + org.apache:apache-jar-resource-bundle:1.4 + + false + false + false + + + + + + + + maven-dependency-plugin + + + + unpack-dependency-notices + prepare-package + + unpack-dependencies + + + true + **\/NOTICE,**\/NOTICE.txt + + + + + + org.codehaus.mojo + exec-maven-plugin + 1.4.0 + + + concat-NOTICE-files + prepare-package + + exec + + + env + + bash + -c + cat maven-shared-archive-resources-for-assembly/META-INF/NOTICE \ + `find ${project.build.directory}/dependency -iname NOTICE -or -iname NOTICE.txt` \ + + + ${project.build.directory}/NOTICE.aggregate + ${project.build.directory} + + + + + + + org.apache.felix + maven-bundle-plugin + 2.5.3 + true + true + + + false yyyy-MM-dd'T'HH:mm - - ${maven.build.timestamp} - 1.6 - + ${maven.build.timestamp} + + 3.0.3 + ${compileSource} 1.5.3 1.2 1.4 + + 3.2.2 3.1 2.1 2.5 1.1.1 2.1 1.6 - r09 - 1.5.5 + 2.1.2 + 11.0.2 + 1.8.8 5.5.23 2.1 6.1.26 6.1.14 - 1.4 + 1.8 1.6.5 - 4.10-HBASE-1 + 4.11 + 1.3 + 1.4.3 1.2.16 1.8.5 2.4.0a - 1.5.8 1.0.1 + thrift 0.8.0 - 3.4.3 + 3.4.5 0.0.1-SNAPSHOT + 2.6.3 /usr /etc/hbase @@ -992,6 +1327,9 @@ org.apache.hadoop.hbase.SmallTests org.apache.hadoop.hbase.MediumTests + true + 900 + 1.0-beta-3 @@ -1007,6 +1345,11 @@ + + com.yammer.metrics + metrics-core + ${metrics-core.version} + com.google.guava guava @@ -1032,6 +1375,11 @@ commons-codec ${commons-codec.version} + + commons-collections + commons-collections + ${commons-collections.version} + commons-httpclient commons-httpclient @@ -1262,7 +1610,9 @@ junit junit ${junit.version} - test + runtime + + true org.mockito @@ -1407,6 +1757,17 @@ Mac_OS_X-${sun.arch.data.model} + + os.windows + + + Windows + + + + cygwin + + @@ -1425,6 +1786,39 @@ + + org.apache.maven.plugins + maven-enforcer-plugin + + + + ${compileSource} + HBase has unsupported dependencies. + HBase requires that all dependencies be compiled with version ${compileSource} or earlier + of the JDK to properly build from source. You appear to be using a newer dependency. You can use + either "mvn -version" or "mvn enforcer:display-info" to verify what version is active. + Non-release builds can temporarily build with a newer JDK version by setting the + 'compileSource' property (eg. mvn -DcompileSource=1.8 clean package). + + + + + + + maven-javadoc-plugin + 2.6.1 + + true + + + + prepare-package + + javadoc + + + + @@ -1486,7 +1880,8 @@ - 1.0.0 + 1.0.4 + 1.4.3 @@ -1553,11 +1948,11 @@ - + - security + security-test - 1.0.0 + 1.0.4 ${project.artifactId}-${project.version}-security @@ -1566,28 +1961,6 @@ org.codehaus.mojo build-helper-maven-plugin - - add-source - - add-source - - - - ${project.basedir}/security/src/main/java - - - - - add-test-source - - add-test-source - - - - ${project.basedir}/security/src/test/java - - - add-test-resource @@ -1612,28 +1985,28 @@ - hadoop-0.22 + hadoop-1.1 hadoop.profile - 22 + 1.1 - 0.22.0 + 1.1.2 + 1.4.3 org.apache.hadoop - hadoop-common + hadoop-core ${hadoop.version} true - hsqldb hsqldb @@ -1654,59 +2027,222 @@ oro oro - - jdiff - jdiff - - - org.apache.lucene - lucene-core - org.apache.hadoop - hadoop-hdfs + hadoop-test ${hadoop.version} true - - - - hsqldb - hsqldb - - - net.sf.kosmosfs - kfs - - - org.eclipse.jdt - core - - - net.java.dev.jets3t - jets3t - - - oro - oro - - - jdiff - jdiff - - - org.apache.lucene - lucene-core - - + test - - org.apache.hadoop - hadoop-mapred - ${hadoop.version} - true - + + + + + org.codehaus.mojo + build-helper-maven-plugin + + + add-test-resource + + add-test-resource + + + + + src/test/resources + + hbase-site.xml + + + + + + + + + + + + + + hadoop-1.2 + + + hadoop.profile + 1.2 + + + + 1.2.1 + 1.4.3 + + + + org.apache.hadoop + hadoop-core + ${hadoop.version} + true + + + hsqldb + hsqldb + + + net.sf.kosmosfs + kfs + + + org.eclipse.jdt + core + + + net.java.dev.jets3t + jets3t + + + oro + oro + + + + + org.apache.hadoop + hadoop-test + ${hadoop.version} + true + test + + + + + + org.codehaus.mojo + build-helper-maven-plugin + + + add-test-resource + + add-test-resource + + + + + src/test/resources + + hbase-site.xml + + + + + + + + + + + + + + hadoop-0.22 + + + hadoop.profile + 22 + + + + 0.22.0 + 1.6.1 + + + + org.apache.hadoop + hadoop-common + ${hadoop.version} + true + + + + hsqldb + hsqldb + + + net.sf.kosmosfs + kfs + + + org.eclipse.jdt + core + + + net.java.dev.jets3t + jets3t + + + oro + oro + + + jdiff + jdiff + + + org.apache.lucene + lucene-core + + + + + org.apache.hadoop + hadoop-hdfs + ${hadoop.version} + true + + + + hsqldb + hsqldb + + + net.sf.kosmosfs + kfs + + + org.eclipse.jdt + core + + + net.java.dev.jets3t + jets3t + + + oro + oro + + + jdiff + jdiff + + + org.apache.lucene + lucene-core + + + + + org.apache.hadoop + hadoop-mapred + ${hadoop.version} + true + hsqldb @@ -1802,7 +2338,8 @@ - 0.23.1-SNAPSHOT + 0.23.7 + 1.6.1 @@ -1870,6 +2407,269 @@ + + + hadoop-0.24 + + + hadoop.profile + 24 + + + + 0.24.0-SNAPSHOT + 1.6.1 + + + + org.apache.hadoop + hadoop-common + ${hadoop.version} + + + org.apache.hadoop + hadoop-annotations + ${hadoop.version} + + + + org.apache.hadoop + hadoop-minicluster + ${hadoop.version} + compile + + + + + + org.codehaus.mojo + build-helper-maven-plugin + + + add-test-resource + + add-test-resource + + + + + src/test/resources + + hbase-site.xml + + + + + + + + + maven-dependency-plugin + + + create-mrapp-generated-classpath + generate-test-resources + + build-classpath + + + + ${project.build.directory}/test-classes/mrapp-generated-classpath + + + + + + + + + + + hadoop-2.0 + + + hadoop.profile + 2.0 + + + + 2.0.0-alpha + 1.6.1 + + + + org.apache.hadoop + hadoop-common + ${hadoop.version} + + + org.apache.hadoop + hadoop-annotations + ${hadoop.version} + + + + org.apache.hadoop + hadoop-minicluster + ${hadoop.version} + compile + + + + + + org.codehaus.mojo + build-helper-maven-plugin + + + add-test-resource + + add-test-resource + + + + + src/test/resources + + hbase-site.xml + + + + + + + + + maven-dependency-plugin + + + create-mrapp-generated-classpath + generate-test-resources + + build-classpath + + + + ${project.build.directory}/test-classes/mrapp-generated-classpath + + + + + + + + + + + hadoop-2.7 + + + hadoop.profile + 2.7 + + + + 2.7.1 + 1.6.1 + 2.5.0 + 2.4 + + + + org.apache.hadoop + hadoop-common + ${hadoop.version} + + + org.apache.hadoop + hadoop-annotations + ${hadoop.version} + + + + org.apache.hadoop + hadoop-minicluster + ${hadoop.version} + compile + + + + + + org.codehaus.mojo + build-helper-maven-plugin + + + add-test-resource + + add-test-resource + + + + + src/test/resources + + hbase-site.xml + + + + + + + + + maven-dependency-plugin + + + create-mrapp-generated-classpath + generate-test-resources + + build-classpath + + + + ${project.build.directory}/test-classes/mrapp-generated-classpath + + + + + + + + @@ -2019,6 +2819,61 @@ + + + + clover + + false + + clover + + + + ${user.home}/.clover.license + 2.6.3 + + + + + com.atlassian.maven.plugins + maven-clover2-plugin + ${clover.version} + + true + true + 50% + true + true + + **/generated/** + + + + + clover-setup + process-sources + + setup + + + + clover + site + + clover + + + + + + + @@ -2026,7 +2881,7 @@ maven-project-info-reports-plugin - 2.1.2 + 2.6 @@ -2043,85 +2898,116 @@ - maven-site-plugin - 2.0.1 - - UTF-8 - UTF-8 - src/site/site.vm - - - + org.apache.maven.plugins maven-javadoc-plugin - 2.6.1 - - true - + 2.10.3 + - default + devapi - javadoc + aggregate + + devapidocs + Developer API + The full HBase API, including private and unstable APIs + + **/generated/* + **/protobuf/* + **/*.scala + + *.generated.master:*.generated:org.apache.hadoop.hbase.tmpl.common:com.google.protobuf:org.apache.hadoop.hbase.spark + true + true + true + true + true + all + true + + -J-Xmx2G + + + + org.mockito + mockito-all + ${mockito-all.version} + + + org.hamcrest + hamcrest-core + ${hamcrest.version} + + + false + + + + + + userapi + + aggregate + + + apidocs + User API + The HBase Application Programmer's API + + org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2* + + + false + + + org.apache.hbase:hbase-annotations + + ${project.reporting.outputDirectory}/devapidocs + Developer API + The full HBase API, including private and unstable APIs + **/generated/* + org.apache.hadoop.hbase.generated.master:org.apache.hadoop.hbase.protobuf.generated:org.apache.hadoop.hbase.tmpl.common + true + true + true + true + true + all + true + + -J-Xmx2G + + + + org.mockito + mockito-all + ${mockito-all.version} + + + org.hamcrest + hamcrest-core + ${hamcrest.version} + + + false + - - - org.apache.maven.plugins maven-jxr-plugin - 2.1 - - org.apache.rat apache-rat-plugin @@ -2148,4 +3034,14 @@ + + + hbase.apache.org + HBase Website at hbase.apache.org + + file:///tmp + + diff --git a/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java b/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java index e85bf42220e1..70fe4b72595d 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java +++ b/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java @@ -21,6 +21,7 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.client.UserProvider; import org.apache.hadoop.hbase.security.HBaseSaslRpcClient; import org.apache.hadoop.hbase.security.HBaseSaslRpcServer.AuthMethod; import org.apache.hadoop.hbase.security.KerberosInfo; @@ -40,6 +41,7 @@ import org.apache.hadoop.util.ReflectionUtils; import javax.net.SocketFactory; +import javax.security.sasl.SaslException; import java.io.*; import java.net.*; import java.security.PrivilegedExceptionAction; @@ -71,6 +73,10 @@ public class SecureClient extends HBaseClient { private static final Log LOG = LogFactory.getLog("org.apache.hadoop.ipc.SecureClient"); + public static final String IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY = + "hbase.ipc.client.fallback-to-simple-auth-allowed"; + public static final boolean IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_DEFAULT = false; + protected static Map> tokenHandlers = new HashMap>(); static { @@ -97,7 +103,7 @@ public SecureConnection(ConnectionId remoteId) throws IOException { User ticket = remoteId.getTicket(); Class protocol = remoteId.getProtocol(); - this.useSasl = User.isSecurityEnabled(); + this.useSasl = userProvider.isHBaseSecurityEnabled(); if (useSasl && protocol != null) { TokenInfo tokenInfo = protocol.getAnnotation(TokenInfo.class); if (tokenInfo != null) { @@ -172,7 +178,7 @@ private synchronized boolean shouldAuthenticateOverKrb() throws IOException { private synchronized boolean setupSaslConnection(final InputStream in2, final OutputStream out2) throws IOException { - saslRpcClient = new HBaseSaslRpcClient(authMethod, token, serverPrincipal); + saslRpcClient = new HBaseSaslRpcClient(authMethod, token, serverPrincipal, fallbackAllowed); return saslRpcClient.saslConnect(in2, out2); } @@ -185,6 +191,14 @@ private synchronized boolean setupSaslConnection(final InputStream in2, * again. * The other problem is to do with ticket expiry. To handle that, * a relogin is attempted. + *

+ * The retry logic is governed by the {@link #shouldAuthenticateOverKrb} + * method. In case when the user doesn't have valid credentials, we don't + * need to retry (from cache or ticket). In such cases, it is prudent to + * throw a runtime exception when we receive a SaslException from the + * underlying authentication implementation, so there is no retry from + * other high level (for eg, HCM or HBaseAdmin). + *

*/ private synchronized void handleSaslConnectionFailure( final int currRetries, @@ -222,8 +236,16 @@ public Object run() throws IOException, InterruptedException { LOG.warn("Exception encountered while connecting to " + "the server : " + ex); } - if (ex instanceof RemoteException) + if (ex instanceof RemoteException) { throw (RemoteException)ex; + } + if (ex instanceof SaslException) { + String msg = "SASL authentication failed." + + " The most likely cause is missing or invalid credentials." + + " Consider 'kinit'."; + LOG.fatal(msg, ex); + throw new RuntimeException(msg, ex); + } throw new IOException(ex); } }); @@ -246,7 +268,7 @@ protected synchronized void setupIOstreams() while (true) { setupConnection(); InputStream inStream = NetUtils.getInputStream(socket); - OutputStream outStream = NetUtils.getOutputStream(socket); + OutputStream outStream = NetUtils.getOutputStream(socket, pingInterval); writeRpcHeader(outStream); if (useSasl) { final InputStream in2 = inStream; @@ -255,7 +277,7 @@ protected synchronized void setupIOstreams() if (authMethod == AuthMethod.KERBEROS) { UserGroupInformation ugi = ticket.getUGI(); if (ugi != null && ugi.getRealUser() != null) { - ticket = User.create(ugi.getRealUser()); + ticket = userProvider.create(ugi.getRealUser()); } } boolean continueSasl = false; @@ -271,8 +293,11 @@ public Boolean run() throws IOException { if (rand == null) { rand = new Random(); } - handleSaslConnectionFailure(numRetries++, MAX_RETRIES, ex, rand, - ticket); + try { + handleSaslConnectionFailure(numRetries++, MAX_RETRIES, ex, rand, ticket); + } catch (InterruptedException e) { + throw new IOException(e); + } continue; } if (continueSasl) { @@ -300,7 +325,14 @@ public Boolean run() throws IOException { start(); return; } - } catch (IOException e) { + } catch (Throwable t) { + failedServers.addToFailedServers(remoteId.address); + IOException e; + if (t instanceof IOException) { + e = (IOException)t; + } else { + e = new IOException("Could not set up Secure IO Streams", t); + } markClosed(e); close(); @@ -346,7 +378,11 @@ protected void receiveResponse() { if (LOG.isDebugEnabled()) LOG.debug(getName() + " got value #" + id); - Call call = calls.remove(id); + // we first get the call by id, then remove it from call map after processed. + // If we remove the call here, thread waiting on the call can not be notified + // if any we encounter any exception in the 'try' block. Refer to 'receiveResponse' + // in org.apache.hadoop.hbase.ipc.HBaseClient.java + Call call = calls.get(id); int state = in.readInt(); // read call status if (LOG.isDebugEnabled()) { @@ -358,15 +394,28 @@ protected void receiveResponse() { if (LOG.isDebugEnabled()) { LOG.debug("call #"+id+", response is:\n"+value.toString()); } - call.setValue(value); + // it's possible that this call may have been cleaned up due to a RPC + // timeout, so check if it still exists before setting the value. + if (call != null) { + call.setValue(value); + } } else if (state == Status.ERROR.state) { - call.setException(new RemoteException(WritableUtils.readString(in), - WritableUtils.readString(in))); + if (call != null) { + call.setException(new RemoteException(WritableUtils.readString(in), WritableUtils + .readString(in))); + } } else if (state == Status.FATAL.state) { + RemoteException exception = new RemoteException(WritableUtils.readString(in), + WritableUtils.readString(in)); + // the call will be removed from call map, we must set Exception here to notify + // the thread waited on the call + if (call != null) { + call.setException(exception); + } // Close the connection - markClosed(new RemoteException(WritableUtils.readString(in), - WritableUtils.readString(in))); + markClosed(exception); } + calls.remove(id); } catch (IOException e) { if (e instanceof SocketTimeoutException && remoteId.rpcTimeout > 0) { // Clean up open calls but don't treat this as a fatal condition, @@ -395,9 +444,7 @@ protected synchronized void close() { // release the resources // first thing to do;take the connection out of the connection list synchronized (connections) { - if (connections.get(remoteId) == this) { - connections.remove(remoteId); - } + connections.removeValue(remoteId, this); } // close the streams and therefore the socket @@ -430,6 +477,9 @@ protected synchronized void close() { } } + private final boolean fallbackAllowed; + private UserProvider userProvider; + /** * Construct an IPC client whose values are of the given {@link org.apache.hadoop.io.Writable} * class. @@ -438,8 +488,15 @@ protected synchronized void close() { * @param factory socket factory */ public SecureClient(Class valueClass, Configuration conf, - SocketFactory factory) { + SocketFactory factory, UserProvider provider) { super(valueClass, conf, factory); + this.fallbackAllowed = + conf.getBoolean(IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY, + IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_DEFAULT); + if (LOG.isDebugEnabled()) { + LOG.debug("fallbackAllowed=" + this.fallbackAllowed); + } + this.userProvider = provider; } /** @@ -447,42 +504,17 @@ public SecureClient(Class valueClass, Configuration conf, * @param valueClass value class * @param conf configuration */ - public SecureClient(Class valueClass, Configuration conf) { - this(valueClass, conf, NetUtils.getDefaultSocketFactory(conf)); + public SecureClient(Class valueClass, Configuration conf, + UserProvider provider) { + this(valueClass, conf, NetUtils.getDefaultSocketFactory(conf), provider); } + /** + * Creates a SecureConnection. Can be overridden by a subclass for testing. + * @param remoteId - the ConnectionId to use for the connection creation. + */ @Override - protected SecureConnection getConnection(InetSocketAddress addr, - Class protocol, - User ticket, - int rpcTimeout, - Call call) - throws IOException, InterruptedException { - if (!running.get()) { - // the client is stopped - throw new IOException("The client is stopped"); - } - SecureConnection connection; - /* we could avoid this allocation for each RPC by having a - * connectionsId object and with set() method. We need to manage the - * refs for keys in HashMap properly. For now its ok. - */ - ConnectionId remoteId = new ConnectionId(addr, protocol, ticket, rpcTimeout); - do { - synchronized (connections) { - connection = (SecureConnection)connections.get(remoteId); - if (connection == null) { - connection = new SecureConnection(remoteId); - connections.put(remoteId, connection); - } - } - } while (!connection.addCall(call)); - - //we don't invoke the method below inside "synchronized (connections)" - //block above. The reason for that is if the server happens to be slow, - //it will take longer to establish a connection and that will slow the - //entire system down. - connection.setupIOstreams(); - return connection; + protected SecureConnection createConnection(ConnectionId remoteId) throws IOException { + return new SecureConnection(remoteId); } -} \ No newline at end of file +} diff --git a/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java b/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java index 506082151997..cfcaf53f0ac8 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java +++ b/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java @@ -21,9 +21,12 @@ import java.io.DataOutput; import java.io.IOException; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.client.UserProvider; import org.apache.hadoop.hbase.security.HBaseSaslRpcServer.AuthMethod; -import org.apache.hadoop.io.Text; import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.io.Text; import org.apache.hadoop.security.UserGroupInformation; /** @@ -61,14 +64,16 @@ public void readFields(DataInput in) throws IOException { if (ugiUsernamePresent) { String username = in.readUTF(); boolean realUserNamePresent = in.readBoolean(); + Configuration conf = HBaseConfiguration.create(); + UserProvider provider = UserProvider.instantiate(conf); if (realUserNamePresent) { String realUserName = in.readUTF(); UserGroupInformation realUserUgi = UserGroupInformation.createRemoteUser(realUserName); - user = User.create( + user = provider.create( UserGroupInformation.createProxyUser(username, realUserUgi)); } else { - user = User.create(UserGroupInformation.createRemoteUser(username)); + user = provider.create(UserGroupInformation.createRemoteUser(username)); } } else { user = null; diff --git a/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java b/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java index 8219bea79930..5e6e0e727c63 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java +++ b/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java @@ -20,10 +20,8 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.conf.Configurable; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.Server; -import org.apache.hadoop.hbase.client.RetriesExhaustedException; +import org.apache.hadoop.hbase.client.UserProvider; import org.apache.hadoop.hbase.io.HbaseObjectWritable; import org.apache.hadoop.hbase.monitoring.MonitoredRPCHandler; import org.apache.hadoop.hbase.security.HBasePolicyProvider; @@ -32,20 +30,11 @@ import org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager; import org.apache.hadoop.hbase.util.Objects; import org.apache.hadoop.io.Writable; -import org.apache.hadoop.metrics.util.MetricsTimeVaryingRate; -import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.security.authorize.ServiceAuthorizationManager; -import javax.net.SocketFactory; -import java.io.DataInput; -import java.io.DataOutput; import java.io.IOException; import java.lang.reflect.*; -import java.net.ConnectException; import java.net.InetSocketAddress; -import java.net.SocketTimeoutException; -import java.util.HashMap; -import java.util.Map; /** * A loadable RPC engine supporting SASL authentication of connections, using @@ -63,94 +52,48 @@ */ public class SecureRpcEngine implements RpcEngine { // Leave this out in the hadoop ipc package but keep class name. Do this - // so that we dont' get the logging of this class's invocations by doing our + // so that we do not get the logging of this class' invocations by doing our // blanket enabling DEBUG on the o.a.h.h. package. protected static final Log LOG = LogFactory.getLog("org.apache.hadoop.ipc.SecureRpcEngine"); - private SecureRpcEngine() { - super(); - } // no public ctor + private Configuration conf; + private SecureClient client; + private UserProvider provider; - /* Cache a client using its socket factory as the hash key */ - static private class ClientCache { - private Map clients = - new HashMap(); - - protected ClientCache() {} - - /** - * Construct & cache an IPC client with the user-provided SocketFactory - * if no cached client exists. - * - * @param conf Configuration - * @param factory socket factory - * @return an IPC client - */ - protected synchronized SecureClient getClient(Configuration conf, - SocketFactory factory) { - // Construct & cache client. The configuration is only used for timeout, - // and Clients have connection pools. So we can either (a) lose some - // connection pooling and leak sockets, or (b) use the same timeout for all - // configurations. Since the IPC is usually intended globally, not - // per-job, we choose (a). - SecureClient client = clients.get(factory); - if (client == null) { - // Make an hbase client instead of hadoop Client. - client = new SecureClient(HbaseObjectWritable.class, conf, factory); - clients.put(factory, client); - } else { - client.incCount(); - } - return client; - } - - /** - * Construct & cache an IPC client with the default SocketFactory - * if no cached client exists. - * - * @param conf Configuration - * @return an IPC client - */ - protected synchronized SecureClient getClient(Configuration conf) { - return getClient(conf, SocketFactory.getDefault()); + @Override + public void setConf(Configuration config) { + this.conf = config; + this.provider = UserProvider.instantiate(config); + if (provider.isHBaseSecurityEnabled()) { + HBaseSaslRpcServer.init(conf); } - - /** - * Stop a RPC client connection - * A RPC client is closed only when its reference count becomes zero. - * @param client client to stop - */ - protected void stopClient(SecureClient client) { - synchronized (this) { - client.decCount(); - if (client.isZeroReference()) { - clients.remove(client.getSocketFactory()); - } - } - if (client.isZeroReference()) { - client.stop(); - } + // check for an already created client + if (this.client != null) { + this.client.stop(); } + this.client = new SecureClient(HbaseObjectWritable.class, conf, provider); } - protected final static ClientCache CLIENTS = new ClientCache(); + @Override + public Configuration getConf() { + return this.conf; + } private static class Invoker implements InvocationHandler { private Class protocol; private InetSocketAddress address; private User ticket; private SecureClient client; - private boolean isClosed = false; final private int rpcTimeout; - public Invoker(Class protocol, - InetSocketAddress address, User ticket, - Configuration conf, SocketFactory factory, int rpcTimeout) { + public Invoker(SecureClient client, + Class protocol, + InetSocketAddress address, User ticket, int rpcTimeout) { this.protocol = protocol; this.address = address; this.ticket = ticket; - this.client = CLIENTS.getClient(conf, factory); + this.client = client; this.rpcTimeout = rpcTimeout; } @@ -162,7 +105,7 @@ public Object invoke(Object proxy, Method method, Object[] args) startTime = System.currentTimeMillis(); } HbaseObjectWritable value = (HbaseObjectWritable) - client.call(new Invocation(method, args), address, + client.call(new Invocation(method, protocol, args), address, protocol, ticket, rpcTimeout); if (logDebug) { long callTime = System.currentTimeMillis() - startTime; @@ -170,14 +113,6 @@ public Object invoke(Object proxy, Method method, Object[] args) } return value.get(); } - - /* close the IPC client that's responsible for this invoker's RPCs */ - synchronized protected void close() { - if (!isClosed) { - isClosed = true; - CLIENTS.stopClient(client); - } - } } /** @@ -187,24 +122,30 @@ synchronized protected void close() { * @param protocol interface * @param clientVersion version we are expecting * @param addr remote address - * @param ticket ticket * @param conf configuration - * @param factory socket factory * @return proxy * @throws java.io.IOException e */ - public VersionedProtocol getProxy( - Class protocol, long clientVersion, - InetSocketAddress addr, User ticket, - Configuration conf, SocketFactory factory, int rpcTimeout) + @Override + public T getProxy( + Class protocol, long clientVersion, + InetSocketAddress addr, + Configuration conf, int rpcTimeout) throws IOException { - if (User.isSecurityEnabled()) { - HBaseSaslRpcServer.init(conf); + if (this.client == null) { + throw new IOException("Client must be initialized by calling setConf(Configuration)"); } - VersionedProtocol proxy = - (VersionedProtocol) Proxy.newProxyInstance( + + T proxy = + (T) Proxy.newProxyInstance( protocol.getClassLoader(), new Class[] { protocol }, - new Invoker(protocol, addr, ticket, conf, factory, rpcTimeout)); + new Invoker(this.client, protocol, addr, provider.getCurrent(), + HBaseRPC.getRpcTimeout(rpcTimeout))); + /* + * TODO: checking protocol version only needs to be done once when we setup a new + * SecureClient.Connection. Doing it every time we retrieve a proxy instance is resulting + * in unnecessary RPC traffic. + */ long serverVersion = proxy.getProtocolVersion(protocol.getName(), clientVersion); if (serverVersion != clientVersion) { @@ -214,50 +155,48 @@ public VersionedProtocol getProxy( return proxy; } - /** - * Stop this proxy and release its invoker's resource - * @param proxy the proxy to be stopped - */ - public void stopProxy(VersionedProtocol proxy) { - if (proxy!=null) { - ((Invoker)Proxy.getInvocationHandler(proxy)).close(); - } - } - - /** Expert: Make multiple, parallel calls to a set of servers. */ + @Override public Object[] call(Method method, Object[][] params, InetSocketAddress[] addrs, Class protocol, User ticket, Configuration conf) throws IOException, InterruptedException { + if (this.client == null) { + throw new IOException("Client must be initialized by calling setConf(Configuration)"); + } Invocation[] invocations = new Invocation[params.length]; - for (int i = 0; i < params.length; i++) - invocations[i] = new Invocation(method, params[i]); - SecureClient client = CLIENTS.getClient(conf); - try { - Writable[] wrappedValues = - client.call(invocations, addrs, protocol, ticket); - - if (method.getReturnType() == Void.TYPE) { - return null; - } + for (int i = 0; i < params.length; i++) { + invocations[i] = new Invocation(method, protocol, params[i]); + } + + Writable[] wrappedValues = + client.call(invocations, addrs, protocol, ticket); + + if (method.getReturnType() == Void.TYPE) { + return null; + } - Object[] values = - (Object[])Array.newInstance(method.getReturnType(), wrappedValues.length); - for (int i = 0; i < values.length; i++) - if (wrappedValues[i] != null) - values[i] = ((HbaseObjectWritable)wrappedValues[i]).get(); + Object[] values = + (Object[])Array.newInstance(method.getReturnType(), wrappedValues.length); + for (int i = 0; i < values.length; i++) + if (wrappedValues[i] != null) + values[i] = ((HbaseObjectWritable)wrappedValues[i]).get(); + + return values; + } - return values; - } finally { - CLIENTS.stopClient(client); + @Override + public void close() { + if (this.client != null) { + this.client.stop(); } } /** Construct a server for a protocol implementation instance listening on a * port and address, with a secret manager. */ + @Override public Server getServer(Class protocol, final Object instance, Class[] ifaces, diff --git a/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java b/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java index 0766f5d23ebb..c5fe8c1640c7 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java +++ b/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java @@ -21,6 +21,8 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.client.UserProvider; import org.apache.hadoop.hbase.io.HbaseObjectWritable; import org.apache.hadoop.hbase.io.WritableWithSize; import org.apache.hadoop.hbase.security.HBaseSaslRpcServer; @@ -47,6 +49,8 @@ import org.apache.hadoop.util.ReflectionUtils; import org.apache.hadoop.util.StringUtils; +import com.google.common.collect.ImmutableSet; + import javax.security.sasl.Sasl; import javax.security.sasl.SaslException; import javax.security.sasl.SaslServer; @@ -83,20 +87,23 @@ public abstract class SecureServer extends HBaseServer { // 3 : Introduce the protocol into the RPC connection header // 4 : Introduced SASL security layer public static final byte CURRENT_VERSION = 4; + public static final Set INSECURE_VERSIONS = ImmutableSet.of(3); + + public static final Log LOG = LogFactory.getLog(SecureServer.class); + private static final Log AUDITLOG = LogFactory.getLog("SecurityLogger." + + SecureServer.class.getName()); - public static final Log LOG = LogFactory.getLog("org.apache.hadoop.ipc.SecureServer"); - private static final Log AUDITLOG = - LogFactory.getLog("SecurityLogger.org.apache.hadoop.ipc.SecureServer"); private static final String AUTH_FAILED_FOR = "Auth failed for "; private static final String AUTH_SUCCESSFUL_FOR = "Auth successful for "; protected SecretManager secretManager; protected ServiceAuthorizationManager authManager; + private UserProvider userProvider; protected class SecureCall extends HBaseServer.Call { public SecureCall(int id, Writable param, Connection connection, - Responder responder) { - super(id, param, connection, responder); + Responder responder, long size) { + super(id, param, connection, responder, size); } @Override @@ -168,9 +175,10 @@ private void wrapWithSasl(ByteBufferOutputStream response) token = ((SecureConnection)connection).saslServer.wrap(buf.array(), buf.arrayOffset(), buf.remaining()); } - if (LOG.isDebugEnabled()) - LOG.debug("Adding saslServer wrapped token of size " + token.length + if (LOG.isTraceEnabled()) { + LOG.trace("Adding saslServer wrapped token of size " + token.length + " as call response."); + } buf.clear(); DataOutputStream saslOut = new DataOutputStream(response); saslOut.writeInt(token.length); @@ -205,7 +213,7 @@ public class SecureConnection extends HBaseServer.Connection { private final int AUTHORIZATION_FAILED_CALLID = -1; // Fake 'call' for SASL context setup private static final int SASL_CALLID = -33; - private final SecureCall saslCall = new SecureCall(SASL_CALLID, null, this, null); + private final SecureCall saslCall = new SecureCall(SASL_CALLID, null, this, null, 0); private boolean useWrap = false; @@ -246,9 +254,9 @@ private User getAuthorizedUgi(String authorizedId) "Can't retrieve username from tokenIdentifier."); } ugi.addTokenIdentifier(tokenId); - return User.create(ugi); + return userProvider.create(ugi); } else { - return User.create(UserGroupInformation.createRemoteUser(authorizedId)); + return userProvider.create(UserGroupInformation.createRemoteUser(authorizedId)); } } @@ -273,8 +281,9 @@ HBaseSaslRpcServer.SASL_PROPS, new SaslDigestCallbackHandler( UserGroupInformation current = UserGroupInformation .getCurrentUser(); String fullName = current.getUserName(); - if (LOG.isDebugEnabled()) - LOG.debug("Kerberos principal name is " + fullName); + if (LOG.isTraceEnabled()) { + LOG.trace("Kerberos principal name is " + fullName); + } final String names[] = HBaseSaslRpcServer.splitKerberosName(fullName); if (names.length != 3) { throw new AccessControlException( @@ -295,13 +304,15 @@ public Object run() throws SaslException { throw new AccessControlException( "Unable to find SASL server implementation for " + authMethod.getMechanismName()); - if (LOG.isDebugEnabled()) - LOG.debug("Created SASL server with mechanism = " + if (LOG.isTraceEnabled()) { + LOG.trace("Created SASL server with mechanism = " + authMethod.getMechanismName()); + } } - if (LOG.isDebugEnabled()) - LOG.debug("Have read input token of size " + saslToken.length + if (LOG.isTraceEnabled()) { + LOG.trace("Have read input token of size " + saslToken.length + " for processing by saslServer.evaluateResponse()"); + } replyToken = saslServer.evaluateResponse(saslToken); } catch (IOException e) { IOException sendToClient = e; @@ -322,28 +333,33 @@ public Object run() throws SaslException { throw e; } if (replyToken != null) { - if (LOG.isDebugEnabled()) - LOG.debug("Will send token of size " + replyToken.length + if (LOG.isTraceEnabled()) { + LOG.trace("Will send token of size " + replyToken.length + " from saslServer."); + } doSaslReply(SaslStatus.SUCCESS, new BytesWritable(replyToken), null, null); } if (saslServer.isComplete()) { - LOG.debug("SASL server context established. Negotiated QoP is " + if (LOG.isDebugEnabled()) { + LOG.debug("SASL server context established. Negotiated QoP is " + saslServer.getNegotiatedProperty(Sasl.QOP)); + } String qop = (String) saslServer.getNegotiatedProperty(Sasl.QOP); useWrap = qop != null && !"auth".equalsIgnoreCase(qop); ticket = getAuthorizedUgi(saslServer.getAuthorizationID()); - LOG.debug("SASL server successfully authenticated client: " + ticket); + if (LOG.isDebugEnabled()) { + LOG.debug("SASL server successfully authenticated client: " + ticket); + } rpcMetrics.authenticationSuccesses.inc(); - AUDITLOG.trace(AUTH_SUCCESSFUL_FOR + ticket); + AUDITLOG.info(AUTH_SUCCESSFUL_FOR + ticket); saslContextEstablished = true; } } else { - if (LOG.isDebugEnabled()) - LOG.debug("Have read input token of size " + saslToken.length + if (LOG.isTraceEnabled()) { + LOG.trace("Have read input token of size " + saslToken.length + " for processing by saslServer.unwrap()"); - + } if (!useWrap) { processOneRpc(saslToken); } else { @@ -400,10 +416,17 @@ public int readAndProcess() throws IOException, InterruptedException { dataLengthBuffer.flip(); if (!HEADER.equals(dataLengthBuffer) || version != CURRENT_VERSION) { //Warning is ok since this is not supposed to happen. - LOG.warn("Incorrect header or version mismatch from " + - hostAddress + ":" + remotePort + - " got version " + version + - " expected version " + CURRENT_VERSION); + if (INSECURE_VERSIONS.contains(version)) { + LOG.warn("An insecure client (version '" + version + "') is attempting to connect " + + " to this version '" + CURRENT_VERSION + "' secure server from " + + hostAddress + ":" + remotePort); + } else { + LOG.warn("Incorrect header or version mismatch from " + + hostAddress + ":" + remotePort + + " got version " + version + + " expected version " + CURRENT_VERSION); + } + return -1; } dataLengthBuffer.clear(); @@ -414,7 +437,7 @@ public int readAndProcess() throws IOException, InterruptedException { AccessControlException ae = new AccessControlException( "Authentication is required"); SecureCall failedCall = new SecureCall(AUTHORIZATION_FAILED_CALLID, null, this, - null); + null, 0); failedCall.setResponse(null, Status.FATAL, ae.getClass().getName(), ae.getMessage()); responder.doRespond(failedCall); @@ -519,7 +542,8 @@ private void processHeader(byte[] buf) throws IOException { // for simple auth or kerberos auth // The user is the real user. Now we create a proxy user UserGroupInformation realUser = ticket.getUGI(); - ticket = User.create( + ticket = + userProvider.create( UserGroupInformation.createProxyUser(protocolUser.getName(), realUser)); // Now the user is a proxy user, set Authentication method Proxy. @@ -547,8 +571,9 @@ private void processUnwrappedData(byte[] inBuf) throws IOException, int unwrappedDataLength = unwrappedDataLengthBuffer.getInt(); if (unwrappedDataLength == HBaseClient.PING_CALL_ID) { - if (LOG.isDebugEnabled()) - LOG.debug("Received ping message"); + if (LOG.isTraceEnabled()) { + LOG.trace("Received ping message"); + } unwrappedDataLengthBuffer.clear(); continue; // ping message } @@ -587,20 +612,39 @@ protected void processData(byte[] buf) throws IOException, InterruptedException DataInputStream dis = new DataInputStream(new ByteArrayInputStream(buf)); int id = dis.readInt(); // try to read an id + long callSize = buf.length; - if (LOG.isDebugEnabled()) { - LOG.debug(" got #" + id); + if (LOG.isTraceEnabled()) { + LOG.trace(" got #" + id); + } + + // Enforcing the call queue size, this triggers a retry in the client + if ((callSize + callQueueSize.get()) > maxQueueSize) { + final SecureCall callTooBig = + new SecureCall(id, null, this, responder, callSize); + ByteArrayOutputStream responseBuffer = new ByteArrayOutputStream(); + setupResponse(responseBuffer, callTooBig, Status.FATAL, null, + IOException.class.getName(), + "Call queue is full, is ipc.server.max.callqueue.size too small?"); + responder.doRespond(callTooBig); + return; } Writable param = ReflectionUtils.newInstance(paramClass, conf); // read param param.readFields(dis); - SecureCall call = new SecureCall(id, param, this, responder); + SecureCall call = new SecureCall(id, param, this, responder, callSize); + callQueueSize.add(callSize); if (priorityCallQueue != null && getQosLevel(param) > highPriorityLevel) { priorityCallQueue.put(call); + updateCallQueueLenMetrics(priorityCallQueue); + } else if (replicationQueue != null && getQosLevel(param) == HConstants.REPLICATION_QOS) { + replicationQueue.put(call); + updateCallQueueLenMetrics(replicationQueue); } else { callQueue.put(call); // queue the call; maybe blocked here + updateCallQueueLenMetrics(callQueue); } } @@ -620,10 +664,12 @@ private boolean authorizeConnection() throws IOException { } rpcMetrics.authorizationSuccesses.inc(); } catch (AuthorizationException ae) { - LOG.debug("Connection authorization failed: "+ae.getMessage(), ae); + if (LOG.isDebugEnabled()) { + LOG.debug("Connection authorization failed: "+ae.getMessage(), ae); + } rpcMetrics.authorizationFailures.inc(); SecureCall failedCall = new SecureCall(AUTHORIZATION_FAILED_CALLID, null, this, - null); + null, 0); failedCall.setResponse(null, Status.FATAL, ae.getClass().getName(), ae.getMessage()); responder.doRespond(failedCall); @@ -661,8 +707,8 @@ protected SecureServer(String bindAddress, int port, conf, serverName, highPriorityLevel); this.authorize = conf.getBoolean(HADOOP_SECURITY_AUTHORIZATION, false); - this.isSecurityEnabled = UserGroupInformation.isSecurityEnabled(); - LOG.debug("security enabled="+isSecurityEnabled); + this.userProvider = UserProvider.instantiate(this.conf); + this.isSecurityEnabled = userProvider.isHBaseSecurityEnabled(); if (isSecurityEnabled) { HBaseSaslRpcServer.init(conf); @@ -725,4 +771,4 @@ public void authorize(User user, protocol, getConf(), addr); } } -} \ No newline at end of file +} diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java b/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java index 0c4b4cbbaee0..cf1d3f1bf27a 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java +++ b/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java @@ -44,7 +44,7 @@ public Service[] getServices() { public static void init(Configuration conf, ServiceAuthorizationManager authManager) { // set service-level authorization security policy - conf.set("hadoop.policy.file", "hbase-policy.xml"); + System.setProperty("hadoop.policy.file", "hbase-policy.xml"); if (conf.getBoolean( ServiceAuthorizationManager.SERVICE_AUTHORIZATION_CONFIG, false)) { authManager.refresh(conf, new HBasePolicyProvider()); diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java b/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java index 809097305b45..c1eb055b1907 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java +++ b/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java @@ -56,6 +56,7 @@ public class HBaseSaslRpcClient { public static final Log LOG = LogFactory.getLog(HBaseSaslRpcClient.class); private final SaslClient saslClient; + private final boolean fallbackAllowed; /** * Create a HBaseSaslRpcClient for an authentication method @@ -66,8 +67,9 @@ public class HBaseSaslRpcClient { * token to use if needed by the authentication method */ public HBaseSaslRpcClient(AuthMethod method, - Token token, String serverPrincipal) - throws IOException { + Token token, String serverPrincipal, + boolean fallbackAllowed) throws IOException { + this.fallbackAllowed = fallbackAllowed; switch (method) { case DIGEST: if (LOG.isDebugEnabled()) @@ -148,8 +150,14 @@ public boolean saslConnect(InputStream inS, OutputStream outS) readStatus(inStream); int len = inStream.readInt(); if (len == HBaseSaslRpcServer.SWITCH_TO_SIMPLE_AUTH) { - if (LOG.isDebugEnabled()) + if (!fallbackAllowed) { + throw new IOException("Server asks us to fall back to SIMPLE auth," + + " but this client is configured to only allow secure" + + " connections."); + } + if (LOG.isDebugEnabled()) { LOG.debug("Server asks us to fall back to simple auth."); + } saslClient.dispose(); return false; } diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java b/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java index fa001895b3ea..5b4b53d32ff9 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java +++ b/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java @@ -41,6 +41,9 @@ import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.InternalScanner; import org.apache.hadoop.hbase.regionserver.StoreFile; +import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp; +import org.apache.hadoop.hbase.filter.RegexStringComparator; +import org.apache.hadoop.hbase.filter.QualifierFilter; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.Pair; import org.apache.hadoop.io.Text; @@ -79,6 +82,7 @@ public class AccessControlLists { /** Internal storage table for access control lists */ public static final String ACL_TABLE_NAME_STR = "_acl_"; public static final byte[] ACL_TABLE_NAME = Bytes.toBytes(ACL_TABLE_NAME_STR); + public static final byte[] ACL_GLOBAL_NAME = ACL_TABLE_NAME; /** Column family used to store ACL grants */ public static final String ACL_LIST_FAMILY_STR = "l"; public static final byte[] ACL_LIST_FAMILY = Bytes.toBytes(ACL_LIST_FAMILY_STR); @@ -117,32 +121,22 @@ static void init(MasterServices master) throws IOException { } /** - * Stores a new table permission grant in the access control lists table. + * Stores a new user permission grant in the access control lists table. * @param conf the configuration - * @param tableName the table to which access is being granted - * @param username the user or group being granted the permission - * @param perm the details of the permission being granted + * @param userPerm the details of the permission to be granted * @throws IOException in the case of an error accessing the metadata table */ - static void addTablePermission(Configuration conf, - byte[] tableName, String username, TablePermission perm) - throws IOException { + static void addUserPermission(Configuration conf, UserPermission userPerm) + throws IOException { + Permission.Action[] actions = userPerm.getActions(); - Put p = new Put(tableName); - byte[] key = Bytes.toBytes(username); - if (perm.getFamily() != null && perm.getFamily().length > 0) { - key = Bytes.add(key, - Bytes.add(new byte[]{ACL_KEY_DELIMITER}, perm.getFamily())); - if (perm.getQualifier() != null && perm.getQualifier().length > 0) { - key = Bytes.add(key, - Bytes.add(new byte[]{ACL_KEY_DELIMITER}, perm.getQualifier())); - } - } + Put p = new Put(userPerm.isGlobal() ? ACL_GLOBAL_NAME : userPerm.getTable()); + byte[] key = userPermissionKey(userPerm); - TablePermission.Action[] actions = perm.getActions(); if ((actions == null) || (actions.length == 0)) { - LOG.warn("No actions associated with user '"+username+"'"); - return; + String msg = "No actions associated with user '" + Bytes.toString(userPerm.getUser()) + "'"; + LOG.warn(msg); + throw new IOException(msg); } byte[] value = new byte[actions.length]; @@ -152,7 +146,7 @@ static void addTablePermission(Configuration conf, p.add(ACL_LIST_FAMILY, key, value); if (LOG.isDebugEnabled()) { LOG.debug("Writing permission for table "+ - Bytes.toString(tableName)+" "+ + Bytes.toString(userPerm.getTable())+" "+ Bytes.toString(key)+": "+Bytes.toStringBinary(value) ); } @@ -175,34 +169,17 @@ static void addTablePermission(Configuration conf, * column qualifier "info:colA") will have no effect. * * @param conf the configuration - * @param tableName the table of the current permission grant - * @param userName the user or group currently granted the permission - * @param perm the details of the permission to be revoked + * @param userPerm the details of the permission to be revoked * @throws IOException if there is an error accessing the metadata table */ - static void removeTablePermission(Configuration conf, - byte[] tableName, String userName, TablePermission perm) - throws IOException { + static void removeUserPermission(Configuration conf, UserPermission userPerm) + throws IOException { + + Delete d = new Delete(userPerm.isGlobal() ? ACL_GLOBAL_NAME : userPerm.getTable()); + byte[] key = userPermissionKey(userPerm); - Delete d = new Delete(tableName); - byte[] key = null; - if (perm.getFamily() != null && perm.getFamily().length > 0) { - key = Bytes.toBytes(userName + ACL_KEY_DELIMITER + - Bytes.toString(perm.getFamily())); - if (perm.getQualifier() != null && perm.getQualifier().length > 0) { - key = Bytes.toBytes(userName + ACL_KEY_DELIMITER + - Bytes.toString(perm.getFamily()) + ACL_KEY_DELIMITER + - Bytes.toString(perm.getQualifier())); - } else { - key = Bytes.toBytes(userName + ACL_KEY_DELIMITER + - Bytes.toString(perm.getFamily())); - } - } else { - key = Bytes.toBytes(userName); - } if (LOG.isDebugEnabled()) { - LOG.debug("Removing permission for user '" + userName+ "': "+ - perm.toString()); + LOG.debug("Removing permission "+ userPerm.toString()); } d.deleteColumns(ACL_LIST_FAMILY, key); HTable acls = null; @@ -214,6 +191,95 @@ static void removeTablePermission(Configuration conf, } } + /** + * Remove specified table from the _acl_ table. + */ + static void removeTablePermissions(Configuration conf, byte[] tableName) + throws IOException{ + Delete d = new Delete(tableName); + + if (LOG.isDebugEnabled()) { + LOG.debug("Removing permissions of removed table "+ Bytes.toString(tableName)); + } + + HTable acls = null; + try { + acls = new HTable(conf, ACL_TABLE_NAME); + acls.delete(d); + } finally { + if (acls != null) acls.close(); + } + } + + /** + * Remove specified table column from the _acl_ table. + */ + static void removeTablePermissions(Configuration conf, byte[] tableName, byte[] column) + throws IOException{ + + if (LOG.isDebugEnabled()) { + LOG.debug("Removing permissions of removed column " + Bytes.toString(column) + + " from table "+ Bytes.toString(tableName)); + } + + HTable acls = null; + try { + acls = new HTable(conf, ACL_TABLE_NAME); + + Scan scan = new Scan(); + scan.addFamily(ACL_LIST_FAMILY); + + String columnName = Bytes.toString(column); + scan.setFilter(new QualifierFilter(CompareOp.EQUAL, new RegexStringComparator( + String.format("(%s%s%s)|(%s%s)$", + ACL_KEY_DELIMITER, columnName, ACL_KEY_DELIMITER, + ACL_KEY_DELIMITER, columnName)))); + + Set qualifierSet = new TreeSet(Bytes.BYTES_COMPARATOR); + ResultScanner scanner = acls.getScanner(scan); + try { + for (Result res : scanner) { + for (byte[] q : res.getFamilyMap(ACL_LIST_FAMILY).navigableKeySet()) { + qualifierSet.add(q); + } + } + } finally { + scanner.close(); + } + + if (qualifierSet.size() > 0) { + Delete d = new Delete(tableName); + for (byte[] qualifier : qualifierSet) { + d.deleteColumns(ACL_LIST_FAMILY, qualifier); + } + acls.delete(d); + } + } finally { + if (acls != null) acls.close(); + } + } + + /** + * Build qualifier key from user permission: + * username + * username,family + * username,family,qualifier + */ + static byte[] userPermissionKey(UserPermission userPerm) { + byte[] qualifier = userPerm.getQualifier(); + byte[] family = userPerm.getFamily(); + byte[] key = userPerm.getUser(); + + if (family != null && family.length > 0) { + key = Bytes.add(key, Bytes.add(new byte[]{ACL_KEY_DELIMITER}, family)); + if (qualifier != null && qualifier.length > 0) { + key = Bytes.add(key, Bytes.add(new byte[]{ACL_KEY_DELIMITER}, qualifier)); + } + } + + return key; + } + /** * Returns {@code true} if the given region is part of the {@code _acl_} * metadata table. @@ -222,6 +288,13 @@ static boolean isAclRegion(HRegion region) { return Bytes.equals(ACL_TABLE_NAME, region.getTableDesc().getName()); } + /** + * Returns {@code true} if the given table is {@code _acl_} metadata table. + */ + static boolean isAclTable(HTableDescriptor desc) { + return Bytes.equals(ACL_TABLE_NAME, desc.getName()); + } + /** * Loads all of the permission grants stored in a region of the {@code _acl_} * table. @@ -325,20 +398,12 @@ static Map> loadAll( * used for storage. *

*/ - static ListMultimap getTablePermissions( - Configuration conf, byte[] tableName) - throws IOException { - /* TODO: -ROOT- and .META. cannot easily be handled because they must be - * online before _acl_ table. Can anything be done here? - */ - if (Bytes.equals(tableName, HConstants.ROOT_TABLE_NAME) || - Bytes.equals(tableName, HConstants.META_TABLE_NAME) || - Bytes.equals(tableName, AccessControlLists.ACL_TABLE_NAME)) { - return ArrayListMultimap.create(0,0); - } + static ListMultimap getTablePermissions(Configuration conf, + byte[] tableName) throws IOException { + if (tableName == null) tableName = ACL_TABLE_NAME; // for normal user tables, we just read the table row from _acl_ - ListMultimap perms = ArrayListMultimap.create(); + ListMultimap perms = ArrayListMultimap.create(); HTable acls = null; try { acls = new HTable(conf, ACL_TABLE_NAME); @@ -348,8 +413,8 @@ static ListMultimap getTablePermissions( if (!row.isEmpty()) { perms = parseTablePermissions(tableName, row); } else { - LOG.info("No permissions found in "+ACL_TABLE_NAME_STR+ - " for table "+Bytes.toString(tableName)); + LOG.info("No permissions found in " + ACL_TABLE_NAME_STR + " for table " + + Bytes.toString(tableName)); } } finally { if (acls != null) acls.close(); diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java b/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java index 6108de898d3a..97b08498fa30 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java +++ b/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java @@ -1,61 +1,84 @@ /* - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at * - * http://www.apache.org/licenses/LICENSE-2.0 + * http://www.apache.org/licenses/LICENSE-2.0 * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. */ package org.apache.hadoop.hbase.security.access; import java.io.IOException; +import java.net.InetAddress; +import java.security.PrivilegedExceptionAction; import java.util.Arrays; import java.util.Collection; +import java.util.Collections; import java.util.HashMap; -import java.util.HashSet; +import java.util.LinkedList; import java.util.List; import java.util.Map; import java.util.Set; +import java.util.TreeSet; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.CoprocessorEnvironment; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HRegionInfo; import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.TableNotDisabledException; +import org.apache.hadoop.hbase.TableNotFoundException; +import org.apache.hadoop.hbase.client.Append; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.Increment; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.UserProvider; import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver; import org.apache.hadoop.hbase.coprocessor.CoprocessorException; import org.apache.hadoop.hbase.coprocessor.MasterCoprocessorEnvironment; import org.apache.hadoop.hbase.coprocessor.MasterObserver; import org.apache.hadoop.hbase.coprocessor.ObserverContext; import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment; +import org.apache.hadoop.hbase.coprocessor.RegionServerCoprocessorEnvironment; +import org.apache.hadoop.hbase.coprocessor.RegionServerObserver; import org.apache.hadoop.hbase.filter.CompareFilter; import org.apache.hadoop.hbase.filter.FilterList; import org.apache.hadoop.hbase.filter.WritableByteArrayComparable; import org.apache.hadoop.hbase.ipc.HBaseRPC; import org.apache.hadoop.hbase.ipc.ProtocolSignature; import org.apache.hadoop.hbase.ipc.RequestContext; +import org.apache.hadoop.hbase.master.MasterServices; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.InternalScanner; import org.apache.hadoop.hbase.regionserver.RegionScanner; +import org.apache.hadoop.hbase.regionserver.Store; +import org.apache.hadoop.hbase.regionserver.StoreFile; import org.apache.hadoop.hbase.regionserver.wal.WALEdit; import org.apache.hadoop.hbase.security.AccessDeniedException; import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.security.access.Permission.Action; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.apache.hadoop.hbase.util.Pair; import com.google.common.collect.ListMultimap; import com.google.common.collect.Lists; @@ -95,7 +118,7 @@ *

*/ public class AccessController extends BaseRegionObserver - implements MasterObserver, AccessControllerProtocol { + implements MasterObserver, RegionServerObserver, AccessControllerProtocol { /** * Represents the result of an authorization check for logging and error * reporting. @@ -106,12 +129,14 @@ private static class AuthResult { private final byte[] family; private final byte[] qualifier; private final Permission.Action action; + private final String request; private final String reason; private final User user; - public AuthResult(boolean allowed, String reason, User user, + public AuthResult(boolean allowed, String request, String reason, User user, Permission.Action action, byte[] table, byte[] family, byte[] qualifier) { this.allowed = allowed; + this.request = request; this.reason = reason; this.user = user; this.table = table; @@ -126,6 +151,8 @@ public AuthResult(boolean allowed, String reason, User user, public String getReason() { return reason; } + public String getRequest() { return request; } + public String toContextString() { return "(user=" + (user != null ? user.getName() : "UNKNOWN") + ", " + "scope=" + (table == null ? "GLOBAL" : Bytes.toString(table)) + ", " + @@ -139,19 +166,23 @@ public String toString() { .append(toContextString()).toString(); } - public static AuthResult allow(String reason, User user, - Permission.Action action, byte[] table) { - return new AuthResult(true, reason, user, action, table, null, null); + public static AuthResult allow(String request, String reason, User user, Permission.Action action, + byte[] table, byte[] family, byte[] qualifier) { + return new AuthResult(true, request, reason, user, action, table, family, qualifier); + } + + public static AuthResult allow(String request, String reason, User user, Permission.Action action, byte[] table) { + return new AuthResult(true, request, reason, user, action, table, null, null); } - public static AuthResult deny(String reason, User user, + public static AuthResult deny(String request, String reason, User user, Permission.Action action, byte[] table) { - return new AuthResult(false, reason, user, action, table, null, null); + return new AuthResult(false, request, reason, user, action, table, null, null); } - public static AuthResult deny(String reason, User user, + public static AuthResult deny(String request, String reason, User user, Permission.Action action, byte[] table, byte[] family, byte[] qualifier) { - return new AuthResult(false, reason, user, action, table, family, qualifier); + return new AuthResult(false, request, reason, user, action, table, family, qualifier); } } @@ -163,7 +194,7 @@ public static AuthResult deny(String reason, User user, /** * Version number for AccessControllerProtocol */ - private static final long PROTOCOL_VERSION = 2L; + private static final long PROTOCOL_VERSION = 1L; TableAuthManager authManager = null; @@ -178,9 +209,10 @@ public static AuthResult deny(String reason, User user, private Map scannerOwners = new MapMaker().weakKeys().makeMap(); + private UserProvider userProvider; + void initialize(RegionCoprocessorEnvironment e) throws IOException { final HRegion region = e.getRegion(); - Map> tables = AccessControlLists.loadAll(region); // For each table, write out the table's permissions to the respective @@ -188,12 +220,10 @@ void initialize(RegionCoprocessorEnvironment e) throws IOException { for (Map.Entry> t: tables.entrySet()) { byte[] table = t.getKey(); - String tableName = Bytes.toString(table); ListMultimap perms = t.getValue(); byte[] serialized = AccessControlLists.writePermissionsAsBytes(perms, - e.getRegion().getConf()); - this.authManager.getZKPermissionWatcher().writeToZookeeper(tableName, - serialized); + regionEnv.getConfiguration()); + this.authManager.getZKPermissionWatcher().writeToZookeeper(table, serialized); } } @@ -204,31 +234,28 @@ void initialize(RegionCoprocessorEnvironment e) throws IOException { */ void updateACL(RegionCoprocessorEnvironment e, final Map> familyMap) { - Set tableSet = new HashSet(); + Set tableSet = new TreeSet(Bytes.BYTES_COMPARATOR); for (Map.Entry> f : familyMap.entrySet()) { List kvs = f.getValue(); for (KeyValue kv: kvs) { - if (Bytes.compareTo(kv.getBuffer(), kv.getFamilyOffset(), + if (Bytes.equals(kv.getBuffer(), kv.getFamilyOffset(), kv.getFamilyLength(), AccessControlLists.ACL_LIST_FAMILY, 0, - AccessControlLists.ACL_LIST_FAMILY.length) == 0) { - String tableName = Bytes.toString(kv.getRow()); - tableSet.add(tableName); + AccessControlLists.ACL_LIST_FAMILY.length)) { + tableSet.add(kv.getRow()); } } } - for (String tableName: tableSet) { + ZKPermissionWatcher zkw = this.authManager.getZKPermissionWatcher(); + Configuration conf = regionEnv.getConfiguration(); + for (byte[] tableName: tableSet) { try { ListMultimap perms = - AccessControlLists.getTablePermissions(regionEnv.getConfiguration(), - Bytes.toBytes(tableName)); - byte[] serialized = AccessControlLists.writePermissionsAsBytes( - perms, e.getRegion().getConf()); - this.authManager.getZKPermissionWatcher().writeToZookeeper(tableName, - serialized); + AccessControlLists.getTablePermissions(conf, tableName); + byte[] serialized = AccessControlLists.writePermissionsAsBytes(perms, conf); + zkw.writeToZookeeper(tableName, serialized); } catch (IOException ex) { - LOG.error("Failed updating permissions mirror for '" + tableName + - "'", ex); + LOG.error("Failed updating permissions mirror for '" + tableName + "'", ex); } } } @@ -247,41 +274,30 @@ void updateACL(RegionCoprocessorEnvironment e, * the request * @return */ - AuthResult permissionGranted(User user, TablePermission.Action permRequest, + AuthResult permissionGranted(String request, User user, TablePermission.Action permRequest, RegionCoprocessorEnvironment e, Map> families) { HRegionInfo hri = e.getRegion().getRegionInfo(); - HTableDescriptor htd = e.getRegion().getTableDesc(); byte[] tableName = hri.getTableName(); // 1. All users need read access to .META. and -ROOT- tables. // this is a very common operation, so deal with it quickly. - if ((hri.isRootRegion() || hri.isMetaRegion()) && - (permRequest == TablePermission.Action.READ)) { - return AuthResult.allow("All users allowed", user, permRequest, - hri.getTableName()); + if (hri.isRootRegion() || hri.isMetaRegion()) { + if (permRequest == TablePermission.Action.READ) { + return AuthResult.allow(request, "All users allowed", user, permRequest, tableName); + } } if (user == null) { - return AuthResult.deny("No user associated with request!", null, - permRequest, hri.getTableName()); + return AuthResult.deny(request, "No user associated with request!", null, permRequest, tableName); } - // 2. The table owner has full privileges - String owner = htd.getOwnerString(); - if (user.getShortName().equals(owner)) { - // owner of the table has full access - return AuthResult.allow("User is table owner", user, permRequest, - hri.getTableName()); - } - - // 3. check for the table-level, if successful we can short-circuit + // 2. check for the table-level, if successful we can short-circuit if (authManager.authorize(user, tableName, (byte[])null, permRequest)) { - return AuthResult.allow("Table permission granted", user, - permRequest, tableName); + return AuthResult.allow(request, "Table permission granted", user, permRequest, tableName); } - // 4. check permissions against the requested families + // 3. check permissions against the requested families if (families != null && families.size() > 0) { // all families must pass for (Map.Entry> family : families.entrySet()) { @@ -299,7 +315,7 @@ AuthResult permissionGranted(User user, TablePermission.Action permRequest, for (byte[] qualifier : familySet) { if (!authManager.authorize(user, tableName, family.getKey(), qualifier, permRequest)) { - return AuthResult.deny("Failed qualifier check", user, + return AuthResult.deny(request, "Failed qualifier check", user, permRequest, tableName, family.getKey(), qualifier); } } @@ -308,33 +324,40 @@ AuthResult permissionGranted(User user, TablePermission.Action permRequest, for (KeyValue kv : kvList) { if (!authManager.authorize(user, tableName, family.getKey(), kv.getQualifier(), permRequest)) { - return AuthResult.deny("Failed qualifier check", user, + return AuthResult.deny(request, "Failed qualifier check", user, permRequest, tableName, family.getKey(), kv.getQualifier()); } } } } else { // no qualifiers and family-level check already failed - return AuthResult.deny("Failed family check", user, permRequest, + return AuthResult.deny(request, "Failed family check", user, permRequest, tableName, family.getKey(), null); } } // all family checks passed - return AuthResult.allow("All family checks passed", user, permRequest, + return AuthResult.allow(request, "All family checks passed", user, permRequest, tableName); } - // 5. no families to check and table level access failed - return AuthResult.deny("No families to check and table permission failed", + // 4. no families to check and table level access failed + return AuthResult.deny(request, "No families to check and table permission failed", user, permRequest, tableName); } private void logResult(AuthResult result) { if (AUDITLOG.isTraceEnabled()) { + InetAddress remoteAddr = null; + RequestContext ctx = RequestContext.get(); + if (ctx != null) { + remoteAddr = ctx.getRemoteAddress(); + } AUDITLOG.trace("Access " + (result.isAllowed() ? "allowed" : "denied") + " for user " + (result.getUser() != null ? result.getUser().getShortName() : "UNKNOWN") + "; reason: " + result.getReason() + + "; remote address: " + (remoteAddr != null ? remoteAddr : "") + + "; request: " + result.getRequest() + "; context: " + result.toContextString()); } } @@ -348,23 +371,55 @@ private User getActiveUser() throws IOException { User user = RequestContext.getRequestUser(); if (!RequestContext.isInRequestContext()) { // for non-rpc handling, fallback to system user - user = User.getCurrent(); + user = userProvider.getCurrent(); } + return user; } + /** + * Authorizes that the current user has any of the given permissions for the + * given table, column family and column qualifier. + * @param tableName Table requested + * @param family Column family requested + * @param qualifier Column qualifier requested + * @throws IOException if obtaining the current user fails + * @throws AccessDeniedException if user has no authorization + */ + private void requirePermission(String request, byte[] tableName, byte[] family, byte[] qualifier, + Action... permissions) throws IOException { + User user = getActiveUser(); + AuthResult result = null; + + for (Action permission : permissions) { + if (authManager.authorize(user, tableName, family, qualifier, permission)) { + result = AuthResult.allow(request, "Table permission granted", user, + permission, tableName, family, qualifier); + break; + } else { + // rest of the world + result = AuthResult.deny(request, "Insufficient permissions", user, + permission, tableName, family, qualifier); + } + } + logResult(result); + if (!result.isAllowed()) { + throw new AccessDeniedException("Insufficient permissions " + result.toContextString()); + } + } + /** * Authorizes that the current user has global privileges for the given action. * @param perm The action being requested * @throws IOException if obtaining the current user fails * @throws AccessDeniedException if authorization is denied */ - private void requirePermission(Permission.Action perm) throws IOException { + private void requirePermission(String request, Permission.Action perm) throws IOException { User user = getActiveUser(); if (authManager.authorize(user, perm)) { - logResult(AuthResult.allow("Global check allowed", user, perm, null)); + logResult(AuthResult.allow(request, "Global check allowed", user, perm, null)); } else { - logResult(AuthResult.deny("Global check failed", user, perm, null)); + logResult(AuthResult.deny(request, "Global check failed", user, perm, null)); throw new AccessDeniedException("Insufficient permissions for user '" + (user != null ? user.getShortName() : "null") +"' (global, action=" + perm.toString() + ")"); @@ -379,7 +434,7 @@ private void requirePermission(Permission.Action perm) throws IOException { * @param families The set of column families present/required in the request * @throws AccessDeniedException if the authorization check failed */ - private void requirePermission(Permission.Action perm, + private void requirePermission(String request, Permission.Action perm, RegionCoprocessorEnvironment env, Collection families) throws IOException { // create a map of family-qualifier @@ -387,7 +442,7 @@ private void requirePermission(Permission.Action perm, for (byte[] family : families) { familyMap.put(family, null); } - requirePermission(perm, env, familyMap); + requirePermission(request, perm, env, familyMap); } /** @@ -398,12 +453,12 @@ private void requirePermission(Permission.Action perm, * @param families The map of column families-qualifiers. * @throws AccessDeniedException if the authorization check failed */ - private void requirePermission(Permission.Action perm, + public void requirePermission(String request, Permission.Action perm, RegionCoprocessorEnvironment env, Map> families) throws IOException { User user = getActiveUser(); - AuthResult result = permissionGranted(user, perm, env, families); + AuthResult result = permissionGranted(request, user, perm, env, families); logResult(result); if (!result.isAllowed()) { @@ -467,17 +522,34 @@ private boolean hasFamilyQualifierPermission(User user, /* ---- MasterObserver implementation ---- */ public void start(CoprocessorEnvironment env) throws IOException { - // if running on HMaster + + ZooKeeperWatcher zk = null; if (env instanceof MasterCoprocessorEnvironment) { - MasterCoprocessorEnvironment e = (MasterCoprocessorEnvironment)env; - this.authManager = TableAuthManager.get( - e.getMasterServices().getZooKeeper(), - e.getConfiguration()); + // if running on HMaster + MasterCoprocessorEnvironment mEnv = (MasterCoprocessorEnvironment) env; + zk = mEnv.getMasterServices().getZooKeeper(); + } else if (env instanceof RegionServerCoprocessorEnvironment) { + RegionServerCoprocessorEnvironment rsEnv = (RegionServerCoprocessorEnvironment) env; + zk = rsEnv.getRegionServerServices().getZooKeeper(); + } else if (env instanceof RegionCoprocessorEnvironment) { + // if running at region + regionEnv = (RegionCoprocessorEnvironment) env; + zk = regionEnv.getRegionServerServices().getZooKeeper(); } - // if running at region - if (env instanceof RegionCoprocessorEnvironment) { - regionEnv = (RegionCoprocessorEnvironment)env; + // set the user provider + this.userProvider = UserProvider.instantiate(env.getConfiguration()); + + // If zk is null or IOException while obtaining auth manager, + // throw RuntimeException so that the coprocessor is unloaded. + if (zk != null) { + try { + this.authManager = TableAuthManager.get(zk, env.getConfiguration()); + } catch (IOException ioe) { + throw new RuntimeException("Error obtaining TableAuthManager", ioe); + } + } else { + throw new RuntimeException("Error obtaining TableAuthManager, zk found null."); } } @@ -488,115 +560,161 @@ public void stop(CoprocessorEnvironment env) { @Override public void preCreateTable(ObserverContext c, HTableDescriptor desc, HRegionInfo[] regions) throws IOException { - requirePermission(Permission.Action.CREATE); - - // default the table owner if not specified - User owner = getActiveUser(); - if (desc.getOwnerString() == null || - desc.getOwnerString().equals("")) { - desc.setOwner(owner); - } + requirePermission("createTable", Permission.Action.CREATE); } @Override public void postCreateTable(ObserverContext c, - HTableDescriptor desc, HRegionInfo[] regions) throws IOException {} + final HTableDescriptor desc, HRegionInfo[] regions) throws IOException { + if (!AccessControlLists.isAclTable(desc)) { + final Configuration conf = c.getEnvironment().getConfiguration(); + final String owner = (desc.getOwnerString() != null) ? desc.getOwnerString() : + getActiveUser().getShortName(); + User.runAsLoginUser(new PrivilegedExceptionAction() { + @Override + public Void run() throws Exception { + UserPermission userperm = new UserPermission(Bytes.toBytes(owner), desc.getName(), null, + Action.values()); + AccessControlLists.addUserPermission(conf, userperm); + return null; + } + }); + } + } @Override - public void preDeleteTable(ObserverContext c, - byte[] tableName) throws IOException { - requirePermission(Permission.Action.CREATE); + public void preDeleteTable(ObserverContext c, byte[] tableName) + throws IOException { + requirePermission("deleteTable", tableName, null, null, Action.ADMIN, Action.CREATE); } + @Override public void postDeleteTable(ObserverContext c, - byte[] tableName) throws IOException {} - + final byte[] tableName) throws IOException { + final Configuration conf = c.getEnvironment().getConfiguration(); + User.runAsLoginUser(new PrivilegedExceptionAction() { + @Override + public Void run() throws Exception { + AccessControlLists.removeTablePermissions(conf, tableName); + return null; + } + }); + } @Override - public void preModifyTable(ObserverContext c, - byte[] tableName, HTableDescriptor htd) throws IOException { - requirePermission(Permission.Action.CREATE); + public void preModifyTable(ObserverContext c, byte[] tableName, + HTableDescriptor htd) throws IOException { + requirePermission("modifyTable", tableName, null, null, Action.ADMIN, Action.CREATE); } + @Override public void postModifyTable(ObserverContext c, - byte[] tableName, HTableDescriptor htd) throws IOException {} - + byte[] tableName, final HTableDescriptor htd) throws IOException { + final Configuration conf = c.getEnvironment().getConfiguration(); + final String owner = (htd.getOwnerString() != null) ? htd.getOwnerString() : + getActiveUser().getShortName(); + User.runAsLoginUser(new PrivilegedExceptionAction() { + @Override + public Void run() throws Exception { + UserPermission userperm = new UserPermission(Bytes.toBytes(owner), htd.getName(), null, + Action.values()); + AccessControlLists.addUserPermission(conf, userperm); + return null; + } + }); + } @Override - public void preAddColumn(ObserverContext c, - byte[] tableName, HColumnDescriptor column) throws IOException { - requirePermission(Permission.Action.CREATE); + public void preAddColumn(ObserverContext c, byte[] tableName, + HColumnDescriptor column) throws IOException { + requirePermission("addColumn", tableName, null, null, Action.ADMIN, Action.CREATE); } + @Override public void postAddColumn(ObserverContext c, byte[] tableName, HColumnDescriptor column) throws IOException {} - @Override - public void preModifyColumn(ObserverContext c, - byte[] tableName, HColumnDescriptor descriptor) throws IOException { - requirePermission(Permission.Action.CREATE); + public void preModifyColumn(ObserverContext c, byte[] tableName, + HColumnDescriptor descriptor) throws IOException { + requirePermission("modifyColumn", tableName, null, null, Action.ADMIN, Action.CREATE); } + @Override public void postModifyColumn(ObserverContext c, byte[] tableName, HColumnDescriptor descriptor) throws IOException {} - @Override - public void preDeleteColumn(ObserverContext c, - byte[] tableName, byte[] col) throws IOException { - requirePermission(Permission.Action.CREATE); + public void preDeleteColumn(ObserverContext c, byte[] tableName, + byte[] col) throws IOException { + requirePermission("deleteColumn", tableName, null, null, Action.ADMIN, Action.CREATE); } + @Override public void postDeleteColumn(ObserverContext c, - byte[] tableName, byte[] col) throws IOException {} - + final byte[] tableName, final byte[] col) throws IOException { + final Configuration conf = c.getEnvironment().getConfiguration(); + User.runAsLoginUser(new PrivilegedExceptionAction() { + @Override + public Void run() throws Exception { + AccessControlLists.removeTablePermissions(conf, tableName, col); + return null; + } + }); + this.authManager.getZKPermissionWatcher().deleteTableACLNode(tableName); + } @Override - public void preEnableTable(ObserverContext c, - byte[] tableName) throws IOException { - /* TODO: Allow for users with global CREATE permission and the table owner */ - requirePermission(Permission.Action.ADMIN); + public void preEnableTable(ObserverContext c, byte[] tableName) + throws IOException { + requirePermission("enableTable", tableName, null, null, Action.ADMIN, Action.CREATE); } + @Override public void postEnableTable(ObserverContext c, byte[] tableName) throws IOException {} @Override - public void preDisableTable(ObserverContext c, - byte[] tableName) throws IOException { - /* TODO: Allow for users with global CREATE permission and the table owner */ - requirePermission(Permission.Action.ADMIN); + public void preDisableTable(ObserverContext c, byte[] tableName) + throws IOException { + if (Bytes.equals(tableName, AccessControlLists.ACL_GLOBAL_NAME)) { + throw new AccessDeniedException("Not allowed to disable " + + AccessControlLists.ACL_TABLE_NAME_STR + " table."); + } + requirePermission("disableTable", tableName, null, null, Action.ADMIN, Action.CREATE); } + @Override public void postDisableTable(ObserverContext c, byte[] tableName) throws IOException {} @Override - public void preMove(ObserverContext c, - HRegionInfo region, ServerName srcServer, ServerName destServer) - throws IOException { - requirePermission(Permission.Action.ADMIN); + public void preMove(ObserverContext c, HRegionInfo region, + ServerName srcServer, ServerName destServer) throws IOException { + requirePermission("move", region.getTableName(), null, null, Action.ADMIN); } + @Override public void postMove(ObserverContext c, HRegionInfo region, ServerName srcServer, ServerName destServer) throws IOException {} @Override - public void preAssign(ObserverContext c, - HRegionInfo regionInfo) throws IOException { - requirePermission(Permission.Action.ADMIN); + public void preAssign(ObserverContext c, HRegionInfo regionInfo) + throws IOException { + requirePermission("assign", regionInfo.getTableName(), null, null, Action.ADMIN); } + @Override public void postAssign(ObserverContext c, HRegionInfo regionInfo) throws IOException {} @Override - public void preUnassign(ObserverContext c, - HRegionInfo regionInfo, boolean force) throws IOException { - requirePermission(Permission.Action.ADMIN); + public void preUnassign(ObserverContext c, HRegionInfo regionInfo, + boolean force) throws IOException { + requirePermission("unassign", regionInfo.getTableName(), null, null, Action.ADMIN); } + @Override public void postUnassign(ObserverContext c, HRegionInfo regionInfo, boolean force) throws IOException {} @@ -604,7 +722,7 @@ public void postUnassign(ObserverContext c, @Override public void preBalance(ObserverContext c) throws IOException { - requirePermission(Permission.Action.ADMIN); + requirePermission("balance", Permission.Action.ADMIN); } @Override public void postBalance(ObserverContext c) @@ -613,7 +731,7 @@ public void postBalance(ObserverContext c) @Override public boolean preBalanceSwitch(ObserverContext c, boolean newValue) throws IOException { - requirePermission(Permission.Action.ADMIN); + requirePermission("balanceSwitch", Permission.Action.ADMIN); return newValue; } @Override @@ -623,13 +741,13 @@ public void postBalanceSwitch(ObserverContext c, @Override public void preShutdown(ObserverContext c) throws IOException { - requirePermission(Permission.Action.ADMIN); + requirePermission("shutdown", Permission.Action.ADMIN); } @Override public void preStopMaster(ObserverContext c) throws IOException { - requirePermission(Permission.Action.ADMIN); + requirePermission("stopMaster", Permission.Action.ADMIN); } @Override @@ -639,31 +757,87 @@ public void postStartMaster(ObserverContext ctx) AccessControlLists.init(ctx.getEnvironment().getMasterServices()); } + @Override + public void preSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + requirePermission("snapshot", Permission.Action.ADMIN); + } + + @Override + public void postSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + } + + @Override + public void preCloneSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + requirePermission("cloneSnapshot", Permission.Action.ADMIN); + } + + @Override + public void postCloneSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + } + + @Override + public void preRestoreSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + requirePermission("restoreSnapshot", Permission.Action.ADMIN); + } + + @Override + public void postRestoreSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + } + + @Override + public void preDeleteSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot) throws IOException { + requirePermission("deleteSnapshot", Permission.Action.ADMIN); + } + + @Override + public void postDeleteSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot) throws IOException { + } /* ---- RegionObserver implementation ---- */ @Override - public void postOpen(ObserverContext c) { - RegionCoprocessorEnvironment e = c.getEnvironment(); - final HRegion region = e.getRegion(); + public void preOpen(ObserverContext e) throws IOException { + RegionCoprocessorEnvironment env = e.getEnvironment(); + final HRegion region = env.getRegion(); if (region == null) { - LOG.error("NULL region from RegionCoprocessorEnvironment in postOpen()"); + LOG.error("NULL region from RegionCoprocessorEnvironment in preOpen()"); return; + } else { + HRegionInfo regionInfo = region.getRegionInfo(); + if (isSpecialTable(regionInfo)) { + isSystemOrSuperUser(regionEnv.getConfiguration()); + } else { + requirePermission("open", Action.ADMIN); + } } + } - try { - this.authManager = TableAuthManager.get( - e.getRegionServerServices().getZooKeeper(), - e.getRegion().getConf()); - } catch (IOException ioe) { - // pass along as a RuntimeException, so that the coprocessor is unloaded - throw new RuntimeException("Error obtaining TableAuthManager", ioe); + @Override + public void postOpen(ObserverContext c) { + RegionCoprocessorEnvironment env = c.getEnvironment(); + final HRegion region = env.getRegion(); + if (region == null) { + LOG.error("NULL region from RegionCoprocessorEnvironment in postOpen()"); + return; } - if (AccessControlLists.isAclRegion(region)) { aclRegion = true; try { - initialize(e); + initialize(env); } catch (IOException ex) { // if we can't obtain permissions, it's better to fail // than perform checks incorrectly @@ -672,11 +846,36 @@ public void postOpen(ObserverContext c) { } } + @Override + public void preFlush(ObserverContext e) throws IOException { + requirePermission("flush", getTableName(e.getEnvironment()), null, null, Action.ADMIN, + Action.CREATE); + } + + @Override + public void preSplit(ObserverContext e) throws IOException { + requirePermission("split", getTableName(e.getEnvironment()), null, null, Action.ADMIN); + } + + @Override + public InternalScanner preCompact(ObserverContext e, + final Store store, final InternalScanner scanner) throws IOException { + requirePermission("compact", getTableName(e.getEnvironment()), null, null, Action.ADMIN, + Action.CREATE); + return scanner; + } + + @Override + public void preCompactSelection(final ObserverContext e, + final Store store, final List candidates) throws IOException { + requirePermission("compactSelection", getTableName(e.getEnvironment()), null, null, Action.ADMIN); + } + @Override public void preGetClosestRowBefore(final ObserverContext c, final byte [] row, final byte [] family, final Result result) throws IOException { - requirePermission(TablePermission.Action.READ, c.getEnvironment(), + requirePermission("getClosestRowBefore", TablePermission.Action.READ, c.getEnvironment(), (family != null ? Lists.newArrayList(family) : null)); } @@ -689,7 +888,7 @@ public void preGet(final ObserverContext c, */ RegionCoprocessorEnvironment e = c.getEnvironment(); User requestUser = getActiveUser(); - AuthResult authResult = permissionGranted(requestUser, + AuthResult authResult = permissionGranted("get", requestUser, TablePermission.Action.READ, e, get.getFamilyMap()); if (!authResult.isAllowed()) { if (hasFamilyQualifierPermission(requestUser, @@ -706,7 +905,7 @@ public void preGet(final ObserverContext c, } else { get.setFilter(filter); } - logResult(AuthResult.allow("Access allowed with filter", requestUser, + logResult(AuthResult.allow("get", "Access allowed with filter", requestUser, TablePermission.Action.READ, authResult.table)); } else { logResult(authResult); @@ -722,7 +921,7 @@ public void preGet(final ObserverContext c, @Override public boolean preExists(final ObserverContext c, final Get get, final boolean exists) throws IOException { - requirePermission(TablePermission.Action.READ, c.getEnvironment(), + requirePermission("exists", TablePermission.Action.READ, c.getEnvironment(), get.familySet()); return exists; } @@ -731,7 +930,7 @@ public boolean preExists(final ObserverContext c, public void prePut(final ObserverContext c, final Put put, final WALEdit edit, final boolean writeToWAL) throws IOException { - requirePermission(TablePermission.Action.WRITE, c.getEnvironment(), + requirePermission("put", TablePermission.Action.WRITE, c.getEnvironment(), put.getFamilyMap()); } @@ -747,7 +946,7 @@ public void postPut(final ObserverContext c, public void preDelete(final ObserverContext c, final Delete delete, final WALEdit edit, final boolean writeToWAL) throws IOException { - requirePermission(TablePermission.Action.WRITE, c.getEnvironment(), + requirePermission("delete", TablePermission.Action.WRITE, c.getEnvironment(), delete.getFamilyMap()); } @@ -766,8 +965,9 @@ public boolean preCheckAndPut(final ObserverContext familyMap = Arrays.asList(new byte[][]{family}); + requirePermission("checkAndPut", TablePermission.Action.READ, c.getEnvironment(), familyMap); + requirePermission("checkAndPut", TablePermission.Action.WRITE, c.getEnvironment(), familyMap); return result; } @@ -777,8 +977,9 @@ public boolean preCheckAndDelete(final ObserverContext familyMap = Arrays.asList(new byte[][]{family}); + requirePermission("checkAndDelete", TablePermission.Action.READ, c.getEnvironment(), familyMap); + requirePermission("checkAndDelete", TablePermission.Action.WRITE, c.getEnvironment(), familyMap); return result; } @@ -787,16 +988,23 @@ public long preIncrementColumnValue(final ObserverContext c, Append append) + throws IOException { + requirePermission("append", TablePermission.Action.WRITE, c.getEnvironment(), append.getFamilyMap()); + return null; + } + @Override public Result preIncrement(final ObserverContext c, final Increment increment) throws IOException { - requirePermission(TablePermission.Action.WRITE, c.getEnvironment(), + requirePermission("increment", TablePermission.Action.WRITE, c.getEnvironment(), increment.getFamilyMap().keySet()); return null; } @@ -810,7 +1018,7 @@ public RegionScanner preScannerOpen(final ObserverContext private void requireScannerOwner(InternalScanner s) throws AccessDeniedException { if (RequestContext.isInRequestContext()) { + String requestUserName = RequestContext.getRequestUserName(); String owner = scannerOwners.get(s); - if (owner != null && !owner.equals(RequestContext.getRequestUserName())) { - throw new AccessDeniedException("User '"+ - RequestContext.getRequestUserName()+"' is not the scanner owner!"); + if (owner != null && !owner.equals(requestUserName)) { + throw new AccessDeniedException("User '"+ requestUserName +"' is not the scanner owner!"); + } + } + } + + /** + * Verifies user has WRITE privileges on + * the Column Families involved in the bulkLoadHFile + * request. Specific Column Write privileges are presently + * ignored. + */ + @Override + public void preBulkLoadHFile(ObserverContext ctx, + List> familyPaths) throws IOException { + List cfs = new LinkedList(); + for(Pair el : familyPaths) { + cfs.add(el.getFirst()); + } + requirePermission("bulkLoadHFile", Permission.Action.CREATE, ctx.getEnvironment(), cfs); + } + + private AuthResult hasSomeAccess(RegionCoprocessorEnvironment e, String request, Action action) throws IOException { + User requestUser = getActiveUser(); + final byte[] tableName = e.getRegion().getTableDesc().getName(); + AuthResult authResult = permissionGranted(request, requestUser, + action, e, Collections.EMPTY_MAP); + if (!authResult.isAllowed()) { + final Configuration conf = e.getConfiguration(); + // hasSomeAccess is called from bulkload pre hooks + List perms = + User.runAsLoginUser(new PrivilegedExceptionAction>() { + @Override + public List run() throws Exception { + return AccessControlLists.getUserPermissions(conf, tableName); + } + }); + for (UserPermission userPerm: perms) { + for (Action userAction: userPerm.getActions()) { + if (userAction.equals(action)) { + return AuthResult.allow(request, "Access allowed", requestUser, + action, tableName); + } + } } } + return authResult; + } + + /** + * Authorization check for + * SecureBulkLoadProtocol.prepareBulkLoad() + * @param e + * @throws IOException + */ + public void prePrepareBulkLoad(RegionCoprocessorEnvironment e) throws IOException { + AuthResult authResult = hasSomeAccess(e, "prepareBulkLoad", Action.WRITE); + logResult(authResult); + if (!authResult.isAllowed()) { + throw new AccessDeniedException("Insufficient permissions (table=" + + e.getRegion().getTableDesc().getNameAsString() + ", action=WRITE)"); + } + } + + /** + * Authorization security check for + * SecureBulkLoadProtocol.cleanupBulkLoad() + * @param e + * @throws IOException + */ + //TODO this should end up as a coprocessor hook + public void preCleanupBulkLoad(RegionCoprocessorEnvironment e) throws IOException { + AuthResult authResult = hasSomeAccess(e, "cleanupBulkLoad", Action.WRITE); + logResult(authResult); + if (!authResult.isAllowed()) { + throw new AccessDeniedException("Insufficient permissions (table=" + + e.getRegion().getTableDesc().getNameAsString() + ", action=WRITE)"); + } } /* ---- AccessControllerProtocol implementation ---- */ @@ -896,73 +1178,94 @@ private void requireScannerOwner(InternalScanner s) * This will be restricted by both client side and endpoint implementations. */ @Override - public void grant(byte[] user, TablePermission permission) - throws IOException { + public void grant(final UserPermission perm) throws IOException { // verify it's only running at .acl. if (aclRegion) { if (LOG.isDebugEnabled()) { - LOG.debug("Received request to grant access permission to '" - + Bytes.toString(user) + "'. " - + permission.toString()); + LOG.debug("Received request to grant access permission " + perm.toString()); } - requirePermission(Permission.Action.ADMIN); + requirePermission("grant", perm.getTable(), perm.getFamily(), perm.getQualifier(), Action.ADMIN); + + User.runAsLoginUser(new PrivilegedExceptionAction() { + @Override + public Void run() throws Exception { + AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm); + return null; + } + }); - AccessControlLists.addTablePermission(regionEnv.getConfiguration(), - permission.getTable(), Bytes.toString(user), permission); if (AUDITLOG.isTraceEnabled()) { // audit log should store permission changes in addition to auth results - AUDITLOG.trace("Granted user '" + Bytes.toString(user) + "' permission " - + permission.toString()); + AUDITLOG.trace("Granted permission " + perm.toString()); } } else { - throw new CoprocessorException(AccessController.class, "This method " + - "can only execute at " + - Bytes.toString(AccessControlLists.ACL_TABLE_NAME) + " table."); + throw new CoprocessorException(AccessController.class, "This method " + + "can only execute at " + Bytes.toString(AccessControlLists.ACL_TABLE_NAME) + " table."); } } @Override - public void revoke(byte[] user, TablePermission permission) - throws IOException{ + @Deprecated + public void grant(byte[] user, TablePermission permission) + throws IOException { + grant(new UserPermission(user, permission.getTable(), + permission.getFamily(), permission.getQualifier(), + permission.getActions())); + } + + @Override + public void revoke(final UserPermission perm) throws IOException { // only allowed to be called on _acl_ region if (aclRegion) { if (LOG.isDebugEnabled()) { - LOG.debug("Received request to revoke access permission for '" - + Bytes.toString(user) + "'. " - + permission.toString()); + LOG.debug("Received request to revoke access permission " + perm.toString()); } - requirePermission(Permission.Action.ADMIN); + requirePermission("revoke", perm.getTable(), perm.getFamily(), + perm.getQualifier(), Action.ADMIN); + + User.runAsLoginUser(new PrivilegedExceptionAction() { + @Override + public Void run() throws Exception { + AccessControlLists.removeUserPermission(regionEnv.getConfiguration(), perm); + return null; + } + }); - AccessControlLists.removeTablePermission(regionEnv.getConfiguration(), - permission.getTable(), Bytes.toString(user), permission); if (AUDITLOG.isTraceEnabled()) { // audit log should record all permission changes - AUDITLOG.trace("Revoked user '" + Bytes.toString(user) + "' permission " - + permission.toString()); + AUDITLOG.trace("Revoked permission " + perm.toString()); } } else { - throw new CoprocessorException(AccessController.class, "This method " + - "can only execute at " + - Bytes.toString(AccessControlLists.ACL_TABLE_NAME) + " table."); + throw new CoprocessorException(AccessController.class, "This method " + + "can only execute at " + Bytes.toString(AccessControlLists.ACL_TABLE_NAME) + " table."); } } @Override - public List getUserPermissions(final byte[] tableName) + @Deprecated + public void revoke(byte[] user, TablePermission permission) throws IOException { + revoke(new UserPermission(user, permission.getTable(), + permission.getFamily(), permission.getQualifier(), + permission.getActions())); + } + + @Override + public List getUserPermissions(final byte[] tableName) throws IOException { // only allowed to be called on _acl_ region if (aclRegion) { - requirePermission(Permission.Action.ADMIN); - - List perms = AccessControlLists.getUserPermissions - (regionEnv.getConfiguration(), tableName); - return perms; + requirePermission("userPermissions", tableName, null, null, Action.ADMIN); + return User.runAsLoginUser(new PrivilegedExceptionAction>() { + @Override + public List run() throws Exception { + return AccessControlLists.getUserPermissions(regionEnv.getConfiguration(), tableName); + } + }); } else { - throw new CoprocessorException(AccessController.class, "This method " + - "can only execute at " + - Bytes.toString(AccessControlLists.ACL_TABLE_NAME) + " table."); + throw new CoprocessorException(AccessController.class, "This method " + + "can only execute at " + Bytes.toString(AccessControlLists.ACL_TABLE_NAME) + " table."); } } @@ -989,12 +1292,12 @@ public void checkPermissions(Permission[] permissions) throws IOException { } } - requirePermission(action, regionEnv, familyMap); + requirePermission("checkPermissions", action, regionEnv, familyMap); } } else { for (Permission.Action action : permission.getActions()) { - requirePermission(action); + requirePermission("checkPermissions", action); } } } @@ -1027,4 +1330,90 @@ private byte[] getTableName(RegionCoprocessorEnvironment e) { } return tableName; } + + + @Override + public void preClose(ObserverContext e, boolean abortRequested) + throws IOException { + requirePermission("close", Permission.Action.ADMIN); + } + + @Override + public void preLockRow(ObserverContext ctx, byte[] regionName, + byte[] row) throws IOException { + requirePermission("lockRow", getTableName(ctx.getEnvironment()), null, null, + Permission.Action.WRITE, Permission.Action.CREATE); + } + + @Override + public void preUnlockRow(ObserverContext ctx, byte[] regionName, + long lockId) throws IOException { + requirePermission("unlockRow", getTableName(ctx.getEnvironment()), null, null, + Permission.Action.WRITE, Permission.Action.CREATE); + } + + private void isSystemOrSuperUser(Configuration conf) throws IOException { + User user = userProvider.getCurrent(); + if (user == null) { + throw new IOException("Unable to obtain the current user, " + + "authorization checks for internal operations will not work correctly!"); + } + + String currentUser = user.getShortName(); + List superusers = Lists.asList(currentUser, + conf.getStrings(AccessControlLists.SUPERUSER_CONF_KEY, new String[0])); + + User activeUser = getActiveUser(); + if (!(superusers.contains(activeUser.getShortName()))) { + throw new AccessDeniedException("User '" + (user != null ? user.getShortName() : "null") + + "is not system or super user."); + } + } + + private boolean isSpecialTable(HRegionInfo regionInfo) { + byte[] tableName = regionInfo.getTableName(); + return tableName.equals(AccessControlLists.ACL_TABLE_NAME) + || tableName.equals(Bytes.toBytes("-ROOT-")) + || tableName.equals(Bytes.toBytes(".META.")); + } + + @Override + public void preStopRegionServer(ObserverContext env) + throws IOException { + requirePermission("stop", Permission.Action.ADMIN); + } + + @Override + public void preGetTableDescriptors(ObserverContext ctx, + List tableNamesList, List descriptors) throws IOException { + // If the list is empty, this is a request for all table descriptors and requires GLOBAL + // ADMIN privs. + if (tableNamesList == null || tableNamesList.isEmpty()) { + requirePermission("getTableDescriptors", Permission.Action.ADMIN); + } + // Otherwise, if the requestor has ADMIN or CREATE privs for all listed tables, the + // request can be granted. + else { + MasterServices masterServices = ctx.getEnvironment().getMasterServices(); + for (String tableName: tableNamesList) { + // Do not deny if the table does not exist + byte[] nameAsBytes = Bytes.toBytes(tableName); + try { + masterServices.checkTableModifiable(nameAsBytes); + } catch (TableNotFoundException ex) { + // Skip checks for a table that does not exist + continue; + } catch (TableNotDisabledException ex) { + // We don't care about this + } + requirePermission("getTableDescriptors", nameAsBytes, null, null, + Permission.Action.ADMIN, Permission.Action.CREATE); + } + } + } + + @Override + public void postGetTableDescriptors(ObserverContext ctx, + List descriptors) throws IOException { + } } diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java b/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java index 78cca4f88df4..2ecb60a94bff 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java +++ b/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java @@ -28,8 +28,15 @@ */ public interface AccessControllerProtocol extends CoprocessorProtocol { - /* V2: Added {@link #checkPermissions(Permission...)}) */ - public static final long VERSION = 2L; + public static final long VERSION = 1L; + + /** + * Grants the given user or group the privilege to perform the given actions + * @param userPermission the details of the provided user permissions + * @throws IOException if the grant could not be applied + */ + public void grant(UserPermission userPermission) + throws IOException; /** * Grants the given user or group the privilege to perform the given actions @@ -38,10 +45,26 @@ public interface AccessControllerProtocol extends CoprocessorProtocol { * the grant * @param permission the details of the provided permissions * @throws IOException if the grant could not be applied + * @deprecated Use {@link #revoke(UserPermission userPermission)} instead */ + @Deprecated public void grant(byte[] user, TablePermission permission) throws IOException; + /** + * Revokes a previously granted privilege from a user or group. + * Note that the provided {@link TablePermission} details must exactly match + * a stored grant. For example, if user "bob" has been granted "READ" access + * to table "data", over column family and qualifer "info:colA", then the + * table, column family and column qualifier must all be specified. + * Attempting to revoke permissions over just the "data" table will have + * no effect. + * @param permission the details of the previously granted permission to revoke + * @throws IOException if the revocation could not be performed + */ + public void revoke(UserPermission userPermission) + throws IOException; + /** * Revokes a previously granted privilege from a user or group. * Note that the provided {@link TablePermission} details must exactly match @@ -54,7 +77,9 @@ public void grant(byte[] user, TablePermission permission) * privileges are being revoked * @param permission the details of the previously granted permission to revoke * @throws IOException if the revocation could not be performed + * @deprecated Use {@link #revoke(UserPermission userPermission)} instead */ + @Deprecated public void revoke(byte[] user, TablePermission permission) throws IOException; @@ -82,5 +107,4 @@ public List getUserPermissions(byte[] tableName) */ public void checkPermissions(Permission[] permissions) throws IOException; - } diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java b/security/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java new file mode 100644 index 000000000000..1a93c47295b6 --- /dev/null +++ b/security/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java @@ -0,0 +1,328 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.security.access; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.hbase.CoprocessorEnvironment; +import org.apache.hadoop.hbase.DoNotRetryIOException; +import org.apache.hadoop.hbase.client.UserProvider; +import org.apache.hadoop.hbase.coprocessor.BaseEndpointCoprocessor; +import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment; +import org.apache.hadoop.hbase.ipc.RequestContext; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Methods; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.token.Token; + +import java.io.IOException; +import java.math.BigInteger; +import java.security.PrivilegedAction; +import java.security.SecureRandom; +import java.util.List; + +/** + * Coprocessor service for bulk loads in secure mode. + * This coprocessor has to be installed as part of enabling + * security in HBase. + * + * This service addresses two issues: + * + * 1. Moving files in a secure filesystem wherein the HBase Client + * and HBase Server are different filesystem users. + * 2. Does moving in a secure manner. Assuming that the filesystem + * is POSIX compliant. + * + * The algorithm is as follows: + * + * 1. Create an hbase owned staging directory which is + * world traversable (711): /hbase/staging + * 2. A user writes out data to his secure output directory: /user/foo/data + * 3. A call is made to hbase to create a secret staging directory + * which globally rwx (777): /user/staging/averylongandrandomdirectoryname + * 4. The user makes the data world readable and writable, then moves it + * into the random staging directory, then calls bulkLoadHFiles() + * + * Like delegation tokens the strength of the security lies in the length + * and randomness of the secret directory. + * + */ +@InterfaceAudience.Private +public class SecureBulkLoadEndpoint extends BaseEndpointCoprocessor + implements SecureBulkLoadProtocol { + + public static final long VERSION = 0L; + + //Random number is 320 bits wide + private static final int RANDOM_WIDTH = 320; + //We picked 32 as the radix, so the character set + //will only contain alpha numeric values + //320/5 = 64 characters + private static final int RANDOM_RADIX = 32; + + private static Log LOG = LogFactory.getLog(SecureBulkLoadEndpoint.class); + + private final static FsPermission PERM_ALL_ACCESS = FsPermission.valueOf("-rwxrwxrwx"); + private final static FsPermission PERM_HIDDEN = FsPermission.valueOf("-rwx--x--x"); + private final static String BULKLOAD_STAGING_DIR = "hbase.bulkload.staging.dir"; + + private SecureRandom random; + private FileSystem fs; + private Configuration conf; + + //two levels so it doesn't get deleted accidentally + //no sticky bit in Hadoop 1.0 + private Path baseStagingDir; + + private RegionCoprocessorEnvironment env; + + private UserProvider provider; + + @Override + public void start(CoprocessorEnvironment env) { + super.start(env); + + this.env = (RegionCoprocessorEnvironment)env; + random = new SecureRandom(); + conf = env.getConfiguration(); + baseStagingDir = getBaseStagingDir(conf); + this.provider = UserProvider.instantiate(conf); + + try { + fs = FileSystem.get(conf); + fs.mkdirs(baseStagingDir, PERM_HIDDEN); + fs.setPermission(baseStagingDir, PERM_HIDDEN); + //no sticky bit in hadoop-1.0, making directory nonempty so it never gets erased + fs.mkdirs(new Path(baseStagingDir,"DONOTERASE"), PERM_HIDDEN); + FileStatus status = fs.getFileStatus(baseStagingDir); + if(status == null) { + throw new IllegalStateException("Failed to create staging directory"); + } + if(!status.getPermission().equals(PERM_HIDDEN)) { + throw new IllegalStateException("Directory already exists but permissions aren't set to '-rwx--x--x' "); + } + } catch (IOException e) { + throw new IllegalStateException("Failed to get FileSystem instance",e); + } + } + + @Override + public String prepareBulkLoad(byte[] tableName) throws IOException { + getAccessController().prePrepareBulkLoad(env); + return createStagingDir(baseStagingDir, getActiveUser(), tableName).toString(); + } + + @Override + public void cleanupBulkLoad(String bulkToken) throws IOException { + getAccessController().preCleanupBulkLoad(env); + fs.delete(createStagingDir(baseStagingDir, + getActiveUser(), + env.getRegion().getTableDesc().getName(), + new Path(bulkToken).getName()), + true); + } + + @Override + public boolean bulkLoadHFiles(final List> familyPaths, + final Token userToken, final String bulkToken, boolean assignSeqNum) throws IOException { + User user = getActiveUser(); + final UserGroupInformation ugi = user.getUGI(); + if(userToken != null) { + ugi.addToken(userToken); + } else if (provider.isHadoopSecurityEnabled()) { + //we allow this to pass through in "simple" security mode + //for mini cluster testing + throw new DoNotRetryIOException("User token cannot be null"); + } + + HRegion region = env.getRegion(); + boolean bypass = false; + if (region.getCoprocessorHost() != null) { + bypass = region.getCoprocessorHost().preBulkLoadHFile(familyPaths); + } + boolean loaded = false; + final IOException[] es = new IOException[1]; + if (!bypass) { + loaded = ugi.doAs(new PrivilegedAction() { + @Override + public Boolean run() { + FileSystem fs = null; + try { + Configuration conf = env.getConfiguration(); + fs = FileSystem.get(conf); + for(Pair el: familyPaths) { + Path p = new Path(el.getSecond()); + LOG.debug("Setting permission for: " + p); + fs.setPermission(p, PERM_ALL_ACCESS); + Path stageFamily = new Path(bulkToken, Bytes.toString(el.getFirst())); + if(!fs.exists(stageFamily)) { + fs.mkdirs(stageFamily); + fs.setPermission(stageFamily, PERM_ALL_ACCESS); + } + } + //We call bulkLoadHFiles as requesting user + //To enable access prior to staging + return env.getRegion().bulkLoadHFiles(familyPaths, + new SecureBulkLoadListener(fs, bulkToken)); + } + catch(DoNotRetryIOException e){ + es[0] = e; + } + catch (Exception e) { + LOG.error("Failed to complete bulk load", e); + } + return false; + } + }); + } + + if (es[0] != null) { + throw es[0]; + } + + if (region.getCoprocessorHost() != null) { + loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths, loaded); + } + return loaded; + } + + @Override + public long getProtocolVersion(String protocol, long clientVersion) + throws IOException { + if (SecureBulkLoadProtocol.class.getName().equals(protocol)) { + return SecureBulkLoadEndpoint.VERSION; + } + LOG.warn("Unknown protocol requested: " + protocol); + return -1; + } + + private AccessController getAccessController() { + return (AccessController) this.env.getRegion() + .getCoprocessorHost().findCoprocessor(AccessController.class.getName()); + } + + private Path createStagingDir(Path baseDir, User user, byte[] tableName) throws IOException { + String randomDir = user.getShortName()+"__"+Bytes.toString(tableName)+"__"+ + (new BigInteger(RANDOM_WIDTH, random).toString(RANDOM_RADIX)); + return createStagingDir(baseDir, user, tableName, randomDir); + } + + private Path createStagingDir(Path baseDir, + User user, + byte[] tableName, + String randomDir) throws IOException { + Path p = new Path(baseDir, randomDir); + fs.mkdirs(p, PERM_ALL_ACCESS); + fs.setPermission(p, PERM_ALL_ACCESS); + return p; + } + + private User getActiveUser() throws IOException { + User user = RequestContext.getRequestUser(); + if (!RequestContext.isInRequestContext()) { + throw new DoNotRetryIOException("Failed to get requesting user"); + } + + //this is for testing + if("simple".equalsIgnoreCase(conf.get(User.HBASE_SECURITY_CONF_KEY))) { + return User.createUserForTesting(conf, user.getShortName(), new String[]{}); + } + + return user; + } + + /** + * This returns the staging path for a given column family. + * This is needed for clean recovery and called reflectively in LoadIncrementalHFiles + */ + public static Path getStagingPath(Configuration conf, String bulkToken, byte[] family) { + Path stageP = new Path(getBaseStagingDir(conf), bulkToken); + return new Path(stageP, Bytes.toString(family)); + } + + private static Path getBaseStagingDir(Configuration conf) { + return new Path(conf.get(BULKLOAD_STAGING_DIR, "/tmp/hbase-staging")); + } + + private static class SecureBulkLoadListener implements HRegion.BulkLoadListener { + private FileSystem fs; + private String stagingDir; + + public SecureBulkLoadListener(FileSystem fs, String stagingDir) { + this.fs = fs; + this.stagingDir = stagingDir; + } + + @Override + public String prepareBulkLoad(final byte[] family, final String srcPath) throws IOException { + Path p = new Path(srcPath); + Path stageP = new Path(stagingDir, new Path(Bytes.toString(family), p.getName())); + + if(!isFile(p)) { + throw new IOException("Path does not reference a file: " + p); + } + + LOG.debug("Moving " + p + " to " + stageP); + if(!fs.rename(p, stageP)) { + throw new IOException("Failed to move HFile: " + p + " to " + stageP); + } + return stageP.toString(); + } + + @Override + public void doneBulkLoad(byte[] family, String srcPath) throws IOException { + LOG.debug("Bulk Load done for: " + srcPath); + } + + @Override + public void failedBulkLoad(final byte[] family, final String srcPath) throws IOException { + Path p = new Path(srcPath); + Path stageP = new Path(stagingDir, + new Path(Bytes.toString(family), p.getName())); + LOG.debug("Moving " + stageP + " back to " + p); + if(!fs.rename(stageP, p)) + throw new IOException("Failed to move HFile: " + stageP + " to " + p); + } + + /** + * Check if the path is referencing a file. + * This is mainly needed to avoid symlinks. + * @param p + * @return true if the p is a file + * @throws IOException + */ + private boolean isFile(Path p) throws IOException { + FileStatus status = fs.getFileStatus(p); + boolean isFile = !status.isDir(); + try { + isFile = isFile && !(Boolean)Methods.call(FileStatus.class, status, "isSymlink", null, null); + } catch (Exception e) { + } + return isFile; + } + } +} diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadProtocol.java b/security/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadProtocol.java new file mode 100644 index 000000000000..63f45fd3f35c --- /dev/null +++ b/security/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadProtocol.java @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.security.access; + +import org.apache.hadoop.hbase.ipc.CoprocessorProtocol; +import org.apache.hadoop.hbase.security.TokenInfo; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.hadoop.security.token.Token; + +import java.io.IOException; +import java.util.List; + +/** + * Provides a secure way to bulk load data onto HBase + * These are internal API. Bulk load should be initiated + * via {@link org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles} + * with security enabled. + */ +@TokenInfo("HBASE_AUTH_TOKEN") +public interface SecureBulkLoadProtocol extends CoprocessorProtocol { + + /** + * Prepare for bulk load. + * Will be called before bulkLoadHFiles() + * @param tableName + * @return a bulkToken which uniquely identifies the bulk session + * @throws IOException + */ + String prepareBulkLoad(byte[] tableName) throws IOException; + + /** + * Cleanup after bulk load. + * Will be called after bulkLoadHFiles(). + * @param bulkToken + * @throws IOException + */ + void cleanupBulkLoad(String bulkToken) throws IOException; + + /** + * Secure version of HRegionServer.bulkLoadHFiles(). + * @param familyPaths column family to HFile path pairs + * @param userToken requesting user's HDFS delegation token + * @param bulkToken + * @param assignSeqId + * @return + * @throws IOException + */ + boolean bulkLoadHFiles(List> familyPaths, + Token userToken, String bulkToken, boolean assignSeqNum) throws IOException; + +} diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java b/security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java index 2c3870f932e6..3e3257cd6e72 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java +++ b/security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java @@ -25,6 +25,7 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.UserProvider; import org.apache.hadoop.hbase.security.User; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; @@ -40,22 +41,59 @@ * Performs authorization checks for a given user's assigned permissions */ public class TableAuthManager { - /** Key for the user and group cache maps for globally assigned permissions */ - private static final String GLOBAL_CACHE_KEY = ".access."; + private static class PermissionCache { + /** Cache of user permissions */ + private ListMultimap userCache = ArrayListMultimap.create(); + /** Cache of group permissions */ + private ListMultimap groupCache = ArrayListMultimap.create(); + + public List getUser(String user) { + return userCache.get(user); + } + + public void putUser(String user, T perm) { + userCache.put(user, perm); + } + + public List replaceUser(String user, Iterable perms) { + return userCache.replaceValues(user, perms); + } + + public List getGroup(String group) { + return groupCache.get(group); + } + + public void putGroup(String group, T perm) { + groupCache.put(group, perm); + } + + public List replaceGroup(String group, Iterable perms) { + return groupCache.replaceValues(group, perms); + } + + /** + * Returns a combined map of user and group permissions, with group names prefixed by + * {@link AccessControlLists#GROUP_PREFIX}. + */ + public ListMultimap getAllPermissions() { + ListMultimap tmp = ArrayListMultimap.create(); + tmp.putAll(userCache); + for (String group : groupCache.keySet()) { + tmp.putAll(AccessControlLists.GROUP_PREFIX + group, groupCache.get(group)); + } + return tmp; + } + } + private static Log LOG = LogFactory.getLog(TableAuthManager.class); private static TableAuthManager instance; - /** Cache of global user permissions */ - private ListMultimap USER_CACHE = ArrayListMultimap.create(); - /** Cache of global group permissions */ - private ListMultimap GROUP_CACHE = ArrayListMultimap.create(); - - private ConcurrentSkipListMap> TABLE_USER_CACHE = - new ConcurrentSkipListMap>(Bytes.BYTES_COMPARATOR); + /** Cache of global permissions */ + private volatile PermissionCache globalCache; - private ConcurrentSkipListMap> TABLE_GROUP_CACHE = - new ConcurrentSkipListMap>(Bytes.BYTES_COMPARATOR); + private ConcurrentSkipListMap> tableCache = + new ConcurrentSkipListMap>(Bytes.BYTES_COMPARATOR); private Configuration conf; private ZKPermissionWatcher zkperms; @@ -63,23 +101,30 @@ public class TableAuthManager { private TableAuthManager(ZooKeeperWatcher watcher, Configuration conf) throws IOException { this.conf = conf; + + // initialize global permissions based on configuration + globalCache = initGlobal(conf); + this.zkperms = new ZKPermissionWatcher(watcher, this, conf); try { this.zkperms.start(); } catch (KeeperException ke) { LOG.error("ZooKeeper initialization failed", ke); } - - // initialize global permissions based on configuration - initGlobal(conf); } - private void initGlobal(Configuration conf) throws IOException { - User user = User.getCurrent(); + /** + * Returns a new {@code PermissionCache} initialized with permission assignments + * from the {@code hbase.superuser} configuration key. + */ + private PermissionCache initGlobal(Configuration conf) throws IOException { + UserProvider userProvider = UserProvider.instantiate(conf); + User user = userProvider.getCurrent(); if (user == null) { throw new IOException("Unable to obtain the current user, " + "authorization checks for internal operations will not work correctly!"); } + PermissionCache newCache = new PermissionCache(); String currentUser = user.getShortName(); // the system user is always included @@ -88,13 +133,14 @@ private void initGlobal(Configuration conf) throws IOException { if (superusers != null) { for (String name : superusers) { if (AccessControlLists.isGroupPrincipal(name)) { - GROUP_CACHE.put(AccessControlLists.getGroupName(name), + newCache.putGroup(AccessControlLists.getGroupName(name), new Permission(Permission.Action.values())); } else { - USER_CACHE.put(name, new Permission(Permission.Action.values())); + newCache.putUser(name, new Permission(Permission.Action.values())); } } } + return newCache; } public ZKPermissionWatcher getZKPermissionWatcher() { @@ -103,60 +149,71 @@ public ZKPermissionWatcher getZKPermissionWatcher() { public void refreshCacheFromWritable(byte[] table, byte[] data) throws IOException { if (data != null && data.length > 0) { - DataInput in = new DataInputStream( new ByteArrayInputStream(data) ); + DataInput in = new DataInputStream(new ByteArrayInputStream(data)); ListMultimap perms = AccessControlLists.readPermissions(in, conf); - cache(table, perms); + if (perms != null) { + if (Bytes.equals(table, AccessControlLists.ACL_GLOBAL_NAME)) { + updateGlobalCache(perms); + } else { + updateTableCache(table, perms); + } + } } else { LOG.debug("Skipping permission cache refresh because writable data is empty"); } } /** - * Updates the internal permissions cache for a single table, splitting - * the permissions listed into separate caches for users and groups to optimize - * group lookups. - * - * @param table - * @param tablePerms + * Updates the internal global permissions cache + * + * @param userPerms */ - private void cache(byte[] table, - ListMultimap tablePerms) { - // split user from group assignments so we don't have to prepend the group - // prefix every time we query for groups - ListMultimap userPerms = ArrayListMultimap.create(); - ListMultimap groupPerms = ArrayListMultimap.create(); - - if (tablePerms != null) { - for (Map.Entry entry : tablePerms.entries()) { + private void updateGlobalCache(ListMultimap userPerms) { + PermissionCache newCache = null; + try { + newCache = initGlobal(conf); + for (Map.Entry entry : userPerms.entries()) { if (AccessControlLists.isGroupPrincipal(entry.getKey())) { - groupPerms.put( - entry.getKey().substring(AccessControlLists.GROUP_PREFIX.length()), - entry.getValue()); + newCache.putGroup(AccessControlLists.getGroupName(entry.getKey()), + new Permission(entry.getValue().getActions())); } else { - userPerms.put(entry.getKey(), entry.getValue()); + newCache.putUser(entry.getKey(), new Permission(entry.getValue().getActions())); } } - TABLE_GROUP_CACHE.put(table, groupPerms); - TABLE_USER_CACHE.put(table, userPerms); + globalCache = newCache; + } catch (IOException e) { + // Never happens + LOG.error("Error occured while updating the global cache", e); } } - private List getUserPermissions(String username, byte[] table) { - ListMultimap tablePerms = TABLE_USER_CACHE.get(table); - if (tablePerms != null) { - return tablePerms.get(username); + /** + * Updates the internal permissions cache for a single table, splitting + * the permissions listed into separate caches for users and groups to optimize + * group lookups. + * + * @param table + * @param tablePerms + */ + private void updateTableCache(byte[] table, ListMultimap tablePerms) { + PermissionCache newTablePerms = new PermissionCache(); + + for (Map.Entry entry : tablePerms.entries()) { + if (AccessControlLists.isGroupPrincipal(entry.getKey())) { + newTablePerms.putGroup(AccessControlLists.getGroupName(entry.getKey()), entry.getValue()); + } else { + newTablePerms.putUser(entry.getKey(), entry.getValue()); + } } - return null; + tableCache.put(table, newTablePerms); } - private List getGroupPermissions(String groupName, byte[] table) { - ListMultimap tablePerms = TABLE_GROUP_CACHE.get(table); - if (tablePerms != null) { - return tablePerms.get(groupName); + private PermissionCache getTablePermissions(byte[] table) { + if (!tableCache.containsKey(table)) { + tableCache.putIfAbsent(table, new PermissionCache()); } - - return null; + return tableCache.get(table); } /** @@ -191,14 +248,14 @@ public boolean authorize(User user, Permission.Action action) { return false; } - if (authorize(USER_CACHE.get(user.getShortName()), action)) { + if (authorize(globalCache.getUser(user.getShortName()), action)) { return true; } String[] groups = user.getGroupNames(); if (groups != null) { for (String group : groups) { - if (authorize(GROUP_CACHE.get(group), action)) { + if (authorize(globalCache.getGroup(group), action)) { return true; } } @@ -227,18 +284,20 @@ private boolean authorize(List perms, byte[] table, byte[] fami public boolean authorize(User user, byte[] table, KeyValue kv, TablePermission.Action action) { - List userPerms = getUserPermissions( - user.getShortName(), table); - if (authorize(userPerms, table, kv, action)) { - return true; - } + PermissionCache tablePerms = tableCache.get(table); + if (tablePerms != null) { + List userPerms = tablePerms.getUser(user.getShortName()); + if (authorize(userPerms, table, kv, action)) { + return true; + } - String[] groupNames = user.getGroupNames(); - if (groupNames != null) { - for (String group : groupNames) { - List groupPerms = getGroupPermissions(group, table); - if (authorize(groupPerms, table, kv, action)) { - return true; + String[] groupNames = user.getGroupNames(); + if (groupNames != null) { + for (String group : groupNames) { + List groupPerms = tablePerms.getGroup(group); + if (authorize(groupPerms, table, kv, action)) { + return true; + } } } } @@ -267,7 +326,7 @@ private boolean authorize(List perms, byte[] table, KeyValue kv * stored user permissions. */ public boolean authorizeUser(String username, Permission.Action action) { - return authorize(USER_CACHE.get(username), action); + return authorize(globalCache.getUser(username), action); } /** @@ -291,7 +350,7 @@ public boolean authorizeUser(String username, byte[] table, byte[] family, if (authorizeUser(username, action)) { return true; } - return authorize(getUserPermissions(username, table), table, family, + return authorize(getTablePermissions(table).getUser(username), table, family, qualifier, action); } @@ -301,7 +360,7 @@ public boolean authorizeUser(String username, byte[] table, byte[] family, * permissions. */ public boolean authorizeGroup(String groupName, Permission.Action action) { - return authorize(GROUP_CACHE.get(groupName), action); + return authorize(globalCache.getGroup(groupName), action); } /** @@ -319,7 +378,7 @@ public boolean authorizeGroup(String groupName, byte[] table, byte[] family, if (authorizeGroup(groupName, action)) { return true; } - return authorize(getGroupPermissions(groupName, table), table, family, action); + return authorize(getTablePermissions(table).getGroup(groupName), table, family, action); } public boolean authorize(User user, byte[] table, byte[] family, @@ -352,24 +411,26 @@ public boolean authorize(User user, byte[] table, byte[] family, */ public boolean matchPermission(User user, byte[] table, byte[] family, TablePermission.Action action) { - List userPerms = getUserPermissions( - user.getShortName(), table); - if (userPerms != null) { - for (TablePermission p : userPerms) { - if (p.matchesFamily(table, family, action)) { - return true; + PermissionCache tablePerms = tableCache.get(table); + if (tablePerms != null) { + List userPerms = tablePerms.getUser(user.getShortName()); + if (userPerms != null) { + for (TablePermission p : userPerms) { + if (p.matchesFamily(table, family, action)) { + return true; + } } } - } - String[] groups = user.getGroupNames(); - if (groups != null) { - for (String group : groups) { - List groupPerms = getGroupPermissions(group, table); - if (groupPerms != null) { - for (TablePermission p : groupPerms) { - if (p.matchesFamily(table, family, action)) { - return true; + String[] groups = user.getGroupNames(); + if (groups != null) { + for (String group : groups) { + List groupPerms = tablePerms.getGroup(group); + if (groupPerms != null) { + for (TablePermission p : groupPerms) { + if (p.matchesFamily(table, family, action)) { + return true; + } } } } @@ -382,24 +443,26 @@ public boolean matchPermission(User user, public boolean matchPermission(User user, byte[] table, byte[] family, byte[] qualifier, TablePermission.Action action) { - List userPerms = getUserPermissions( - user.getShortName(), table); - if (userPerms != null) { - for (TablePermission p : userPerms) { - if (p.matchesFamilyQualifier(table, family, qualifier, action)) { - return true; + PermissionCache tablePerms = tableCache.get(table); + if (tablePerms != null) { + List userPerms = tablePerms.getUser(user.getShortName()); + if (userPerms != null) { + for (TablePermission p : userPerms) { + if (p.matchesFamilyQualifier(table, family, qualifier, action)) { + return true; + } } } - } - String[] groups = user.getGroupNames(); - if (groups != null) { - for (String group : groups) { - List groupPerms = getGroupPermissions(group, table); - if (groupPerms != null) { - for (TablePermission p : groupPerms) { - if (p.matchesFamilyQualifier(table, family, qualifier, action)) { - return true; + String[] groups = user.getGroupNames(); + if (groups != null) { + for (String group : groups) { + List groupPerms = tablePerms.getGroup(group); + if (groupPerms != null) { + for (TablePermission p : groupPerms) { + if (p.matchesFamilyQualifier(table, family, qualifier, action)) { + return true; + } } } } @@ -410,8 +473,7 @@ public boolean matchPermission(User user, } public void remove(byte[] table) { - TABLE_USER_CACHE.remove(table); - TABLE_GROUP_CACHE.remove(table); + tableCache.remove(table); } /** @@ -423,13 +485,9 @@ public void remove(byte[] table) { */ public void setUserPermissions(String username, byte[] table, List perms) { - ListMultimap tablePerms = TABLE_USER_CACHE.get(table); - if (tablePerms == null) { - tablePerms = ArrayListMultimap.create(); - TABLE_USER_CACHE.put(table, tablePerms); - } - tablePerms.replaceValues(username, perms); - writeToZooKeeper(table, tablePerms, TABLE_GROUP_CACHE.get(table)); + PermissionCache tablePerms = getTablePermissions(table); + tablePerms.replaceUser(username, perms); + writeToZooKeeper(table, tablePerms); } /** @@ -441,30 +499,18 @@ public void setUserPermissions(String username, byte[] table, */ public void setGroupPermissions(String group, byte[] table, List perms) { - ListMultimap tablePerms = TABLE_GROUP_CACHE.get(table); - if (tablePerms == null) { - tablePerms = ArrayListMultimap.create(); - TABLE_GROUP_CACHE.put(table, tablePerms); - } - tablePerms.replaceValues(group, perms); - writeToZooKeeper(table, TABLE_USER_CACHE.get(table), tablePerms); + PermissionCache tablePerms = getTablePermissions(table); + tablePerms.replaceGroup(group, perms); + writeToZooKeeper(table, tablePerms); } public void writeToZooKeeper(byte[] table, - ListMultimap userPerms, - ListMultimap groupPerms) { - ListMultimap tmp = ArrayListMultimap.create(); - if (userPerms != null) { - tmp.putAll(userPerms); - } - if (groupPerms != null) { - for (String group : groupPerms.keySet()) { - tmp.putAll(AccessControlLists.GROUP_PREFIX + group, - groupPerms.get(group)); - } + PermissionCache tablePerms) { + byte[] serialized = new byte[0]; + if (tablePerms != null) { + serialized = AccessControlLists.writePermissionsAsBytes(tablePerms.getAllPermissions(), conf); } - byte[] serialized = AccessControlLists.writePermissionsAsBytes(tmp, conf); - zkperms.writeToZookeeper(Bytes.toString(table), serialized); + zkperms.writeToZookeeper(table, serialized); } static Map managerMap = diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java b/security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java index 8a5c467ab4ad..fd5b755f7d79 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java +++ b/security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java @@ -40,6 +40,27 @@ public UserPermission() { super(); } + /** + * Creates a new instance for the given user. + * @param user the user + * @param assigned the list of allowed actions + */ + public UserPermission(byte[] user, Action... assigned) { + super(null, null, null, assigned); + this.user = user; + } + + /** + * Creates a new instance for the given user, + * matching the actions with the given codes. + * @param user the user + * @param actionCodes the list of allowed action codes + */ + public UserPermission(byte[] user, byte[] actionCodes) { + super(null, null, null, actionCodes); + this.user = user; + } + /** * Creates a new instance for the given user, table and column family. * @param user the user @@ -92,6 +113,14 @@ public byte[] getUser() { return user; } + /** + * Returns true if this permission describes a global user permission. + */ + public boolean isGlobal() { + byte[] tableName = getTable(); + return(tableName == null || tableName.length == 0); + } + @Override public boolean equals(Object obj) { if (!(obj instanceof UserPermission)) { diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java b/security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java index f7e8654abe04..e9cf8975968f 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java +++ b/security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java @@ -146,19 +146,35 @@ private void refreshNodes(List nodes) { * @param tableName * @param permsData */ - public void writeToZookeeper(String tableName, - byte[] permsData) { - String zkNode = - ZKUtil.joinZNode(ZKUtil.joinZNode(watcher.baseZNode, ACL_NODE), - tableName); + public void writeToZookeeper(byte[] tableName, byte[] parmsData) { + String zkNode = ZKUtil.joinZNode(watcher.baseZNode, ACL_NODE); + zkNode = ZKUtil.joinZNode(zkNode, Bytes.toString(tableName)); + try { ZKUtil.createWithParents(watcher, zkNode); - ZKUtil.updateExistingNodeData(watcher, zkNode, - permsData, -1); + ZKUtil.updateExistingNodeData(watcher, zkNode, parmsData, -1); } catch (KeeperException e) { - LOG.error("Failed updating permissions for table '" + tableName + - "'", e); + LOG.error("Failed updating permissions for table '" + + Bytes.toString(tableName) + "'", e); watcher.abort("Failed writing node "+zkNode+" to zookeeper", e); } } + + /*** + * Delete the acl notify node of table + * @param tableName + */ + public void deleteTableACLNode(final byte[] tableName) { + String zkNode = ZKUtil.joinZNode(watcher.baseZNode, ACL_NODE); + zkNode = ZKUtil.joinZNode(zkNode, Bytes.toString(tableName)); + + try { + ZKUtil.deleteNode(watcher, zkNode); + } catch (KeeperException.NoNodeException e) { + LOG.warn("No acl notify node of table '" + tableName + "'"); + } catch (KeeperException e) { + LOG.error("Failed deleting acl node of table '" + tableName + "'", e); + watcher.abort("Failed deleting node " + zkNode, e); + } + } } diff --git a/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java b/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java index 0e7e87267b53..0a3a3a67ab18 100644 --- a/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java +++ b/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java @@ -31,6 +31,7 @@ import org.apache.hadoop.hbase.security.AccessDeniedException; import org.apache.hadoop.hbase.security.User; import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod; import org.apache.hadoop.security.token.SecretManager; import org.apache.hadoop.security.token.Token; @@ -80,8 +81,7 @@ public Token getAuthenticationToken() } if (currentUser == null) { throw new AccessDeniedException("No authenticated user for request!"); - } else if (ugi.getAuthenticationMethod() != - UserGroupInformation.AuthenticationMethod.KERBEROS) { + } else if (!isAllowedDelegationTokenOp(ugi)) { LOG.warn("Token generation denied for user="+currentUser.getName() +", authMethod="+ugi.getAuthenticationMethod()); throw new AccessDeniedException( @@ -91,6 +91,23 @@ public Token getAuthenticationToken() return secretManager.generateToken(currentUser.getName()); } + /** + * @param ugi + * @return true if delegation token operation is allowed + */ + private boolean isAllowedDelegationTokenOp(UserGroupInformation ugi) throws IOException { + AuthenticationMethod authMethod = ugi.getAuthenticationMethod(); + if (authMethod == AuthenticationMethod.PROXY) { + authMethod = ugi.getRealUser().getAuthenticationMethod(); + } + if (authMethod != AuthenticationMethod.KERBEROS + && authMethod != AuthenticationMethod.KERBEROS_SSL + && authMethod != AuthenticationMethod.CERTIFICATE) { + return false; + } + return true; + } + @Override public String whoami() { return RequestContext.getRequestUserName(); diff --git a/security/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSecureLoadIncrementalHFiles.java b/security/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSecureLoadIncrementalHFiles.java new file mode 100644 index 000000000000..10aea8089b0b --- /dev/null +++ b/security/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSecureLoadIncrementalHFiles.java @@ -0,0 +1,55 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.security.access.AccessControlLists; +import org.apache.hadoop.hbase.security.access.SecureTestUtil; +import org.junit.BeforeClass; +import org.junit.experimental.categories.Category; + +/** + * Reruns TestLoadIncrementalHFiles using LoadIncrementalHFiles using secure mode. + * This suite is unable to verify the security handoff/turnover + * as miniCluster is running as system user thus has root privileges + * and delegation tokens don't seem to work on miniDFS. + * + * Thus SecureBulkload can only be completely verified by running + * integration tests against a secure cluster. This suite is still + * invaluable as it verifies the other mechanisms that need to be + * supported as part of a LoadIncrementalFiles call. + */ +@Category(LargeTests.class) +public class TestSecureLoadIncrementalHFiles extends TestLoadIncrementalHFiles{ + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + useSecureHBaseOverride = true; + // setup configuration + SecureTestUtil.enableSecurity(util.getConfiguration()); + + util.startMiniCluster(); + + // Wait for the ACL table to become available + util.waitTableAvailable(AccessControlLists.ACL_TABLE_NAME, 30000); + } + +} + diff --git a/security/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSecureLoadIncrementalHFilesSplitRecovery.java b/security/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSecureLoadIncrementalHFilesSplitRecovery.java new file mode 100644 index 000000000000..38ac80899111 --- /dev/null +++ b/security/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSecureLoadIncrementalHFilesSplitRecovery.java @@ -0,0 +1,66 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.UserProvider; +import org.apache.hadoop.hbase.security.access.AccessControlLists; +import org.apache.hadoop.hbase.security.access.SecureTestUtil; + +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + + +/** + * Reruns TestSecureLoadIncrementalHFilesSplitRecovery + * using LoadIncrementalHFiles in secure mode. + * This suite is unable to verify the security handoff/turnover + * as miniCluster is running as system user thus has root privileges + * and delegation tokens don't seem to work on miniDFS. + * + * Thus SecureBulkload can only be completely verified by running + * integration tests against a secure cluster. This suite is still + * invaluable as it verifies the other mechanisms that need to be + * supported as part of a LoadIncrementalFiles call. + */ +@Category(LargeTests.class) +public class TestSecureLoadIncrementalHFilesSplitRecovery extends TestLoadIncrementalHFilesSplitRecovery { + + //This "overrides" the parent static method + //make sure they are in sync + @BeforeClass + public static void setupCluster() throws Exception { + useSecureHBaseOverride = true; + util = new HBaseTestingUtility(); + // setup configuration + SecureTestUtil.enableSecurity(util.getConfiguration()); + util.startMiniCluster(); + + // Wait for the ACL table to become available + util.waitTableAvailable(AccessControlLists.ACL_TABLE_NAME, 30000); + } + + //Disabling this test as it does not work in secure mode + @Test + @Override + public void testBulkLoadPhaseFailure() { + } +} + diff --git a/security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java b/security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java index 1a087b6a9268..5d55760a5a7c 100644 --- a/security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java +++ b/security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java @@ -21,6 +21,7 @@ import java.io.IOException; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; import org.apache.hadoop.hbase.ipc.SecureRpcEngine; import org.apache.hadoop.hbase.security.User; @@ -32,8 +33,10 @@ public static void enableSecurity(Configuration conf) throws IOException { conf.set("hadoop.security.authorization", "false"); conf.set("hadoop.security.authentication", "simple"); conf.set("hbase.rpc.engine", SecureRpcEngine.class.getName()); - conf.set("hbase.coprocessor.master.classes", AccessController.class.getName()); - conf.set("hbase.coprocessor.region.classes", AccessController.class.getName()); + conf.set(CoprocessorHost.MASTER_COPROCESSOR_CONF_KEY, AccessController.class.getName()); + conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, AccessController.class.getName()+ + ","+SecureBulkLoadEndpoint.class.getName()); + conf.set(CoprocessorHost.REGIONSERVER_COPROCESSOR_CONF_KEY, AccessController.class.getName()); // add the process running user to superusers String currentUser = User.getCurrent().getName(); conf.set("hbase.superuser", "admin,"+currentUser); diff --git a/security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java b/security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java index 0a2cad27c73b..7f7ef55ec5ee 100644 --- a/security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java +++ b/security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java @@ -70,7 +70,7 @@ public static void setupBeforeClass() throws Exception { conf.set("hbase.superuser", conf.get("hbase.superuser", "") + String.format(",%s.hfs.0,%s.hfs.1,%s.hfs.2", baseuser, baseuser, baseuser)); TEST_UTIL.startMiniCluster(); - TEST_UTIL.waitTableAvailable(AccessControlLists.ACL_TABLE_NAME, 5000); + TEST_UTIL.waitTableAvailable(AccessControlLists.ACL_TABLE_NAME, 30000); ADMIN = User.createUserForTesting(conf, "admin", new String[]{"supergroup"}); READER = User.createUserForTesting(conf, "reader", new String[0]); @@ -95,10 +95,12 @@ public Object run() throws Exception { AccessControlLists.ACL_TABLE_NAME); AccessControllerProtocol acls = aclmeta.coprocessorProxy( AccessControllerProtocol.class, Bytes.toBytes("testtable")); - TablePermission perm = new TablePermission(TABLE, null, Permission.Action.READ); - acls.grant(Bytes.toBytes(READER.getShortName()), perm); - perm = new TablePermission(TABLE, FAMILY, PUBLIC_COL, Permission.Action.READ); - acls.grant(Bytes.toBytes(LIMITED.getShortName()), perm); + UserPermission perm = new UserPermission(Bytes.toBytes(READER.getShortName()), + TABLE, null, Permission.Action.READ); + acls.grant(perm); + perm = new UserPermission(Bytes.toBytes(LIMITED.getShortName()), + TABLE, FAMILY, PUBLIC_COL, Permission.Action.READ); + acls.grant(perm); return null; } }); diff --git a/security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java b/security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java index fe04c5a69ba6..688e30d9d835 100644 --- a/security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java +++ b/security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java @@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.security.access; import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -26,18 +27,27 @@ import java.security.PrivilegedExceptionAction; import java.util.List; import java.util.Map; +import java.util.NavigableMap; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.hbase.Coprocessor; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HRegionInfo; import org.apache.hadoop.hbase.HServerAddress; import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.UnknownRowLockException; +import org.apache.hadoop.hbase.client.Append; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HBaseAdmin; @@ -51,10 +61,26 @@ import org.apache.hadoop.hbase.coprocessor.CoprocessorException; import org.apache.hadoop.hbase.coprocessor.MasterCoprocessorEnvironment; import org.apache.hadoop.hbase.coprocessor.ObserverContext; +import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment; +import org.apache.hadoop.hbase.coprocessor.RegionServerCoprocessorEnvironment; +import org.apache.hadoop.hbase.io.hfile.CacheConfig; +import org.apache.hadoop.hbase.io.hfile.HFile; +import org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles; import org.apache.hadoop.hbase.master.MasterCoprocessorHost; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost; +import org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost; import org.apache.hadoop.hbase.security.AccessDeniedException; import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.security.access.AccessControlLists; +import org.apache.hadoop.hbase.security.access.AccessControllerProtocol; +import org.apache.hadoop.hbase.security.access.Permission; +import org.apache.hadoop.hbase.security.access.UserPermission; +import org.apache.hadoop.hbase.security.access.Permission.Action; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.JVMClusterUtil; +import org.apache.hadoop.ipc.RemoteException; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -65,66 +91,109 @@ * levels of authorized users. */ @Category(LargeTests.class) +@SuppressWarnings("rawtypes") public class TestAccessController { - private static Log LOG = LogFactory.getLog(TestAccessController.class); + private static final Log LOG = LogFactory.getLog(TestAccessController.class); private static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); private static Configuration conf; // user with all permissions private static User SUPERUSER; - // table owner user - private static User USER_OWNER; + // user granted with all global permission + private static User USER_ADMIN; // user with rw permissions private static User USER_RW; + // user with rw permissions on table. + private static User USER_RW_ON_TABLE; // user with read-only permissions private static User USER_RO; + // user is table owner. will have all permissions on table + private static User USER_OWNER; + // user with create table permissions alone + private static User USER_CREATE; // user with no permissions private static User USER_NONE; private static byte[] TEST_TABLE = Bytes.toBytes("testtable"); + private static byte[] TEST_TABLE2 = Bytes.toBytes("testtable2"); private static byte[] TEST_FAMILY = Bytes.toBytes("f1"); private static MasterCoprocessorEnvironment CP_ENV; + private static RegionCoprocessorEnvironment RCP_ENV; + private static RegionServerCoprocessorEnvironment RSCP_ENV; private static AccessController ACCESS_CONTROLLER; @BeforeClass public static void setupBeforeClass() throws Exception { // setup configuration conf = TEST_UTIL.getConfiguration(); + conf.set("hbase.master.hfilecleaner.plugins", + "org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner," + + "org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner"); + conf.set("hbase.master.logcleaner.plugins", + "org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner"); SecureTestUtil.enableSecurity(conf); TEST_UTIL.startMiniCluster(); - MasterCoprocessorHost cpHost = TEST_UTIL.getMiniHBaseCluster() - .getMaster().getCoprocessorHost(); + MasterCoprocessorHost cpHost = TEST_UTIL.getMiniHBaseCluster().getMaster().getCoprocessorHost(); cpHost.load(AccessController.class, Coprocessor.PRIORITY_HIGHEST, conf); - ACCESS_CONTROLLER = (AccessController)cpHost.findCoprocessor( - AccessController.class.getName()); + ACCESS_CONTROLLER = (AccessController) cpHost.findCoprocessor(AccessController.class.getName()); CP_ENV = cpHost.createEnvironment(AccessController.class, ACCESS_CONTROLLER, - Coprocessor.PRIORITY_HIGHEST, 1, conf); + Coprocessor.PRIORITY_HIGHEST, 1, conf); + RegionServerCoprocessorHost rsHost = TEST_UTIL.getMiniHBaseCluster().getRegionServer(0) + .getCoprocessorHost(); + RSCP_ENV = rsHost.createEnvironment(AccessController.class, ACCESS_CONTROLLER, + Coprocessor.PRIORITY_HIGHEST, 1, conf); + + // Wait for the ACL table to become available + TEST_UTIL.waitTableAvailable(AccessControlLists.ACL_TABLE_NAME, 30000); // create a set of test users - SUPERUSER = User.createUserForTesting(conf, "admin", new String[]{"supergroup"}); - USER_OWNER = User.createUserForTesting(conf, "owner", new String[0]); + SUPERUSER = User.createUserForTesting(conf, "admin", new String[] { "supergroup" }); + USER_ADMIN = User.createUserForTesting(conf, "admin2", new String[0]); USER_RW = User.createUserForTesting(conf, "rwuser", new String[0]); USER_RO = User.createUserForTesting(conf, "rouser", new String[0]); + USER_RW_ON_TABLE = User.createUserForTesting(conf, "rwuser_1", new String[0]); + USER_OWNER = User.createUserForTesting(conf, "owner", new String[0]); + USER_CREATE = User.createUserForTesting(conf, "tbl_create", new String[0]); USER_NONE = User.createUserForTesting(conf, "nouser", new String[0]); HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); HTableDescriptor htd = new HTableDescriptor(TEST_TABLE); htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); - htd.setOwnerString(USER_OWNER.getShortName()); + htd.setOwner(USER_OWNER); admin.createTable(htd); + TEST_UTIL.waitTableEnabled(TEST_TABLE, 5000); + + HRegion region = TEST_UTIL.getHBaseCluster().getRegions(TEST_TABLE).get(0); + RegionCoprocessorHost rcpHost = region.getCoprocessorHost(); + RCP_ENV = rcpHost.createEnvironment(AccessController.class, ACCESS_CONTROLLER, + Coprocessor.PRIORITY_HIGHEST, 1, conf); // initilize access control - HTable meta = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); - AccessControllerProtocol protocol = - meta.coprocessorProxy(AccessControllerProtocol.class, TEST_TABLE); - protocol.grant(Bytes.toBytes(USER_RW.getShortName()), - new TablePermission(TEST_TABLE, TEST_FAMILY, Permission.Action.READ, - Permission.Action.WRITE)); - - protocol.grant(Bytes.toBytes(USER_RO.getShortName()), - new TablePermission(TEST_TABLE, TEST_FAMILY, Permission.Action.READ)); + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + TEST_TABLE); + + protocol.grant(new UserPermission(Bytes.toBytes(USER_ADMIN.getShortName()), + Permission.Action.ADMIN, Permission.Action.CREATE, Permission.Action.READ, + Permission.Action.WRITE)); + + protocol.grant(new UserPermission(Bytes.toBytes(USER_RW.getShortName()), TEST_TABLE, + TEST_FAMILY, Permission.Action.READ, Permission.Action.WRITE)); + + protocol.grant(new UserPermission(Bytes.toBytes(USER_RO.getShortName()), TEST_TABLE, + TEST_FAMILY, Permission.Action.READ)); + + protocol.grant(new UserPermission(Bytes.toBytes(USER_CREATE.getShortName()), TEST_TABLE, null, + Permission.Action.CREATE)); + + protocol.grant(new UserPermission(Bytes.toBytes(USER_RW_ON_TABLE.getShortName()), TEST_TABLE, + null, Permission.Action.READ, Permission.Action.WRITE)); + } finally { + acl.close(); + } } @AfterClass @@ -132,54 +201,70 @@ public static void tearDownAfterClass() throws Exception { TEST_UTIL.shutdownMiniCluster(); } - public void verifyAllowed(User user, PrivilegedExceptionAction action) - throws Exception { - try { - user.runAs(action); - } catch (AccessDeniedException ade) { - fail("Expected action to pass for user '" + user.getShortName() + - "' but was denied"); + public void verifyAllowed(User user, PrivilegedExceptionAction... actions) throws Exception { + for (PrivilegedExceptionAction action : actions) { + try { + user.runAs(action); + } catch (AccessDeniedException ade) { + fail("Expected action to pass for user '" + user.getShortName() + "' but was denied"); + } catch (UnknownRowLockException exp){ + //expected + } } } - public void verifyAllowed(PrivilegedExceptionAction action, User... users) - throws Exception { + public void verifyAllowed(PrivilegedExceptionAction action, User... users) throws Exception { for (User user : users) { verifyAllowed(user, action); } } - public void verifyDenied(User user, PrivilegedExceptionAction action) - throws Exception { - try { - user.runAs(action); - fail("Expected AccessDeniedException for user '" + user.getShortName() + "'"); - } catch (RetriesExhaustedWithDetailsException e) { - // in case of batch operations, and put, the client assembles a - // RetriesExhaustedWithDetailsException instead of throwing an - // AccessDeniedException - boolean isAccessDeniedException = false; - for ( Throwable ex : e.getCauses()) { - if (ex instanceof AccessDeniedException) { - isAccessDeniedException = true; - break; + public void verifyDenied(User user, PrivilegedExceptionAction... actions) throws Exception { + for (PrivilegedExceptionAction action : actions) { + try { + user.runAs(action); + fail("Expected AccessDeniedException for user '" + user.getShortName() + "'"); + } catch (AccessDeniedException ade) { + // expected result + } catch (IOException e) { + boolean isAccessDeniedException = false; + if(e instanceof RetriesExhaustedWithDetailsException) { + // in case of batch operations, and put, the client assembles a + // RetriesExhaustedWithDetailsException instead of throwing an + // AccessDeniedException + for(Throwable ex : ((RetriesExhaustedWithDetailsException) e).getCauses()) { + if (ex instanceof AccessDeniedException) { + isAccessDeniedException = true; + break; + } + } + } + else { + if (e instanceof RemoteException) { + e = ((RemoteException)e).unwrapRemoteException(); + } + // For doBulkLoad calls AccessDeniedException + // is buried in the stack trace + Throwable ex = e; + do { + if (ex instanceof AccessDeniedException) { + isAccessDeniedException = true; + break; + } + } while((ex = ex.getCause()) != null); + } + if (!isAccessDeniedException) { + fail("Not receiving AccessDeniedException for user '" + user.getShortName() + "'"); } } - if (!isAccessDeniedException ) { - fail("Not receiving AccessDeniedException for user '" + - user.getShortName() + "'"); - } - } catch (AccessDeniedException ade) { - // expected result } } - public void verifyDenied(PrivilegedExceptionAction action, User... users) - throws Exception { - for (User user : users) { - verifyDenied(user, action); - } + public void verifyDenied(PrivilegedExceptionAction action, User... users) throws Exception { + for (User user : users) { + verifyDenied(user, action); } + } @Test public void testTableCreate() throws Exception { @@ -187,61 +272,47 @@ public void testTableCreate() throws Exception { public Object run() throws Exception { HTableDescriptor htd = new HTableDescriptor("testnewtable"); htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); - ACCESS_CONTROLLER.preCreateTable( - ObserverContext.createAndPrepare(CP_ENV, null), htd, null); + ACCESS_CONTROLLER.preCreateTable(ObserverContext.createAndPrepare(CP_ENV, null), htd, null); return null; } }; // verify that superuser can create tables - verifyAllowed(SUPERUSER, createTable); + verifyAllowed(createTable, SUPERUSER, USER_ADMIN); // all others should be denied - verifyDenied(USER_OWNER, createTable); - verifyDenied(USER_RW, createTable); - verifyDenied(USER_RO, createTable); - verifyDenied(USER_NONE, createTable); + verifyDenied(createTable, USER_CREATE, USER_RW, USER_RO, USER_NONE); } @Test public void testTableModify() throws Exception { - PrivilegedExceptionAction disableTable = new PrivilegedExceptionAction() { + PrivilegedExceptionAction modifyTable = new PrivilegedExceptionAction() { public Object run() throws Exception { HTableDescriptor htd = new HTableDescriptor(TEST_TABLE); htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); - htd.addFamily(new HColumnDescriptor("fam_"+User.getCurrent().getShortName())); - ACCESS_CONTROLLER.preModifyTable(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE, htd); + htd.addFamily(new HColumnDescriptor("fam_" + User.getCurrent().getShortName())); + ACCESS_CONTROLLER.preModifyTable(ObserverContext.createAndPrepare(CP_ENV, null), + TEST_TABLE, htd); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, disableTable); - verifyDenied(USER_RW, disableTable); - verifyDenied(USER_RO, disableTable); - verifyDenied(USER_NONE, disableTable); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, disableTable); + verifyAllowed(modifyTable, SUPERUSER, USER_ADMIN, USER_CREATE, USER_OWNER); + verifyDenied(modifyTable, USER_RW, USER_RO, USER_NONE); } @Test public void testTableDelete() throws Exception { - PrivilegedExceptionAction disableTable = new PrivilegedExceptionAction() { + PrivilegedExceptionAction deleteTable = new PrivilegedExceptionAction() { public Object run() throws Exception { - ACCESS_CONTROLLER.preDeleteTable(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE); + ACCESS_CONTROLLER + .preDeleteTable(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, disableTable); - verifyDenied(USER_RW, disableTable); - verifyDenied(USER_RO, disableTable); - verifyDenied(USER_NONE, disableTable); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, disableTable); + verifyAllowed(deleteTable, SUPERUSER, USER_ADMIN, USER_CREATE, USER_OWNER); + verifyDenied(deleteTable, USER_RW, USER_RO, USER_NONE); } @Test @@ -249,19 +320,14 @@ public void testAddColumn() throws Exception { final HColumnDescriptor hcd = new HColumnDescriptor("fam_new"); PrivilegedExceptionAction action = new PrivilegedExceptionAction() { public Object run() throws Exception { - ACCESS_CONTROLLER.preAddColumn(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE, hcd); + ACCESS_CONTROLLER.preAddColumn(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE, + hcd); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_CREATE, USER_OWNER); + verifyDenied(action, USER_RW, USER_RO, USER_NONE); } @Test @@ -270,151 +336,136 @@ public void testModifyColumn() throws Exception { hcd.setMaxVersions(10); PrivilegedExceptionAction action = new PrivilegedExceptionAction() { public Object run() throws Exception { - ACCESS_CONTROLLER.preModifyColumn(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE, hcd); + ACCESS_CONTROLLER.preModifyColumn(ObserverContext.createAndPrepare(CP_ENV, null), + TEST_TABLE, hcd); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_CREATE, USER_OWNER); + verifyDenied(action, USER_RW, USER_RO, USER_NONE); } @Test public void testDeleteColumn() throws Exception { PrivilegedExceptionAction action = new PrivilegedExceptionAction() { public Object run() throws Exception { - ACCESS_CONTROLLER.preDeleteColumn(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE, TEST_FAMILY); + ACCESS_CONTROLLER.preDeleteColumn(ObserverContext.createAndPrepare(CP_ENV, null), + TEST_TABLE, TEST_FAMILY); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_CREATE, USER_OWNER); + verifyDenied(action, USER_RW, USER_RO, USER_NONE); } @Test public void testTableDisable() throws Exception { PrivilegedExceptionAction disableTable = new PrivilegedExceptionAction() { public Object run() throws Exception { - ACCESS_CONTROLLER.preDisableTable(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE); + ACCESS_CONTROLLER.preDisableTable(ObserverContext.createAndPrepare(CP_ENV, null), + TEST_TABLE); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, disableTable); - verifyDenied(USER_RW, disableTable); - verifyDenied(USER_RO, disableTable); - verifyDenied(USER_NONE, disableTable); + PrivilegedExceptionAction disableAclTable = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preDisableTable(ObserverContext.createAndPrepare(CP_ENV, null), + AccessControlLists.ACL_TABLE_NAME); + return null; + } + }; - // verify that superuser can create tables - verifyAllowed(SUPERUSER, disableTable); + verifyAllowed(disableTable, SUPERUSER, USER_ADMIN, USER_CREATE, USER_OWNER); + verifyDenied(disableTable, USER_RW, USER_RO, USER_NONE); + + // No user should be allowed to disable _acl_ table + verifyDenied(disableAclTable, SUPERUSER, USER_ADMIN, USER_CREATE, USER_OWNER, USER_RW, USER_RO); } @Test public void testTableEnable() throws Exception { PrivilegedExceptionAction enableTable = new PrivilegedExceptionAction() { public Object run() throws Exception { - ACCESS_CONTROLLER.preEnableTable(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE); + ACCESS_CONTROLLER + .preEnableTable(ObserverContext.createAndPrepare(CP_ENV, null), TEST_TABLE); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, enableTable); - verifyDenied(USER_RW, enableTable); - verifyDenied(USER_RO, enableTable); - verifyDenied(USER_NONE, enableTable); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, enableTable); + verifyAllowed(enableTable, SUPERUSER, USER_ADMIN, USER_CREATE, USER_OWNER); + verifyDenied(enableTable, USER_RW, USER_RO, USER_NONE); } @Test public void testMove() throws Exception { + Map regions; HTable table = new HTable(TEST_UTIL.getConfiguration(), TEST_TABLE); - Map regions = table.getRegionsInfo(); - final Map.Entry firstRegion = - regions.entrySet().iterator().next(); + try { + regions = table.getRegionsInfo(); + } finally { + table.close(); + } + final Map.Entry firstRegion = regions.entrySet().iterator().next(); final ServerName server = TEST_UTIL.getHBaseCluster().getRegionServer(0).getServerName(); PrivilegedExceptionAction action = new PrivilegedExceptionAction() { public Object run() throws Exception { ACCESS_CONTROLLER.preMove(ObserverContext.createAndPrepare(CP_ENV, null), - firstRegion.getKey(), server, server); + firstRegion.getKey(), server, server); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER); + verifyDenied(action, USER_CREATE, USER_RW, USER_RO, USER_NONE); } @Test public void testAssign() throws Exception { + Map regions; HTable table = new HTable(TEST_UTIL.getConfiguration(), TEST_TABLE); - Map regions = table.getRegionsInfo(); - final Map.Entry firstRegion = - regions.entrySet().iterator().next(); + try { + regions = table.getRegionsInfo(); + } finally { + table.close(); + } + final Map.Entry firstRegion = regions.entrySet().iterator().next(); PrivilegedExceptionAction action = new PrivilegedExceptionAction() { public Object run() throws Exception { ACCESS_CONTROLLER.preAssign(ObserverContext.createAndPrepare(CP_ENV, null), - firstRegion.getKey()); + firstRegion.getKey()); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER); + verifyDenied(action, USER_CREATE, USER_RW, USER_RO, USER_NONE); } @Test public void testUnassign() throws Exception { + Map regions; HTable table = new HTable(TEST_UTIL.getConfiguration(), TEST_TABLE); - Map regions = table.getRegionsInfo(); - final Map.Entry firstRegion = - regions.entrySet().iterator().next(); + try { + regions = table.getRegionsInfo(); + } finally { + table.close(); + } + final Map.Entry firstRegion = regions.entrySet().iterator().next(); PrivilegedExceptionAction action = new PrivilegedExceptionAction() { public Object run() throws Exception { ACCESS_CONTROLLER.preUnassign(ObserverContext.createAndPrepare(CP_ENV, null), - firstRegion.getKey(), false); + firstRegion.getKey(), false); return null; } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER); + verifyDenied(action, USER_CREATE, USER_RW, USER_RO, USER_NONE); } @Test @@ -426,14 +477,8 @@ public Object run() throws Exception { } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN); + verifyDenied(action, USER_CREATE, USER_OWNER, USER_RW, USER_RO, USER_NONE); } @Test @@ -445,14 +490,8 @@ public Object run() throws Exception { } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN); + verifyDenied(action, USER_CREATE, USER_OWNER, USER_RW, USER_RO, USER_NONE); } @Test @@ -464,14 +503,8 @@ public Object run() throws Exception { } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN); + verifyDenied(action, USER_CREATE, USER_OWNER, USER_RW, USER_RO, USER_NONE); } @Test @@ -483,36 +516,75 @@ public Object run() throws Exception { } }; - // all others should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_RO, action); - verifyDenied(USER_NONE, action); - - // verify that superuser can create tables - verifyAllowed(SUPERUSER, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN); + verifyDenied(action, USER_CREATE, USER_OWNER, USER_RW, USER_RO, USER_NONE); } private void verifyWrite(PrivilegedExceptionAction action) throws Exception { - // should be denied - verifyDenied(USER_NONE, action); - verifyDenied(USER_RO, action); - - // should be allowed - verifyAllowed(SUPERUSER, action); - verifyAllowed(USER_OWNER, action); - verifyAllowed(USER_RW, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER, USER_RW); + verifyDenied(action, USER_NONE, USER_CREATE, USER_RO); + } + + @Test + public void testSplit() throws Exception { + PrivilegedExceptionAction action = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preSplit(ObserverContext.createAndPrepare(RCP_ENV, null)); + return null; + } + }; + + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER); + verifyDenied(action, USER_CREATE, USER_RW, USER_RO, USER_NONE); + } + + @Test + public void testFlush() throws Exception { + PrivilegedExceptionAction action = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preFlush(ObserverContext.createAndPrepare(RCP_ENV, null)); + return null; + } + }; + + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER, USER_CREATE); + verifyDenied(action, USER_RW, USER_RO, USER_NONE); + } + + @Test + public void testCompact() throws Exception { + PrivilegedExceptionAction action = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preCompact(ObserverContext.createAndPrepare(RCP_ENV, null), null, null); + return null; + } + }; + + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER, USER_CREATE); + verifyDenied(action, USER_RW, USER_RO, USER_NONE); + } + + @Test + public void testPreCompactSelection() throws Exception { + PrivilegedExceptionAction action = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preCompactSelection(ObserverContext.createAndPrepare(RCP_ENV, null), null, null); + return null; + } + }; + + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER); + verifyDenied(action, USER_CREATE, USER_RW, USER_RO, USER_NONE); } private void verifyRead(PrivilegedExceptionAction action) throws Exception { - // should be denied - verifyDenied(USER_NONE, action); - - // should be allowed - verifyAllowed(SUPERUSER, action); - verifyAllowed(USER_OWNER, action); - verifyAllowed(USER_RW, action); - verifyAllowed(USER_RO, action); + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER, USER_RW, USER_RO); + verifyDenied(action, USER_NONE, USER_CREATE); + } + + private void verifyReadWrite(PrivilegedExceptionAction action) throws Exception { + verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_OWNER, USER_RW); + verifyDenied(action, USER_NONE, USER_CREATE, USER_RO); } @Test @@ -523,7 +595,11 @@ public Object run() throws Exception { Get g = new Get(Bytes.toBytes("random_row")); g.addFamily(TEST_FAMILY); HTable t = new HTable(conf, TEST_TABLE); - t.get(g); + try { + t.get(g); + } finally { + t.close(); + } return null; } }; @@ -536,14 +612,18 @@ public Object run() throws Exception { s.addFamily(TEST_FAMILY); HTable table = new HTable(conf, TEST_TABLE); - ResultScanner scanner = table.getScanner(s); try { - for (Result r = scanner.next(); r != null; r = scanner.next()) { - // do nothing + ResultScanner scanner = table.getScanner(s); + try { + for (Result r = scanner.next(); r != null; r = scanner.next()) { + // do nothing + } + } catch (IOException e) { + } finally { + scanner.close(); } - } catch (IOException e) { } finally { - scanner.close(); + table.close(); } return null; } @@ -560,7 +640,11 @@ public Object run() throws Exception { Put p = new Put(Bytes.toBytes("random_row")); p.add(TEST_FAMILY, Bytes.toBytes("Qualifier"), Bytes.toBytes(1)); HTable t = new HTable(conf, TEST_TABLE); - t.put(p); + try { + t.put(p); + } finally { + t.close(); + } return null; } }; @@ -572,7 +656,11 @@ public Object run() throws Exception { Delete d = new Delete(Bytes.toBytes("random_row")); d.deleteFamily(TEST_FAMILY); HTable t = new HTable(conf, TEST_TABLE); - t.delete(d); + try { + t.delete(d); + } finally { + t.close(); + } return null; } }; @@ -584,15 +672,247 @@ public Object run() throws Exception { Increment inc = new Increment(Bytes.toBytes("random_row")); inc.addColumn(TEST_FAMILY, Bytes.toBytes("Qualifier"), 1); HTable t = new HTable(conf, TEST_TABLE); - t.increment(inc); + try { + t.increment(inc); + } finally { + t.close(); + } return null; } }; verifyWrite(incrementAction); } + @Test + public void testReadWrite() throws Exception { + // action for checkAndDelete + PrivilegedExceptionAction checkAndDeleteAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + Delete d = new Delete(Bytes.toBytes("random_row")); + d.deleteFamily(TEST_FAMILY); + HTable t = new HTable(conf, TEST_TABLE); + try { + t.checkAndDelete(Bytes.toBytes("random_row"), TEST_FAMILY, Bytes.toBytes("q"), + Bytes.toBytes("test_value"), d); + } finally { + t.close(); + } + return null; + } + }; + verifyReadWrite(checkAndDeleteAction); + + // action for checkAndPut() + PrivilegedExceptionAction checkAndPut = new PrivilegedExceptionAction() { + public Object run() throws Exception { + Put p = new Put(Bytes.toBytes("random_row")); + p.add(TEST_FAMILY, Bytes.toBytes("Qualifier"), Bytes.toBytes(1)); + HTable t = new HTable(conf, TEST_TABLE); + try { + t.checkAndPut(Bytes.toBytes("random_row"), TEST_FAMILY, Bytes.toBytes("q"), + Bytes.toBytes("test_value"), p); + } finally { + t.close(); + } + return null; + } + }; + verifyReadWrite(checkAndPut); + } + + @Test + public void testBulkLoad() throws Exception { + FileSystem fs = TEST_UTIL.getTestFileSystem(); + final Path dir = TEST_UTIL.getDataTestDir("testBulkLoad"); + fs.mkdirs(dir); + //need to make it globally writable + //so users creating HFiles have write permissions + fs.setPermission(dir, FsPermission.valueOf("-rwxrwxrwx")); + + PrivilegedExceptionAction bulkLoadAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + int numRows = 3; + + //Making the assumption that the test table won't split between the range + byte[][][] hfileRanges = {{{(byte)0}, {(byte)9}}}; + + Path bulkLoadBasePath = new Path(dir, new Path(User.getCurrent().getName())); + new BulkLoadHelper(bulkLoadBasePath) + .bulkLoadHFile(TEST_TABLE, TEST_FAMILY, Bytes.toBytes("q"), hfileRanges, numRows); + + return null; + } + }; + verifyAllowed(bulkLoadAction, SUPERUSER, USER_ADMIN, USER_OWNER, USER_CREATE); + verifyDenied(bulkLoadAction, USER_RW, USER_RO, USER_NONE); + + // Reinit after the bulk upload + TEST_UTIL.getHBaseAdmin().disableTable(TEST_TABLE); + TEST_UTIL.getHBaseAdmin().enableTable(TEST_TABLE); + } + + public class BulkLoadHelper { + private final FileSystem fs; + private final Path loadPath; + private final Configuration conf; + + public BulkLoadHelper(Path loadPath) throws IOException { + fs = TEST_UTIL.getTestFileSystem(); + conf = TEST_UTIL.getConfiguration(); + loadPath = loadPath.makeQualified(fs); + this.loadPath = loadPath; + } + + private void createHFile(Path path, + byte[] family, byte[] qualifier, + byte[] startKey, byte[] endKey, int numRows) throws IOException { + + HFile.Writer writer = null; + long now = System.currentTimeMillis(); + try { + writer = HFile.getWriterFactory(conf, new CacheConfig(conf)) + .withPath(fs, path) + .withComparator(KeyValue.KEY_COMPARATOR) + .create(); + // subtract 2 since numRows doesn't include boundary keys + for (byte[] key : Bytes.iterateOnSplits(startKey, endKey, true, numRows-2)) { + KeyValue kv = new KeyValue(key, family, qualifier, now, key); + writer.append(kv); + } + } finally { + if(writer != null) + writer.close(); + } + } + + private void bulkLoadHFile( + byte[] tableName, + byte[] family, + byte[] qualifier, + byte[][][] hfileRanges, + int numRowsPerRange) throws Exception { + + Path familyDir = new Path(loadPath, Bytes.toString(family)); + fs.mkdirs(familyDir); + int hfileIdx = 0; + for (byte[][] range : hfileRanges) { + byte[] from = range[0]; + byte[] to = range[1]; + createHFile(new Path(familyDir, "hfile_"+(hfileIdx++)), + family, qualifier, from, to, numRowsPerRange); + } + //set global read so RegionServer can move it + setPermission(loadPath, FsPermission.valueOf("-rwxrwxrwx")); + + HTable table = new HTable(conf, tableName); + try { + TEST_UTIL.waitTableAvailable(tableName, 30000); + conf.setBoolean("hbase.mapreduce.bulkload.assign.sequenceNumbers", true); + LoadIncrementalHFiles loader = new LoadIncrementalHFiles(conf); + loader.doBulkLoad(loadPath, table); + } finally { + table.close(); + } + } + + public void setPermission(Path dir, FsPermission perm) throws IOException { + if(!fs.getFileStatus(dir).isDir()) { + fs.setPermission(dir,perm); + } + else { + for(FileStatus el : fs.listStatus(dir)) { + fs.setPermission(el.getPath(), perm); + setPermission(el.getPath() , perm); + } + } + } + } + + @Test + public void testAppend() throws Exception { + + PrivilegedExceptionAction appendAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + byte[] row = Bytes.toBytes("random_row"); + byte[] qualifier = Bytes.toBytes("q"); + Put put = new Put(row); + put.add(TEST_FAMILY, qualifier, Bytes.toBytes(1)); + Append append = new Append(row); + append.add(TEST_FAMILY, qualifier, Bytes.toBytes(2)); + HTable t = new HTable(conf, TEST_TABLE); + try { + t.put(put); + t.append(append); + } finally { + t.close(); + } + return null; + } + }; + + verifyAllowed(appendAction, SUPERUSER, USER_ADMIN, USER_OWNER, USER_RW); + verifyDenied(appendAction, USER_CREATE, USER_RO, USER_NONE); + } + @Test public void testGrantRevoke() throws Exception { + + PrivilegedExceptionAction grantAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + TEST_TABLE); + protocol.grant(new UserPermission(Bytes.toBytes(USER_RO.getShortName()), TEST_TABLE, + TEST_FAMILY, (byte[]) null, Action.READ)); + } finally { + acl.close(); + } + return null; + } + }; + + PrivilegedExceptionAction revokeAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + TEST_TABLE); + protocol.revoke(new UserPermission(Bytes.toBytes(USER_RO.getShortName()), TEST_TABLE, + TEST_FAMILY, (byte[]) null, Action.READ)); + } finally { + acl.close(); + } + return null; + } + }; + + PrivilegedExceptionAction getPermissionsAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + TEST_TABLE); + protocol.getUserPermissions(TEST_TABLE); + } finally { + acl.close(); + } + return null; + } + }; + + verifyAllowed(grantAction, SUPERUSER, USER_ADMIN, USER_OWNER); + verifyDenied(grantAction, USER_CREATE, USER_RW, USER_RO, USER_NONE); + + verifyAllowed(revokeAction, SUPERUSER, USER_ADMIN, USER_OWNER); + verifyDenied(revokeAction, USER_CREATE, USER_RW, USER_RO, USER_NONE); + + verifyAllowed(getPermissionsAction, SUPERUSER, USER_ADMIN, USER_OWNER); + verifyDenied(getPermissionsAction, USER_CREATE, USER_RW, USER_RO, USER_NONE); + } + + @Test + public void testPostGrantRevoke() throws Exception { final byte[] tableName = Bytes.toBytes("TempTable"); final byte[] family1 = Bytes.toBytes("f1"); final byte[] family2 = Bytes.toBytes("f2"); @@ -607,18 +927,13 @@ public void testGrantRevoke() throws Exception { HTableDescriptor htd = new HTableDescriptor(tableName); htd.addFamily(new HColumnDescriptor(family1)); htd.addFamily(new HColumnDescriptor(family2)); - htd.setOwnerString(USER_OWNER.getShortName()); admin.createTable(htd); // create temp users - User user = User.createUserForTesting(TEST_UTIL.getConfiguration(), - "user", new String[0]); - - // perms only stored against the first region - HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); - AccessControllerProtocol protocol = - acl.coprocessorProxy(AccessControllerProtocol.class, - tableName); + User tblUser = User + .createUserForTesting(TEST_UTIL.getConfiguration(), "tbluser", new String[0]); + User gblUser = User + .createUserForTesting(TEST_UTIL.getConfiguration(), "gbluser", new String[0]); // prepare actions: PrivilegedExceptionAction putActionAll = new PrivilegedExceptionAction() { @@ -627,7 +942,11 @@ public Object run() throws Exception { p.add(family1, qualifier, Bytes.toBytes("v1")); p.add(family2, qualifier, Bytes.toBytes("v2")); HTable t = new HTable(conf, tableName); - t.put(p); + try { + t.put(p); + } finally { + t.close(); + } return null; } }; @@ -636,7 +955,11 @@ public Object run() throws Exception { Put p = new Put(Bytes.toBytes("a")); p.add(family1, qualifier, Bytes.toBytes("v1")); HTable t = new HTable(conf, tableName); - t.put(p); + try { + t.put(p); + } finally { + t.close(); + } return null; } }; @@ -645,7 +968,11 @@ public Object run() throws Exception { Put p = new Put(Bytes.toBytes("a")); p.add(family2, qualifier, Bytes.toBytes("v2")); HTable t = new HTable(conf, tableName); - t.put(p); + try { + t.put(p); + } finally { + t.close(); + } return null; } }; @@ -655,7 +982,11 @@ public Object run() throws Exception { g.addFamily(family1); g.addFamily(family2); HTable t = new HTable(conf, tableName); - t.get(g); + try { + t.get(g); + } finally { + t.close(); + } return null; } }; @@ -664,7 +995,11 @@ public Object run() throws Exception { Get g = new Get(Bytes.toBytes("random_row")); g.addFamily(family1); HTable t = new HTable(conf, tableName); - t.get(g); + try { + t.get(g); + } finally { + t.close(); + } return null; } }; @@ -673,7 +1008,11 @@ public Object run() throws Exception { Get g = new Get(Bytes.toBytes("random_row")); g.addFamily(family2); HTable t = new HTable(conf, tableName); - t.get(g); + try { + t.get(g); + } finally { + t.close(); + } return null; } }; @@ -683,7 +1022,11 @@ public Object run() throws Exception { d.deleteFamily(family1); d.deleteFamily(family2); HTable t = new HTable(conf, tableName); - t.delete(d); + try { + t.delete(d); + } finally { + t.close(); + } return null; } }; @@ -692,7 +1035,11 @@ public Object run() throws Exception { Delete d = new Delete(Bytes.toBytes("random_row")); d.deleteFamily(family1); HTable t = new HTable(conf, tableName); - t.delete(d); + try { + t.delete(d); + } finally { + t.close(); + } return null; } }; @@ -701,140 +1048,180 @@ public Object run() throws Exception { Delete d = new Delete(Bytes.toBytes("random_row")); d.deleteFamily(family2); HTable t = new HTable(conf, tableName); - t.delete(d); + try { + t.delete(d); + } finally { + t.close(); + } return null; } }; // initial check: - verifyDenied(user, getActionAll); - verifyDenied(user, getAction1); - verifyDenied(user, getAction2); + verifyDenied(tblUser, getActionAll, getAction1, getAction2); + verifyDenied(tblUser, putActionAll, putAction1, putAction2); + verifyDenied(tblUser, deleteActionAll, deleteAction1, deleteAction2); - verifyDenied(user, putActionAll); - verifyDenied(user, putAction1); - verifyDenied(user, putAction2); - - verifyDenied(user, deleteActionAll); - verifyDenied(user, deleteAction1); - verifyDenied(user, deleteAction2); + verifyDenied(gblUser, getActionAll, getAction1, getAction2); + verifyDenied(gblUser, putActionAll, putAction1, putAction2); + verifyDenied(gblUser, deleteActionAll, deleteAction1, deleteAction2); // grant table read permission - protocol.grant(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, null, Permission.Action.READ)); - Thread.sleep(100); - // check - verifyAllowed(user, getActionAll); - verifyAllowed(user, getAction1); - verifyAllowed(user, getAction2); - - verifyDenied(user, putActionAll); - verifyDenied(user, putAction1); - verifyDenied(user, putAction2); + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(new UserPermission(Bytes.toBytes(tblUser.getShortName()), tableName, null, + Permission.Action.READ)); + protocol.grant(new UserPermission(Bytes.toBytes(gblUser.getShortName()), + Permission.Action.READ)); + } finally { + acl.close(); + } - verifyDenied(user, deleteActionAll); - verifyDenied(user, deleteAction1); - verifyDenied(user, deleteAction2); + Thread.sleep(100); + + // check + verifyAllowed(tblUser, getActionAll, getAction1, getAction2); + verifyDenied(tblUser, putActionAll, putAction1, putAction2); + verifyDenied(tblUser, deleteActionAll, deleteAction1, deleteAction2); + + verifyAllowed(gblUser, getActionAll, getAction1, getAction2); + verifyDenied(gblUser, putActionAll, putAction1, putAction2); + verifyDenied(gblUser, deleteActionAll, deleteAction1, deleteAction2); // grant table write permission - protocol.grant(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, null, Permission.Action.WRITE)); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(new UserPermission(Bytes.toBytes(tblUser.getShortName()), tableName, null, + Permission.Action.WRITE)); + protocol.grant(new UserPermission(Bytes.toBytes(gblUser.getShortName()), + Permission.Action.WRITE)); + } finally { + acl.close(); + } + Thread.sleep(100); - verifyDenied(user, getActionAll); - verifyDenied(user, getAction1); - verifyDenied(user, getAction2); - verifyAllowed(user, putActionAll); - verifyAllowed(user, putAction1); - verifyAllowed(user, putAction2); + verifyDenied(tblUser, getActionAll, getAction1, getAction2); + verifyAllowed(tblUser, putActionAll, putAction1, putAction2); + verifyAllowed(tblUser, deleteActionAll, deleteAction1, deleteAction2); - verifyAllowed(user, deleteActionAll); - verifyAllowed(user, deleteAction1); - verifyAllowed(user, deleteAction2); + verifyDenied(gblUser, getActionAll, getAction1, getAction2); + verifyAllowed(gblUser, putActionAll, putAction1, putAction2); + verifyAllowed(gblUser, deleteActionAll, deleteAction1, deleteAction2); // revoke table permission - protocol.grant(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, null, Permission.Action.READ, - Permission.Action.WRITE)); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(new UserPermission(Bytes.toBytes(tblUser.getShortName()), tableName, null, + Permission.Action.READ, Permission.Action.WRITE)); + protocol.revoke(new UserPermission(Bytes.toBytes(tblUser.getShortName()), tableName, null)); + protocol.revoke(new UserPermission(Bytes.toBytes(gblUser.getShortName()))); + } finally { + acl.close(); + } - protocol.revoke(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, null)); Thread.sleep(100); - verifyDenied(user, getActionAll); - verifyDenied(user, getAction1); - verifyDenied(user, getAction2); - verifyDenied(user, putActionAll); - verifyDenied(user, putAction1); - verifyDenied(user, putAction2); + verifyDenied(tblUser, getActionAll, getAction1, getAction2); + verifyDenied(tblUser, putActionAll, putAction1, putAction2); + verifyDenied(tblUser, deleteActionAll, deleteAction1, deleteAction2); - verifyDenied(user, deleteActionAll); - verifyDenied(user, deleteAction1); - verifyDenied(user, deleteAction2); + verifyDenied(gblUser, getActionAll, getAction1, getAction2); + verifyDenied(gblUser, putActionAll, putAction1, putAction2); + verifyDenied(gblUser, deleteActionAll, deleteAction1, deleteAction2); // grant column family read permission - protocol.grant(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, family1, Permission.Action.READ)); - Thread.sleep(100); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(new UserPermission(Bytes.toBytes(tblUser.getShortName()), tableName, family1, + Permission.Action.READ)); + protocol.grant(new UserPermission(Bytes.toBytes(gblUser.getShortName()), + Permission.Action.READ)); + } finally { + acl.close(); + } - verifyAllowed(user, getActionAll); - verifyAllowed(user, getAction1); - verifyDenied(user, getAction2); + Thread.sleep(100); - verifyDenied(user, putActionAll); - verifyDenied(user, putAction1); - verifyDenied(user, putAction2); + // Access should be denied for family2 + verifyAllowed(tblUser, getActionAll, getAction1); + verifyDenied(tblUser, getAction2); + verifyDenied(tblUser, putActionAll, putAction1, putAction2); + verifyDenied(tblUser, deleteActionAll, deleteAction1, deleteAction2); - verifyDenied(user, deleteActionAll); - verifyDenied(user, deleteAction1); - verifyDenied(user, deleteAction2); + verifyAllowed(gblUser, getActionAll, getAction1, getAction2); + verifyDenied(gblUser, putActionAll, putAction1, putAction2); + verifyDenied(gblUser, deleteActionAll, deleteAction1, deleteAction2); // grant column family write permission - protocol.grant(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, family2, Permission.Action.WRITE)); - Thread.sleep(100); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(new UserPermission(Bytes.toBytes(tblUser.getShortName()), tableName, family2, + Permission.Action.WRITE)); + protocol.grant(new UserPermission(Bytes.toBytes(gblUser.getShortName()), + Permission.Action.WRITE)); + } finally { + acl.close(); + } - verifyAllowed(user, getActionAll); - verifyAllowed(user, getAction1); - verifyDenied(user, getAction2); + Thread.sleep(100); - verifyDenied(user, putActionAll); - verifyDenied(user, putAction1); - verifyAllowed(user, putAction2); + // READ from family1, WRITE to family2 are allowed + verifyAllowed(tblUser, getActionAll, getAction1); + verifyAllowed(tblUser, putAction2, deleteAction2); + verifyDenied(tblUser, getAction2); + verifyDenied(tblUser, putActionAll, putAction1); + verifyDenied(tblUser, deleteActionAll, deleteAction1); - verifyDenied(user, deleteActionAll); - verifyDenied(user, deleteAction1); - verifyAllowed(user, deleteAction2); + verifyDenied(gblUser, getActionAll, getAction1, getAction2); + verifyAllowed(gblUser, putActionAll, putAction1, putAction2); + verifyAllowed(gblUser, deleteActionAll, deleteAction1, deleteAction2); // revoke column family permission - protocol.revoke(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, family2)); - Thread.sleep(100); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.revoke(new UserPermission(Bytes.toBytes(tblUser.getShortName()), tableName, family2)); + protocol.revoke(new UserPermission(Bytes.toBytes(gblUser.getShortName()))); + } finally { + acl.close(); + } - verifyAllowed(user, getActionAll); - verifyAllowed(user, getAction1); - verifyDenied(user, getAction2); + Thread.sleep(100); - verifyDenied(user, putActionAll); - verifyDenied(user, putAction1); - verifyDenied(user, putAction2); + // Revoke on family2 should not have impact on family1 permissions + verifyAllowed(tblUser, getActionAll, getAction1); + verifyDenied(tblUser, getAction2); + verifyDenied(tblUser, putActionAll, putAction1, putAction2); + verifyDenied(tblUser, deleteActionAll, deleteAction1, deleteAction2); - verifyDenied(user, deleteActionAll); - verifyDenied(user, deleteAction1); - verifyDenied(user, deleteAction2); + // Should not have access as global permissions are completely revoked + verifyDenied(gblUser, getActionAll, getAction1, getAction2); + verifyDenied(gblUser, putActionAll, putAction1, putAction2); + verifyDenied(gblUser, deleteActionAll, deleteAction1, deleteAction2); // delete table admin.disableTable(tableName); admin.deleteTable(tableName); } - private boolean hasFoundUserPermission(UserPermission userPermission, - List perms) { + private boolean hasFoundUserPermission(UserPermission userPermission, List perms) { return perms.contains(userPermission); } @Test - public void testGrantRevokeAtQualifierLevel() throws Exception { + public void testPostGrantRevokeAtQualifierLevel() throws Exception { final byte[] tableName = Bytes.toBytes("testGrantRevokeAtQualifierLevel"); final byte[] family1 = Bytes.toBytes("f1"); final byte[] family2 = Bytes.toBytes("f2"); @@ -842,7 +1229,6 @@ public void testGrantRevokeAtQualifierLevel() throws Exception { // create table HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); - if (admin.tableExists(tableName)) { admin.disableTable(tableName); admin.deleteTable(tableName); @@ -850,23 +1236,21 @@ public void testGrantRevokeAtQualifierLevel() throws Exception { HTableDescriptor htd = new HTableDescriptor(tableName); htd.addFamily(new HColumnDescriptor(family1)); htd.addFamily(new HColumnDescriptor(family2)); - htd.setOwnerString(USER_OWNER.getShortName()); admin.createTable(htd); // create temp users - User user = User.createUserForTesting(TEST_UTIL.getConfiguration(), - "user", new String[0]); - - HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); - AccessControllerProtocol protocol = - acl.coprocessorProxy(AccessControllerProtocol.class, tableName); + User user = User.createUserForTesting(TEST_UTIL.getConfiguration(), "user", new String[0]); PrivilegedExceptionAction getQualifierAction = new PrivilegedExceptionAction() { public Object run() throws Exception { Get g = new Get(Bytes.toBytes("random_row")); g.addColumn(family1, qualifier); HTable t = new HTable(conf, tableName); - t.get(g); + try { + t.get(g); + } finally { + t.close(); + } return null; } }; @@ -875,7 +1259,11 @@ public Object run() throws Exception { Put p = new Put(Bytes.toBytes("random_row")); p.add(family1, qualifier, Bytes.toBytes("v1")); HTable t = new HTable(conf, tableName); - t.put(p); + try { + t.put(p); + } finally { + t.close(); + } return null; } }; @@ -883,22 +1271,40 @@ public Object run() throws Exception { public Object run() throws Exception { Delete d = new Delete(Bytes.toBytes("random_row")); d.deleteColumn(family1, qualifier); - //d.deleteFamily(family1); + // d.deleteFamily(family1); HTable t = new HTable(conf, tableName); - t.delete(d); + try { + t.delete(d); + } finally { + t.close(); + } return null; } }; - protocol.revoke(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, family1)); + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.revoke(new UserPermission(Bytes.toBytes(user.getShortName()), tableName, family1)); + } finally { + acl.close(); + } + verifyDenied(user, getQualifierAction); verifyDenied(user, putQualifierAction); verifyDenied(user, deleteQualifierAction); - protocol.grant(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, family1, qualifier, - Permission.Action.READ)); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(new UserPermission(Bytes.toBytes(user.getShortName()), tableName, family1, + qualifier, Permission.Action.READ)); + } finally { + acl.close(); + } + Thread.sleep(100); verifyAllowed(user, getQualifierAction); @@ -907,9 +1313,16 @@ public Object run() throws Exception { // only grant write permission // TODO: comment this portion after HBASE-3583 - protocol.grant(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, family1, qualifier, - Permission.Action.WRITE)); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(new UserPermission(Bytes.toBytes(user.getShortName()), tableName, family1, + qualifier, Permission.Action.WRITE)); + } finally { + acl.close(); + } + Thread.sleep(100); verifyDenied(user, getQualifierAction); @@ -917,9 +1330,16 @@ public Object run() throws Exception { verifyAllowed(user, deleteQualifierAction); // grant both read and write permission. - protocol.grant(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, family1, qualifier, - Permission.Action.READ, Permission.Action.WRITE)); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(new UserPermission(Bytes.toBytes(user.getShortName()), tableName, family1, + qualifier, Permission.Action.READ, Permission.Action.WRITE)); + } finally { + acl.close(); + } + Thread.sleep(100); verifyAllowed(user, getQualifierAction); @@ -927,8 +1347,16 @@ public Object run() throws Exception { verifyAllowed(user, deleteQualifierAction); // revoke family level permission won't impact column level. - protocol.revoke(Bytes.toBytes(user.getShortName()), - new TablePermission(tableName, family1, qualifier)); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.revoke(new UserPermission(Bytes.toBytes(user.getShortName()), tableName, family1, + qualifier)); + } finally { + acl.close(); + } + Thread.sleep(100); verifyDenied(user, getQualifierAction); @@ -957,113 +1385,160 @@ public void testPermissionList() throws Exception { HTableDescriptor htd = new HTableDescriptor(tableName); htd.addFamily(new HColumnDescriptor(family1)); htd.addFamily(new HColumnDescriptor(family2)); - htd.setOwnerString(USER_OWNER.getShortName()); + htd.setOwner(USER_OWNER); admin.createTable(htd); + List perms; HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); - AccessControllerProtocol protocol = - acl.coprocessorProxy(AccessControllerProtocol.class, tableName); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + perms = protocol.getUserPermissions(tableName); + } finally { + acl.close(); + } - List perms = protocol.getUserPermissions(tableName); + UserPermission ownerperm = new UserPermission(Bytes.toBytes(USER_OWNER.getName()), tableName, + null, Action.values()); + assertTrue("Owner should have all permissions on table", + hasFoundUserPermission(ownerperm, perms)); - UserPermission up = new UserPermission(user, - tableName, family1, qualifier, Permission.Action.READ); + UserPermission up = new UserPermission(user, tableName, family1, qualifier, + Permission.Action.READ); assertFalse("User should not be granted permission: " + up.toString(), - hasFoundUserPermission(up, perms)); + hasFoundUserPermission(up, perms)); // grant read permission - UserPermission upToSet = new UserPermission(user, - tableName, family1, qualifier, Permission.Action.READ); - protocol.grant(user, upToSet); - perms = protocol.getUserPermissions(tableName); + UserPermission upToSet = new UserPermission(user, tableName, family1, qualifier, + Permission.Action.READ); + + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(upToSet); + perms = protocol.getUserPermissions(tableName); + } finally { + acl.close(); + } - UserPermission upToVerify = new UserPermission(user, - tableName, family1, qualifier, Permission.Action.READ); + UserPermission upToVerify = new UserPermission(user, tableName, family1, qualifier, + Permission.Action.READ); assertTrue("User should be granted permission: " + upToVerify.toString(), - hasFoundUserPermission(upToVerify, perms)); + hasFoundUserPermission(upToVerify, perms)); - upToVerify = new UserPermission(user, tableName, family1, qualifier, - Permission.Action.WRITE); + upToVerify = new UserPermission(user, tableName, family1, qualifier, Permission.Action.WRITE); assertFalse("User should not be granted permission: " + upToVerify.toString(), - hasFoundUserPermission(upToVerify, perms)); + hasFoundUserPermission(upToVerify, perms)); // grant read+write - upToSet = new UserPermission(user, tableName, family1, qualifier, - Permission.Action.WRITE, Permission.Action.READ); - protocol.grant(user, upToSet); - perms = protocol.getUserPermissions(tableName); + upToSet = new UserPermission(user, tableName, family1, qualifier, Permission.Action.WRITE, + Permission.Action.READ); + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.grant(upToSet); + perms = protocol.getUserPermissions(tableName); + } finally { + acl.close(); + } - upToVerify = new UserPermission(user, tableName, family1, qualifier, - Permission.Action.WRITE, Permission.Action.READ); + upToVerify = new UserPermission(user, tableName, family1, qualifier, Permission.Action.WRITE, + Permission.Action.READ); assertTrue("User should be granted permission: " + upToVerify.toString(), - hasFoundUserPermission(upToVerify, perms)); + hasFoundUserPermission(upToVerify, perms)); + + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + protocol.revoke(upToSet); + perms = protocol.getUserPermissions(tableName); + } finally { + acl.close(); + } - protocol.revoke(user, upToSet); - perms = protocol.getUserPermissions(tableName); assertFalse("User should not be granted permission: " + upToVerify.toString(), hasFoundUserPermission(upToVerify, perms)); - // delete table + // disable table before modification admin.disableTable(tableName); + + User newOwner = User.createUserForTesting(conf, "new_owner", new String[] {}); + htd.setOwner(newOwner); + admin.modifyTable(tableName, htd); + + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + tableName); + perms = protocol.getUserPermissions(tableName); + } finally { + acl.close(); + } + + UserPermission newOwnerperm = new UserPermission(Bytes.toBytes(newOwner.getName()), tableName, + null, Action.values()); + assertTrue("New owner should have all permissions on table", + hasFoundUserPermission(newOwnerperm, perms)); + + // delete table admin.deleteTable(tableName); } - /** global operations*/ + /** global operations */ private void verifyGlobal(PrivilegedExceptionAction action) throws Exception { - // should be allowed - verifyAllowed(SUPERUSER, action); - - // should be denied - verifyDenied(USER_OWNER, action); - verifyDenied(USER_RW, action); - verifyDenied(USER_NONE, action); - verifyDenied(USER_RO, action); + verifyAllowed(action, SUPERUSER); + + verifyDenied(action, USER_CREATE, USER_RW, USER_NONE, USER_RO); } public void checkGlobalPerms(Permission.Action... actions) throws IOException { - HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); - AccessControllerProtocol protocol = - acl.coprocessorProxy(AccessControllerProtocol.class, new byte[0]); - Permission[] perms = new Permission[actions.length]; - for (int i=0; i < actions.length; i++) { + for (int i = 0; i < actions.length; i++) { perms[i] = new Permission(actions[i]); } - - protocol.checkPermissions(perms); + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + new byte[0]); + protocol.checkPermissions(perms); + } finally { + acl.close(); + } } public void checkTablePerms(byte[] table, byte[] family, byte[] column, Permission.Action... actions) throws IOException { Permission[] perms = new Permission[actions.length]; - for (int i=0; i < actions.length; i++) { + for (int i = 0; i < actions.length; i++) { perms[i] = new TablePermission(table, family, column, actions[i]); } checkTablePerms(table, perms); } - public void checkTablePerms(byte[] table, Permission...perms) throws IOException { + public void checkTablePerms(byte[] table, Permission... perms) throws IOException { HTable acl = new HTable(conf, table); - AccessControllerProtocol protocol = - acl.coprocessorProxy(AccessControllerProtocol.class, new byte[0]); - - protocol.checkPermissions(perms); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + new byte[0]); + protocol.checkPermissions(perms); + } finally { + acl.close(); + } } - public void grant(AccessControllerProtocol protocol, User user, byte[] t, byte[] f, - byte[] q, Permission.Action... actions) throws IOException { - protocol.grant(Bytes.toBytes(user.getShortName()), new TablePermission(t, f, q, actions)); + public void grant(AccessControllerProtocol protocol, User user, byte[] t, byte[] f, byte[] q, + Permission.Action... actions) throws IOException { + protocol.grant(new UserPermission(Bytes.toBytes(user.getShortName()), t, f, q, actions)); } @Test public void testCheckPermissions() throws Exception { - final HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); - final AccessControllerProtocol protocol = - acl.coprocessorProxy(AccessControllerProtocol.class, TEST_TABLE); - - //-------------------------------------- - //test global permissions + // -------------------------------------- + // test global permissions PrivilegedExceptionAction globalAdmin = new PrivilegedExceptionAction() { @Override public Void run() throws Exception { @@ -1071,11 +1546,11 @@ public Void run() throws Exception { return null; } }; - //verify that only superuser can admin + // verify that only superuser can admin verifyGlobal(globalAdmin); - //-------------------------------------- - //test multiple permissions + // -------------------------------------- + // test multiple permissions PrivilegedExceptionAction globalReadWrite = new PrivilegedExceptionAction() { @Override public Void run() throws Exception { @@ -1086,8 +1561,8 @@ public Void run() throws Exception { verifyGlobal(globalReadWrite); - //-------------------------------------- - //table/column/qualifier level permissions + // -------------------------------------- + // table/column/qualifier level permissions final byte[] TEST_Q1 = Bytes.toBytes("q1"); final byte[] TEST_Q2 = Bytes.toBytes("q2"); @@ -1095,9 +1570,16 @@ public Void run() throws Exception { User userColumn = User.createUserForTesting(conf, "user_check_perms_family", new String[0]); User userQualifier = User.createUserForTesting(conf, "user_check_perms_q", new String[0]); - grant(protocol, userTable, TEST_TABLE, null, null, Permission.Action.READ); - grant(protocol, userColumn, TEST_TABLE, TEST_FAMILY, null, Permission.Action.READ); - grant(protocol, userQualifier, TEST_TABLE, TEST_FAMILY, TEST_Q1, Permission.Action.READ); + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + TEST_TABLE); + grant(protocol, userTable, TEST_TABLE, null, null, Permission.Action.READ); + grant(protocol, userColumn, TEST_TABLE, TEST_FAMILY, null, Permission.Action.READ); + grant(protocol, userQualifier, TEST_TABLE, TEST_FAMILY, TEST_Q1, Permission.Action.READ); + } finally { + acl.close(); + } PrivilegedExceptionAction tableRead = new PrivilegedExceptionAction() { @Override @@ -1127,9 +1609,8 @@ public Void run() throws Exception { @Override public Void run() throws Exception { checkTablePerms(TEST_TABLE, new Permission[] { - new TablePermission(TEST_TABLE, TEST_FAMILY, TEST_Q1, Permission.Action.READ), - new TablePermission(TEST_TABLE, TEST_FAMILY, TEST_Q2, Permission.Action.READ), - }); + new TablePermission(TEST_TABLE, TEST_FAMILY, TEST_Q1, Permission.Action.READ), + new TablePermission(TEST_TABLE, TEST_FAMILY, TEST_Q2, Permission.Action.READ), }); return null; } }; @@ -1137,10 +1618,8 @@ public Void run() throws Exception { PrivilegedExceptionAction globalAndTableRead = new PrivilegedExceptionAction() { @Override public Void run() throws Exception { - checkTablePerms(TEST_TABLE, new Permission[] { - new Permission(Permission.Action.READ), - new TablePermission(TEST_TABLE, null, (byte[])null, Permission.Action.READ), - }); + checkTablePerms(TEST_TABLE, new Permission[] { new Permission(Permission.Action.READ), + new TablePermission(TEST_TABLE, null, (byte[]) null, Permission.Action.READ), }); return null; } }; @@ -1169,31 +1648,465 @@ public Void run() throws Exception { verifyAllowed(noCheck, SUPERUSER, userTable, userColumn, userQualifier); - //-------------------------------------- - //test family level multiple permissions + // -------------------------------------- + // test family level multiple permissions PrivilegedExceptionAction familyReadWrite = new PrivilegedExceptionAction() { @Override public Void run() throws Exception { checkTablePerms(TEST_TABLE, TEST_FAMILY, null, Permission.Action.READ, - Permission.Action.WRITE); + Permission.Action.WRITE); return null; } }; - // should be allowed + verifyAllowed(familyReadWrite, SUPERUSER, USER_OWNER, USER_RW); - // should be denied - verifyDenied(familyReadWrite, USER_NONE, USER_RO); + verifyDenied(familyReadWrite, USER_NONE, USER_CREATE, USER_RO); - //-------------------------------------- - //check for wrong table region + // -------------------------------------- + // check for wrong table region + acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); try { - //but ask for TablePermissions for TEST_TABLE - protocol.checkPermissions(new Permission[] {(Permission) new TablePermission( - TEST_TABLE, null, (byte[])null, Permission.Action.CREATE)}); - fail("this should have thrown CoprocessorException"); - } catch(CoprocessorException ex) { - //expected + AccessControllerProtocol protocol = acl.coprocessorProxy(AccessControllerProtocol.class, + TEST_TABLE); + try { + // but ask for TablePermissions for TEST_TABLE + protocol.checkPermissions(new Permission[] { (Permission) new TablePermission(TEST_TABLE, + null, (byte[]) null, Permission.Action.CREATE) }); + fail("this should have thrown CoprocessorException"); + } catch (CoprocessorException ex) { + // expected + } + } finally { + acl.close(); } + } + + @Test + public void testLockAction() throws Exception { + PrivilegedExceptionAction lockAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preLockRow(ObserverContext.createAndPrepare(RCP_ENV, null), null, + Bytes.toBytes("random_row")); + return null; + } + }; + verifyAllowed(lockAction, SUPERUSER, USER_ADMIN, USER_OWNER, USER_CREATE, USER_RW_ON_TABLE); + verifyDenied(lockAction, USER_RO, USER_RW, USER_NONE); + } + + @Test + public void testUnLockAction() throws Exception { + PrivilegedExceptionAction unLockAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preUnlockRow(ObserverContext.createAndPrepare(RCP_ENV, null), null, + 123456); + return null; + } + }; + verifyAllowed(unLockAction, SUPERUSER, USER_ADMIN, USER_OWNER, USER_RW_ON_TABLE); + verifyDenied(unLockAction, USER_NONE, USER_RO, USER_RW); + } + + @Test + public void testStopRegionServer() throws Exception { + PrivilegedExceptionAction action = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preStopRegionServer(ObserverContext.createAndPrepare(RSCP_ENV, null)); + return null; + } + }; + verifyAllowed(action, SUPERUSER, USER_ADMIN); + verifyDenied(action, USER_CREATE, USER_OWNER, USER_RW, USER_RO, USER_NONE); } + + @Test + public void testOpenRegion() throws Exception { + PrivilegedExceptionAction action = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preOpen(ObserverContext.createAndPrepare(RCP_ENV, null)); + return null; + } + }; + + verifyAllowed(action, SUPERUSER, USER_ADMIN); + verifyDenied(action, USER_CREATE, USER_RW, USER_RO, USER_NONE, USER_OWNER); + } + + @Test + public void testCloseRegion() throws Exception { + PrivilegedExceptionAction action = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preClose(ObserverContext.createAndPrepare(RCP_ENV, null), false); + return null; + } + }; + + verifyAllowed(action, SUPERUSER, USER_ADMIN); + verifyDenied(action, USER_CREATE, USER_RW, USER_RO, USER_NONE, USER_OWNER); + } + + + @Test + public void testSnapshot() throws Exception { + PrivilegedExceptionAction snapshotAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preSnapshot(ObserverContext.createAndPrepare(CP_ENV, null), + null, null); + return null; + } + }; + + PrivilegedExceptionAction deleteAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preDeleteSnapshot(ObserverContext.createAndPrepare(CP_ENV, null), + null); + return null; + } + }; + + PrivilegedExceptionAction restoreAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preRestoreSnapshot(ObserverContext.createAndPrepare(CP_ENV, null), + null, null); + return null; + } + }; + + PrivilegedExceptionAction cloneAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + ACCESS_CONTROLLER.preCloneSnapshot(ObserverContext.createAndPrepare(CP_ENV, null), + null, null); + return null; + } + }; + + verifyAllowed(snapshotAction, SUPERUSER, USER_ADMIN); + verifyDenied(snapshotAction, USER_CREATE, USER_RW, USER_RO, USER_NONE, USER_OWNER); + + verifyAllowed(cloneAction, SUPERUSER, USER_ADMIN); + verifyDenied(deleteAction, USER_CREATE, USER_RW, USER_RO, USER_NONE, USER_OWNER); + + verifyAllowed(restoreAction, SUPERUSER, USER_ADMIN); + verifyDenied(restoreAction, USER_CREATE, USER_RW, USER_RO, USER_NONE, USER_OWNER); + + verifyAllowed(deleteAction, SUPERUSER, USER_ADMIN); + verifyDenied(cloneAction, USER_CREATE, USER_RW, USER_RO, USER_NONE, USER_OWNER); + } + + @Test + public void testGlobalAuthorizationForNewRegisteredRS() throws Exception { + LOG.debug("Test for global authorization for a new registered RegionServer."); + MiniHBaseCluster hbaseCluster = TEST_UTIL.getHBaseCluster(); + final HRegionServer oldRs = hbaseCluster.getRegionServer(0); + + // Since each RegionServer running on different user, add global + // permissions for the new user. + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy( + AccessControllerProtocol.class, TEST_TABLE); + String currentUser = User.getCurrent().getShortName(); + // User name for the new RegionServer we plan to add. + String activeUserForNewRs = currentUser + ".hfs." + + hbaseCluster.getLiveRegionServerThreads().size(); + + protocol.grant(new UserPermission(Bytes.toBytes(activeUserForNewRs), + Permission.Action.ADMIN, Permission.Action.CREATE, + Permission.Action.READ, Permission.Action.WRITE)); + + } finally { + acl.close(); + } + final HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + HTableDescriptor htd = new HTableDescriptor(TEST_TABLE2); + htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); + htd.setOwner(USER_OWNER); + admin.createTable(htd); + + // Starting a new RegionServer. + JVMClusterUtil.RegionServerThread newRsThread = hbaseCluster + .startRegionServer(); + final HRegionServer newRs = newRsThread.getRegionServer(); + + // Move region to the new RegionServer. + final HTable table = new HTable(TEST_UTIL.getConfiguration(), TEST_TABLE2); + try { + NavigableMap regions = table + .getRegionLocations(); + final Map.Entry firstRegion = regions.entrySet() + .iterator().next(); + + PrivilegedExceptionAction moveAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + admin.move(firstRegion.getKey().getEncodedNameAsBytes(), + Bytes.toBytes(newRs.getServerName().getServerName())); + return null; + } + }; + SUPERUSER.runAs(moveAction); + + final int RETRIES_LIMIT = 10; + int retries = 0; + while (newRs.getOnlineRegions(TEST_TABLE2).size() < 1 && retries < RETRIES_LIMIT) { + LOG.debug("Waiting for region to be opened. Already retried " + retries + + " times."); + try { + Thread.sleep(1000); + } catch (InterruptedException e) { + } + retries++; + if (retries == RETRIES_LIMIT - 1) { + fail("Retry exhaust for waiting region to be opened."); + } + } + // Verify write permission for user "admin2" who has the global + // permissions. + PrivilegedExceptionAction putAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + Put put = new Put(Bytes.toBytes("test")); + put.add(TEST_FAMILY, Bytes.toBytes("qual"), Bytes.toBytes("value")); + table.put(put); + return null; + } + }; + USER_ADMIN.runAs(putAction); + } finally { + table.close(); + } + } + + @Test + public void testTableDescriptorsEnumeration() throws Exception { + User TABLE_ADMIN = User.createUserForTesting(conf, "UserA", new String[0]); + + // Grant TABLE ADMIN privs on test table to UserA + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy( + AccessControllerProtocol.class, TEST_TABLE); + protocol.grant(new UserPermission(Bytes.toBytes(TABLE_ADMIN.getShortName()), + TEST_TABLE, null, Permission.Action.ADMIN)); + } finally { + acl.close(); + } + + PrivilegedExceptionAction listTablesAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + try { + admin.listTables(); + } finally { + admin.close(); + } + return null; + } + }; + + PrivilegedExceptionAction getTableDescAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + try { + admin.getTableDescriptor(TEST_TABLE); + } finally { + admin.close(); + } + return null; + } + }; + + verifyAllowed(listTablesAction, SUPERUSER, USER_ADMIN); + verifyDenied(listTablesAction, USER_CREATE, USER_RW, USER_RO, USER_NONE, TABLE_ADMIN); + + verifyAllowed(getTableDescAction, SUPERUSER, USER_ADMIN, USER_CREATE, TABLE_ADMIN); + verifyDenied(getTableDescAction, USER_RW, USER_RO, USER_NONE); + } + + @Test + public void testTableDeletion() throws Exception { + final User tableAdmin = User.createUserForTesting(conf, "TestUser", new String[0]); + + // We need to create a new table here because we will be testing what + // happens when it is deleted + final byte[] tableName = Bytes.toBytes("testTableDeletion"); + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + HTableDescriptor htd = new HTableDescriptor(tableName); + htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); + admin.createTable(htd); + TEST_UTIL.waitTableEnabled(tableName, 5000); + + // Grant TABLE ADMIN privs + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy( + AccessControllerProtocol.class, tableName); + protocol.grant(new UserPermission(Bytes.toBytes(tableAdmin.getShortName()), + tableName, null, Permission.Action.ADMIN)); + } finally { + acl.close(); + } + + PrivilegedExceptionAction deleteTableAction = new PrivilegedExceptionAction() { + public Object run() throws Exception { + HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + try { + admin.disableTable(tableName); + admin.deleteTable(tableName); + } finally { + admin.close(); + } + return null; + } + }; + + verifyDenied(deleteTableAction, USER_RW, USER_RO, USER_NONE); + verifyAllowed(deleteTableAction, tableAdmin); + } + + @Test + public void testCreateWithCorrectOwner() throws Exception { + final byte[] tableName = Bytes.toBytes("testCreateWithCorrectOwner"); + + // Create a test user + User testUser = User.createUserForTesting(TEST_UTIL.getConfiguration(), "TestUser", + new String[0]); + + // Grant the test user the ability to create tables + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy( + AccessControllerProtocol.class, AccessControlLists.ACL_TABLE_NAME); + protocol.grant(new UserPermission(Bytes.toBytes(testUser.getShortName()), + Permission.Action.CREATE)); + } finally { + acl.close(); + } + + verifyAllowed(new PrivilegedExceptionAction() { + @Override + public Object run() throws Exception { + HTableDescriptor desc = new HTableDescriptor(tableName); + desc.addFamily(new HColumnDescriptor(TEST_FAMILY)); + HBaseAdmin admin = new HBaseAdmin(conf); + try { + admin.createTable(desc); + } finally { + admin.close(); + } + return null; + } + }, testUser); + TEST_UTIL.waitTableEnabled(tableName, 5000); + + // Verify that owner permissions have been granted to the test user on the + // table just created + List perms = AccessControlLists.getTablePermissions(conf, tableName) + .get(testUser.getShortName()); + assertNotNull(perms); + assertFalse(perms.isEmpty()); + // Should be RWXCA + assertTrue(perms.get(0).implies(Permission.Action.READ)); + assertTrue(perms.get(0).implies(Permission.Action.WRITE)); + assertTrue(perms.get(0).implies(Permission.Action.EXEC)); + assertTrue(perms.get(0).implies(Permission.Action.CREATE)); + assertTrue(perms.get(0).implies(Permission.Action.ADMIN)); + } + + @Test + public void testACLTableAccess() throws Exception { + final Configuration conf = TEST_UTIL.getConfiguration(); + + final byte[] tableName = Bytes.toBytes("testACLTableAccess"); + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + HTableDescriptor htd = new HTableDescriptor(tableName); + htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); + admin.createTable(htd); + TEST_UTIL.waitTableEnabled(tableName, 5000); + + // Global users + User globalRead = User.createUserForTesting(conf, "globalRead", new String[0]); + User globalWrite = User.createUserForTesting(conf, "globalWrite", new String[0]); + User globalCreate = User.createUserForTesting(conf, "globalCreate", new String[0]); + User globalAdmin = User.createUserForTesting(conf, "globalAdmin", new String[0]); + + // Table users + User tableRead = User.createUserForTesting(conf, "tableRead", new String[0]); + User tableWrite = User.createUserForTesting(conf, "tableWrite", new String[0]); + User tableCreate = User.createUserForTesting(conf, "tableCreate", new String[0]); + User tableAdmin = User.createUserForTesting(conf, "tableAdmin", new String[0]); + + // Set up grants + HTable acl = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + AccessControllerProtocol protocol = acl.coprocessorProxy( + AccessControllerProtocol.class, AccessControlLists.ACL_TABLE_NAME); + protocol.grant(new UserPermission(Bytes.toBytes(globalRead.getShortName()), Action.READ)); + protocol.grant(new UserPermission(Bytes.toBytes(globalWrite.getShortName()), Action.WRITE)); + protocol.grant(new UserPermission(Bytes.toBytes(globalCreate.getShortName()), + Action.CREATE)); + protocol.grant(new UserPermission(Bytes.toBytes(globalAdmin.getShortName()), + Action.ADMIN)); + protocol.grant(new UserPermission(Bytes.toBytes(tableRead.getShortName()), tableName, + null, Action.READ)); + protocol.grant(new UserPermission(Bytes.toBytes(tableWrite.getShortName()), tableName, + null, Action.WRITE)); + protocol.grant(new UserPermission(Bytes.toBytes(tableCreate.getShortName()), tableName, + null, Action.CREATE)); + protocol.grant(new UserPermission(Bytes.toBytes(tableAdmin.getShortName()), tableName, + null, Action.ADMIN)); + } finally { + acl.close(); + } + + // Write tests + + PrivilegedExceptionAction writeAction = new PrivilegedExceptionAction() { + @Override + public Object run() throws Exception { + HTable t = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + t.put(new Put(Bytes.toBytes("test")).add(AccessControlLists.ACL_LIST_FAMILY, + Bytes.toBytes("q"), Bytes.toBytes("value"))); + return null; + } finally { + t.close(); + } + } + }; + + // All writes to ACL table denied except for GLOBAL WRITE permission and superuser + + verifyDenied(writeAction, globalAdmin, globalCreate, globalRead); + verifyDenied(writeAction, tableAdmin, tableCreate, tableRead, tableWrite); + verifyAllowed(writeAction, SUPERUSER, globalWrite); + + // Read tests + + PrivilegedExceptionAction scanAction = new PrivilegedExceptionAction() { + @Override + public Object run() throws Exception { + HTable t = new HTable(conf, AccessControlLists.ACL_TABLE_NAME); + try { + ResultScanner s = t.getScanner(new Scan()); + try { + for (Result r = s.next(); r != null; r = s.next()) { + // do nothing + } + } finally { + s.close(); + } + return null; + } finally { + t.close(); + } + } + }; + + // All reads from ACL table denied except for GLOBAL READ and superuser + + verifyDenied(scanAction, globalAdmin, globalCreate, globalWrite); + verifyDenied(scanAction, tableCreate, tableAdmin, tableRead, tableWrite); + verifyAllowed(scanAction, SUPERUSER, globalRead); + } + } diff --git a/security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java b/security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java index 39fc73e78985..a3a0f8a9610e 100644 --- a/security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java +++ b/security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java @@ -42,8 +42,10 @@ import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.security.User; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.junit.After; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -87,6 +89,10 @@ public static void beforeClass() throws Exception { SecureTestUtil.enableSecurity(conf); UTIL.startMiniCluster(); + + // Wait for the ACL table to become available + UTIL.waitTableAvailable(AccessControlLists.ACL_TABLE_NAME, 30000); + ZKW = new ZooKeeperWatcher(UTIL.getConfiguration(), "TestTablePermissions", ABORTABLE); @@ -99,19 +105,28 @@ public static void afterClass() throws Exception { UTIL.shutdownMiniCluster(); } + @After + public void tearDown() throws Exception { + Configuration conf = UTIL.getConfiguration(); + AccessControlLists.removeTablePermissions(conf, TEST_TABLE); + AccessControlLists.removeTablePermissions(conf, TEST_TABLE2); + AccessControlLists.removeTablePermissions(conf, AccessControlLists.ACL_TABLE_NAME); + } + @Test public void testBasicWrite() throws Exception { Configuration conf = UTIL.getConfiguration(); // add some permissions - AccessControlLists.addTablePermission(conf, TEST_TABLE, - "george", new TablePermission(TEST_TABLE, null, - TablePermission.Action.READ, TablePermission.Action.WRITE)); - AccessControlLists.addTablePermission(conf, TEST_TABLE, - "hubert", new TablePermission(TEST_TABLE, null, - TablePermission.Action.READ)); - AccessControlLists.addTablePermission(conf, TEST_TABLE, - "humphrey", new TablePermission(TEST_TABLE, TEST_FAMILY, TEST_QUALIFIER, - TablePermission.Action.READ)); + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("george"), TEST_TABLE, null, (byte[])null, + UserPermission.Action.READ, UserPermission.Action.WRITE)); + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("hubert"), TEST_TABLE, null, (byte[])null, + UserPermission.Action.READ)); + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("humphrey"), + TEST_TABLE, TEST_FAMILY, TEST_QUALIFIER, + UserPermission.Action.READ)); // retrieve the same ListMultimap perms = @@ -165,8 +180,8 @@ public void testBasicWrite() throws Exception { assertFalse(actions.contains(TablePermission.Action.WRITE)); // table 2 permissions - AccessControlLists.addTablePermission(conf, TEST_TABLE2, "hubert", - new TablePermission(TEST_TABLE2, null, + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("hubert"), TEST_TABLE2, null, (byte[])null, TablePermission.Action.READ, TablePermission.Action.WRITE)); // check full load @@ -197,16 +212,21 @@ public void testBasicWrite() throws Exception { @Test public void testPersistence() throws Exception { Configuration conf = UTIL.getConfiguration(); - AccessControlLists.addTablePermission(conf, TEST_TABLE, "albert", - new TablePermission(TEST_TABLE, null, TablePermission.Action.READ)); - AccessControlLists.addTablePermission(conf, TEST_TABLE, "betty", - new TablePermission(TEST_TABLE, null, TablePermission.Action.READ, - TablePermission.Action.WRITE)); - AccessControlLists.addTablePermission(conf, TEST_TABLE, "clark", - new TablePermission(TEST_TABLE, TEST_FAMILY, TablePermission.Action.READ)); - AccessControlLists.addTablePermission(conf, TEST_TABLE, "dwight", - new TablePermission(TEST_TABLE, TEST_FAMILY, TEST_QUALIFIER, - TablePermission.Action.WRITE)); + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("albert"), TEST_TABLE, null, + (byte[])null, TablePermission.Action.READ)); + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("betty"), TEST_TABLE, null, + (byte[])null, TablePermission.Action.READ, + TablePermission.Action.WRITE)); + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("clark"), + TEST_TABLE, TEST_FAMILY, + TablePermission.Action.READ)); + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("dwight"), + TEST_TABLE, TEST_FAMILY, TEST_QUALIFIER, + TablePermission.Action.WRITE)); // verify permissions survive changes in table metadata ListMultimap preperms = @@ -313,4 +333,60 @@ public void testEquals() throws Exception { assertFalse(p1.equals(p2)); assertFalse(p2.equals(p1)); } + + @Test + public void testGlobalPermission() throws Exception { + Configuration conf = UTIL.getConfiguration(); + + // add some permissions + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("user1"), + Permission.Action.READ, Permission.Action.WRITE)); + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("user2"), + Permission.Action.CREATE)); + AccessControlLists.addUserPermission(conf, + new UserPermission(Bytes.toBytes("user3"), + Permission.Action.ADMIN, Permission.Action.READ, Permission.Action.CREATE)); + + ListMultimap perms = AccessControlLists.getTablePermissions(conf, null); + List user1Perms = perms.get("user1"); + assertEquals("Should have 1 permission for user1", 1, user1Perms.size()); + assertEquals("user1 should have WRITE permission", + new Permission.Action[] { Permission.Action.READ, Permission.Action.WRITE }, + user1Perms.get(0).getActions()); + + List user2Perms = perms.get("user2"); + assertEquals("Should have 1 permission for user2", 1, user2Perms.size()); + assertEquals("user2 should have CREATE permission", + new Permission.Action[] { Permission.Action.CREATE }, + user2Perms.get(0).getActions()); + + List user3Perms = perms.get("user3"); + assertEquals("Should have 1 permission for user3", 1, user3Perms.size()); + assertEquals("user3 should have ADMIN, READ, CREATE permission", + new Permission.Action[] { + Permission.Action.ADMIN, Permission.Action.READ, Permission.Action.CREATE + }, + user3Perms.get(0).getActions()); + } + + @Test + public void testAuthManager() throws Exception { + Configuration conf = UTIL.getConfiguration(); + /* test a race condition causing TableAuthManager to sometimes fail global permissions checks + * when the global cache is being updated + */ + TableAuthManager authManager = TableAuthManager.get(ZKW, conf); + // currently running user is the system user and should have global admin perms + User currentUser = User.getCurrent(); + assertTrue(authManager.authorize(currentUser, Permission.Action.ADMIN)); + for (int i=1; i<=50; i++) { + AccessControlLists.addUserPermission(conf, new UserPermission(Bytes.toBytes("testauth"+i), + Permission.Action.ADMIN, Permission.Action.READ, Permission.Action.WRITE)); + // make sure the system user still shows as authorized + assertTrue("Failed current user auth check on iter "+i, + authManager.authorize(currentUser, Permission.Action.ADMIN)); + } + } } diff --git a/security/src/test/resources/hbase-site.xml b/security/src/test/resources/hbase-site.xml index dcc7df2fe57b..4f1dd5f15d0c 100644 --- a/security/src/test/resources/hbase-site.xml +++ b/security/src/test/resources/hbase-site.xml @@ -2,8 +2,6 @@ + + ${basedir}/NOTICE.txt + ${basedir}/LICENSE.txt + @@ -108,4 +113,26 @@ 0644 + + + + ${project.build.directory}/maven-shared-archive-resources-for-assembly/META-INF/LICENSE + . + LICENSE.txt + unix + + + ${project.build.directory}/NOTICE.aggregate + . + NOTICE.txt + unix + + + ${basedir}/src/main/resources/META-INF/LEGAL + . + LEGAL + unix + + + diff --git a/src/assembly/resources/supplemental-models.xml b/src/assembly/resources/supplemental-models.xml new file mode 100644 index 000000000000..6b727f9c836c --- /dev/null +++ b/src/assembly/resources/supplemental-models.xml @@ -0,0 +1,1664 @@ + + + + + + + + + org.apache.zookeeper + zookeeper + + + The Apache Software Foundation + http://www.apache.org/ + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + commons-beanutils + commons-beanutils + + + The Apache Software Foundation + http://www.apache.org/ + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.apache.hadoop + hadoop-core + + + The Apache Software Foundation + http://www.apache.org/ + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.xerial.snappy + snappy-java + + + The Apache Software Foundation + http://www.apache.org/ + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + + com.github.stephenc.findbugs + findbugs-annotations + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + com.github.stephenc.high-scale-lib + high-scale-lib + Highly Scalable Java + + + + Public Domain + repo + http://creativecommons.org/licenses/publicdomain/ + +The person or persons who have associated work with this document (the +"Dedicator" or "Certifier") hereby either (a) certifies that, to the best +of his knowledge, the work of authorship identified is in the public +domain of the country from which the work is published, or (b) hereby +dedicates whatever copyright the dedicators holds in the work of +authorship identified below (the "Work") to the public domain. A +certifier, moreover, dedicates any copyright interest he may have in the +associated work, and for these purposes, is described as a "dedicator" +below. + +A certifier has taken reasonable steps to verify the copyright status of +this work. Certifier recognizes that his good faith efforts may not +shield him from liability if in fact the work certified is not in the +public domain. + +Dedicator makes this dedication for the benefit of the public at large and +to the detriment of the Dedicator's heirs and successors. Dedicator +intends this dedication to be an overt act of relinquishment in perpetuity +of all present and future rights under copyright law, whether vested or +contingent, in the Work. Dedicator understands that such relinquishment of +all rights includes the relinquishment of all rights to enforce (by +lawsuit or otherwise) those copyrights in the Work. + +Dedicator recognizes that, once placed in the public domain, the Work may +be freely reproduced, distributed, transmitted, used, modified, built +upon, or otherwise exploited by anyone for any purpose, commercial or +non-commercial, and in any way, including by methods that have not yet +been invented or conceived. + + + + + + + + org.apache.httpcomponents + httpclient + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.apache.httpcomponents + httpcore + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.jboss.netty + netty + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + io.netty + netty + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + commons-httpclient + commons-httpclient + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.mortbay.jetty + jetty-util + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.cloudera.htrace + htrace-core + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + + net.java.dev.jets3t + jets3t + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.mortbay.jetty + jetty + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.mortbay.jetty + jetty-sslengine + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.mortbay.jetty + jsp-api-2.1 + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + com.yammer.metrics + metrics-core + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + + org.codehaus.jettison + jettison + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + +Copyright 2006 Envoi Solutions LLC + + + + + + + + + com.google.protobuf + protobuf-java + Protocol Buffer Java API + + + + + New BSD license + http://www.opensource.org/licenses/bsd-license.php + repo + +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Code generated by the Protocol Buffer compiler is owned by the owner +of the input file used when generating it. This code is not +standalone and requires a support library to be linked with it. This +support library is itself covered by the above license. + + + + + + + + com.jcraft + jsch + JSch + + + + + BSD license + http://www.jcraft.com/jsch/LICENSE.txt + +Copyright (c) 2002-2015 Atsuhiko Yamanaka, JCraft,Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the distribution. + + 3. The names of the authors may not be used to endorse or promote products + derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, +INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL JCRAFT, +INC. OR ANY CONTRIBUTORS TO THIS SOFTWARE BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, +EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + + + + + + com.thoughtworks.paranamer + paranamer + ParaNamer Core + + + + BSD 3-Clause License + https://github.com/codehaus/paranamer-git/blob/paranamer-2.3/LICENSE.txt + repo + + Copyright (c) 2006 Paul Hammant & ThoughtWorks Inc + + + + + + + + org.jruby.jcodings + jcodings + JCodings + + + + MIT License + http://www.opensource.org/licenses/mit-license.php + repo + +Copyright (c) 2008-2012 The JCodings Authors + + + + + + + + org.jruby.joni + joni + Joni + + + + MIT License + http://www.opensource.org/licenses/mit-license.php + repo + +Copyright (c) 2008-2014 The Joni Authors + + + + + + + + org.slf4j + slf4j-api + SLF4J API Module + + + + MIT License + http://www.opensource.org/licenses/mit-license.php + repo + +Copyright (c) 2004-2013 QOS.ch + + + + + + + + org.slf4j + slf4j-log4j12 + SLF4J LOG4J-12 Binding + + + + MIT License + http://www.opensource.org/licenses/mit-license.php + repo + +Copyright (c) 2004-2008 QOS.ch + + + + + + + + xmlenc + xmlenc + xmlenc Library + + + + BSD 3-Clause License + http://www.opensource.org/licenses/bsd-license.php + repo + +Copyright 2003-2005, Ernst de Haan <wfe.dehaan@gmail.com> + + + + + + + + org.tukaani + xz + + + + Public Domain + repo + +Licensing of XZ for Java +======================== + + All the files in this package have been written by Lasse Collin + and/or Igor Pavlov. All these files have been put into the + public domain. You can do whatever you want with these files. + + This software is provided "as is", without any warranty. + + + + + + + + + aopalliance + aopalliance + AOP alliance + + + + Public Domain + repo + +LICENCE: all the source code provided by AOP Alliance is Public Domain. + + + + + + + + asm + asm + ASM: a very small and fast Java bytecode manipulation framework + + + + BSD 3-Clause License + http://cvs.forge.objectweb.org/cgi-bin/viewcvs.cgi/*checkout*/asm/asm/LICENSE.txt?rev=1.3&only_with_tag=ASM_3_1_MVN + repo + +Copyright (c) 2000-2005 INRIA, France Telecom + + + + + + + + org.fusesource.leveldbjni + leveldbjni-all + + + + + BSD 3-Clause License + http://www.opensource.org/licenses/BSD-3-Clause + repo + +Copyright (c) 2011 FuseSource Corp. All rights reserved. + + + + + + + + + org.hamcrest + hamcrest-core + + + + New BSD license + http://www.opensource.org/licenses/bsd-license.php + repo + +Copyright (c) 2000-2006, www.hamcrest.org +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +Redistributions of source code must retain the above copyright notice, this list of +conditions and the following disclaimer. Redistributions in binary form must reproduce +the above copyright notice, this list of conditions and the following disclaimer in +the documentation and/or other materials provided with the distribution. + +Neither the name of Hamcrest nor the names of its contributors may be used to endorse +or promote products derived from this software without specific prior written +permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY +WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + + + + + + + + + + javax.activation + activation + JavaBeans Activation Framework (JAF) + http://java.sun.com/products/javabeans/jaf/index.jsp + + + + Common Development and Distribution License (CDDL) v1.0 + https://glassfish.dev.java.net/public/CDDLv1.0.html + repo + + + + + + + + + javax.xml.bind + jaxb-api + JAXB API bundle for GlassFish V3 + + + + CDDL 1.1 + https://glassfish.java.net/public/CDDL+GPL_1_1.html + repo + +Copyright (c) 2010 Oracle and/or its affiliates. + + + + + + + + junit + junit + JUnit + http://junit.org/ + + + + Common Public License Version 1.0 + http://www.opensource.org/licenses/cpl1.0.txt + repo + + + + + + + + + + com.sun.jersey + jersey-client + + https://java.net/projects/jersey/ + + + + CDDL 1.1 + https://glassfish.java.net/public/CDDL+GPL_1_1.html + repo + +Copyright (c) 2010-2011 Oracle and/or its affiliates. + + + + + + + + com.sun.jersey + jersey-core + https://java.net/projects/jersey/ + + + + CDDL 1.1 + https://glassfish.java.net/public/CDDL+GPL_1_1.html + repo + +Copyright (c) 2010-2011 Oracle and/or its affiliates. + + + + + + + + com.sun.jersey + jersey-json + https://java.net/projects/jersey/ + + + + CDDL 1.1 + https://glassfish.java.net/public/CDDL+GPL_1_1.html + repo + +Copyright (c) 2010-2011 Oracle and/or its affiliates. + + + + + + + + com.sun.jersey + jersey-server + https://java.net/projects/jersey/ + + + + CDDL 1.1 + https://glassfish.java.net/public/CDDL+GPL_1_1.html + repo + +Copyright (c) 2010-2011 Oracle and/or its affiliates. + + + + + + + + com.sun.jersey.contribs + jersey-guice + https://java.net/projects/jersey/ + + + + CDDL 1.1 + https://glassfish.java.net/public/CDDL+GPL_1_1.html + repo + +Copyright (c) 2010-2011 Oracle and/or its affiliates. + + + + + + + + com.sun.xml.bind + jaxb-impl + JAXB Reference Implementation for GlassFish + https://jaxb.java.net/ + + + + CDDL 1.1 + https://glassfish.java.net/public/CDDL+GPL_1_1.html + repo + +Copyright (c) 2010 Oracle and/or its affiliates. + + + + + + + + javax.xml.bind + jaxb-api + JAXB API bundle for GlassFish V3 + https://jaxb.java.net/ + + + + CDDL 1.1 + https://glassfish.java.net/public/CDDL+GPL_1_1.html + repo + +Copyright (c) 2010 Oracle and/or its affiliates. + + + + + + + + + javax.servlet + servlet-api + Java Servlet API v2.5 + http://search.maven.org/#artifactdetails%7Cjavax.servlet%7Cservlet-api%7C2.5%7Cjar + + + + Common Development and Distribution License (CDDL) v1.0 + https://glassfish.dev.java.net/public/CDDLv1.0.html + repo + +Copyright 1999-2005 Sun Microsystems, Inc. +Portions copyright 2002 International Business Machines Corporation +Portions copyright Apache Software Foundation + + + + + + + + org.mortbay.jetty + servlet-api-2.5 + Servlet Specification 2.5 API + http://www.eclipse.org/jetty/ + + + + Common Development and Distribution License (CDDL) v1.0 + https://glassfish.dev.java.net/public/CDDLv1.0.html + repo + +Copyright 1999-2005 Sun Microsystems, Inc. +Portions copyright 2002 International Business Machines Corporation +Portions copyright Apache Software Foundation + + + + + + + + org.mortbay.jetty + jsp-2.1 + JSP2.1 Jasper implementation from Glassfish + http://www.eclipse.org/jetty/ + + + + Common Development and Distribution License (CDDL) v1.0 + https://glassfish.dev.java.net/public/CDDLv1.0.html + repo + + +Copyright 2005 Sun Microsystems, Inc. and portions Copyright Apache Software Foundation. + + + + + + + + org.jamon + jamon-runtime + Jamon runtime support classes + http://www.jamon.org/ + + + + http://www.mozilla.org/MPL/MPL-1.1.txt + Mozilla Public License Version 1.1 + repo + + MOZILLA PUBLIC LICENSE + Version 1.1 + + --------------- + +1. Definitions. + + 1.0.1. "Commercial Use" means distribution or otherwise making the + Covered Code available to a third party. + + 1.1. "Contributor" means each entity that creates or contributes to + the creation of Modifications. + + 1.2. "Contributor Version" means the combination of the Original + Code, prior Modifications used by a Contributor, and the Modifications + made by that particular Contributor. + + 1.3. "Covered Code" means the Original Code or Modifications or the + combination of the Original Code and Modifications, in each case + including portions thereof. + + 1.4. "Electronic Distribution Mechanism" means a mechanism generally + accepted in the software development community for the electronic + transfer of data. + + 1.5. "Executable" means Covered Code in any form other than Source + Code. + + 1.6. "Initial Developer" means the individual or entity identified + as the Initial Developer in the Source Code notice required by Exhibit + A. + + 1.7. "Larger Work" means a work which combines Covered Code or + portions thereof with code not governed by the terms of this License. + + 1.8. "License" means this document. + + 1.8.1. "Licensable" means having the right to grant, to the maximum + extent possible, whether at the time of the initial grant or + subsequently acquired, any and all of the rights conveyed herein. + + 1.9. "Modifications" means any addition to or deletion from the + substance or structure of either the Original Code or any previous + Modifications. When Covered Code is released as a series of files, a + Modification is: + A. Any addition to or deletion from the contents of a file + containing Original Code or previous Modifications. + + B. Any new file that contains any part of the Original Code or + previous Modifications. + + 1.10. "Original Code" means Source Code of computer software code + which is described in the Source Code notice required by Exhibit A as + Original Code, and which, at the time of its release under this + License is not already Covered Code governed by this License. + + 1.10.1. "Patent Claims" means any patent claim(s), now owned or + hereafter acquired, including without limitation, method, process, + and apparatus claims, in any patent Licensable by grantor. + + 1.11. "Source Code" means the preferred form of the Covered Code for + making modifications to it, including all modules it contains, plus + any associated interface definition files, scripts used to control + compilation and installation of an Executable, or source code + differential comparisons against either the Original Code or another + well known, available Covered Code of the Contributor's choice. The + Source Code can be in a compressed or archival form, provided the + appropriate decompression or de-archiving software is widely available + for no charge. + + 1.12. "You" (or "Your") means an individual or a legal entity + exercising rights under, and complying with all of the terms of, this + License or a future version of this License issued under Section 6.1. + For legal entities, "You" includes any entity which controls, is + controlled by, or is under common control with You. For purposes of + this definition, "control" means (a) the power, direct or indirect, + to cause the direction or management of such entity, whether by + contract or otherwise, or (b) ownership of more than fifty percent + (50%) of the outstanding shares or beneficial ownership of such + entity. + +2. Source Code License. + + 2.1. The Initial Developer Grant. + The Initial Developer hereby grants You a world-wide, royalty-free, + non-exclusive license, subject to third party intellectual property + claims: + (a) under intellectual property rights (other than patent or + trademark) Licensable by Initial Developer to use, reproduce, + modify, display, perform, sublicense and distribute the Original + Code (or portions thereof) with or without Modifications, and/or + as part of a Larger Work; and + + (b) under Patents Claims infringed by the making, using or + selling of Original Code, to make, have made, use, practice, + sell, and offer for sale, and/or otherwise dispose of the + Original Code (or portions thereof). + + (c) the licenses granted in this Section 2.1(a) and (b) are + effective on the date Initial Developer first distributes + Original Code under the terms of this License. + + (d) Notwithstanding Section 2.1(b) above, no patent license is + granted: 1) for code that You delete from the Original Code; 2) + separate from the Original Code; or 3) for infringements caused + by: i) the modification of the Original Code or ii) the + combination of the Original Code with other software or devices. + + 2.2. Contributor Grant. + Subject to third party intellectual property claims, each Contributor + hereby grants You a world-wide, royalty-free, non-exclusive license + + (a) under intellectual property rights (other than patent or + trademark) Licensable by Contributor, to use, reproduce, modify, + display, perform, sublicense and distribute the Modifications + created by such Contributor (or portions thereof) either on an + unmodified basis, with other Modifications, as Covered Code + and/or as part of a Larger Work; and + + (b) under Patent Claims infringed by the making, using, or + selling of Modifications made by that Contributor either alone + and/or in combination with its Contributor Version (or portions + of such combination), to make, use, sell, offer for sale, have + made, and/or otherwise dispose of: 1) Modifications made by that + Contributor (or portions thereof); and 2) the combination of + Modifications made by that Contributor with its Contributor + Version (or portions of such combination). + + (c) the licenses granted in Sections 2.2(a) and 2.2(b) are + effective on the date Contributor first makes Commercial Use of + the Covered Code. + + (d) Notwithstanding Section 2.2(b) above, no patent license is + granted: 1) for any code that Contributor has deleted from the + Contributor Version; 2) separate from the Contributor Version; + 3) for infringements caused by: i) third party modifications of + Contributor Version or ii) the combination of Modifications made + by that Contributor with other software (except as part of the + Contributor Version) or other devices; or 4) under Patent Claims + infringed by Covered Code in the absence of Modifications made by + that Contributor. + +3. Distribution Obligations. + + 3.1. Application of License. + The Modifications which You create or to which You contribute are + governed by the terms of this License, including without limitation + Section 2.2. The Source Code version of Covered Code may be + distributed only under the terms of this License or a future version + of this License released under Section 6.1, and You must include a + copy of this License with every copy of the Source Code You + distribute. You may not offer or impose any terms on any Source Code + version that alters or restricts the applicable version of this + License or the recipients' rights hereunder. However, You may include + an additional document offering the additional rights described in + Section 3.5. + + 3.2. Availability of Source Code. + Any Modification which You create or to which You contribute must be + made available in Source Code form under the terms of this License + either on the same media as an Executable version or via an accepted + Electronic Distribution Mechanism to anyone to whom you made an + Executable version available; and if made available via Electronic + Distribution Mechanism, must remain available for at least twelve (12) + months after the date it initially became available, or at least six + (6) months after a subsequent version of that particular Modification + has been made available to such recipients. You are responsible for + ensuring that the Source Code version remains available even if the + Electronic Distribution Mechanism is maintained by a third party. + + 3.3. Description of Modifications. + You must cause all Covered Code to which You contribute to contain a + file documenting the changes You made to create that Covered Code and + the date of any change. You must include a prominent statement that + the Modification is derived, directly or indirectly, from Original + Code provided by the Initial Developer and including the name of the + Initial Developer in (a) the Source Code, and (b) in any notice in an + Executable version or related documentation in which You describe the + origin or ownership of the Covered Code. + + 3.4. Intellectual Property Matters + (a) Third Party Claims. + If Contributor has knowledge that a license under a third party's + intellectual property rights is required to exercise the rights + granted by such Contributor under Sections 2.1 or 2.2, + Contributor must include a text file with the Source Code + distribution titled "LEGAL" which describes the claim and the + party making the claim in sufficient detail that a recipient will + know whom to contact. If Contributor obtains such knowledge after + the Modification is made available as described in Section 3.2, + Contributor shall promptly modify the LEGAL file in all copies + Contributor makes available thereafter and shall take other steps + (such as notifying appropriate mailing lists or newsgroups) + reasonably calculated to inform those who received the Covered + Code that new knowledge has been obtained. + + (b) Contributor APIs. + If Contributor's Modifications include an application programming + interface and Contributor has knowledge of patent licenses which + are reasonably necessary to implement that API, Contributor must + also include this information in the LEGAL file. + + (c) Representations. + Contributor represents that, except as disclosed pursuant to + Section 3.4(a) above, Contributor believes that Contributor's + Modifications are Contributor's original creation(s) and/or + Contributor has sufficient rights to grant the rights conveyed by + this License. + + 3.5. Required Notices. + You must duplicate the notice in Exhibit A in each file of the Source + Code. If it is not possible to put such notice in a particular Source + Code file due to its structure, then You must include such notice in a + location (such as a relevant directory) where a user would be likely + to look for such a notice. If You created one or more Modification(s) + You may add your name as a Contributor to the notice described in + Exhibit A. You must also duplicate this License in any documentation + for the Source Code where You describe recipients' rights or ownership + rights relating to Covered Code. You may choose to offer, and to + charge a fee for, warranty, support, indemnity or liability + obligations to one or more recipients of Covered Code. However, You + may do so only on Your own behalf, and not on behalf of the Initial + Developer or any Contributor. You must make it absolutely clear than + any such warranty, support, indemnity or liability obligation is + offered by You alone, and You hereby agree to indemnify the Initial + Developer and every Contributor for any liability incurred by the + Initial Developer or such Contributor as a result of warranty, + support, indemnity or liability terms You offer. + + 3.6. Distribution of Executable Versions. + You may distribute Covered Code in Executable form only if the + requirements of Section 3.1-3.5 have been met for that Covered Code, + and if You include a notice stating that the Source Code version of + the Covered Code is available under the terms of this License, + including a description of how and where You have fulfilled the + obligations of Section 3.2. The notice must be conspicuously included + in any notice in an Executable version, related documentation or + collateral in which You describe recipients' rights relating to the + Covered Code. You may distribute the Executable version of Covered + Code or ownership rights under a license of Your choice, which may + contain terms different from this License, provided that You are in + compliance with the terms of this License and that the license for the + Executable version does not attempt to limit or alter the recipient's + rights in the Source Code version from the rights set forth in this + License. If You distribute the Executable version under a different + license You must make it absolutely clear that any terms which differ + from this License are offered by You alone, not by the Initial + Developer or any Contributor. You hereby agree to indemnify the + Initial Developer and every Contributor for any liability incurred by + the Initial Developer or such Contributor as a result of any such + terms You offer. + + 3.7. Larger Works. + You may create a Larger Work by combining Covered Code with other code + not governed by the terms of this License and distribute the Larger + Work as a single product. In such a case, You must make sure the + requirements of this License are fulfilled for the Covered Code. + +4. Inability to Comply Due to Statute or Regulation. + + If it is impossible for You to comply with any of the terms of this + License with respect to some or all of the Covered Code due to + statute, judicial order, or regulation then You must: (a) comply with + the terms of this License to the maximum extent possible; and (b) + describe the limitations and the code they affect. Such description + must be included in the LEGAL file described in Section 3.4 and must + be included with all distributions of the Source Code. Except to the + extent prohibited by statute or regulation, such description must be + sufficiently detailed for a recipient of ordinary skill to be able to + understand it. + +5. Application of this License. + + This License applies to code to which the Initial Developer has + attached the notice in Exhibit A and to related Covered Code. + +6. Versions of the License. + + 6.1. New Versions. + Netscape Communications Corporation ("Netscape") may publish revised + and/or new versions of the License from time to time. Each version + will be given a distinguishing version number. + + 6.2. Effect of New Versions. + Once Covered Code has been published under a particular version of the + License, You may always continue to use it under the terms of that + version. You may also choose to use such Covered Code under the terms + of any subsequent version of the License published by Netscape. No one + other than Netscape has the right to modify the terms applicable to + Covered Code created under this License. + + 6.3. Derivative Works. + If You create or use a modified version of this License (which you may + only do in order to apply it to code which is not already Covered Code + governed by this License), You must (a) rename Your license so that + the phrases "Mozilla", "MOZILLAPL", "MOZPL", "Netscape", + "MPL", "NPL" or any confusingly similar phrase do not appear in your + license (except to note that your license differs from this License) + and (b) otherwise make it clear that Your version of the license + contains terms which differ from the Mozilla Public License and + Netscape Public License. (Filling in the name of the Initial + Developer, Original Code or Contributor in the notice described in + Exhibit A shall not of themselves be deemed to be modifications of + this License.) + +7. DISCLAIMER OF WARRANTY. + + COVERED CODE IS PROVIDED UNDER THIS LICENSE ON AN "AS IS" BASIS, + WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, + WITHOUT LIMITATION, WARRANTIES THAT THE COVERED CODE IS FREE OF + DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR PURPOSE OR NON-INFRINGING. + THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE COVERED CODE + IS WITH YOU. SHOULD ANY COVERED CODE PROVE DEFECTIVE IN ANY RESPECT, + YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER CONTRIBUTOR) ASSUME THE + COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION. THIS DISCLAIMER + OF WARRANTY CONSTITUTES AN ESSENTIAL PART OF THIS LICENSE. NO USE OF + ANY COVERED CODE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS DISCLAIMER. + +8. TERMINATION. + + 8.1. This License and the rights granted hereunder will terminate + automatically if You fail to comply with terms herein and fail to cure + such breach within 30 days of becoming aware of the breach. All + sublicenses to the Covered Code which are properly granted shall + survive any termination of this License. Provisions which, by their + nature, must remain in effect beyond the termination of this License + shall survive. + + 8.2. If You initiate litigation by asserting a patent infringement + claim (excluding declatory judgment actions) against Initial Developer + or a Contributor (the Initial Developer or Contributor against whom + You file such action is referred to as "Participant") alleging that: + + (a) such Participant's Contributor Version directly or indirectly + infringes any patent, then any and all rights granted by such + Participant to You under Sections 2.1 and/or 2.2 of this License + shall, upon 60 days notice from Participant terminate prospectively, + unless if within 60 days after receipt of notice You either: (i) + agree in writing to pay Participant a mutually agreeable reasonable + royalty for Your past and future use of Modifications made by such + Participant, or (ii) withdraw Your litigation claim with respect to + the Contributor Version against such Participant. If within 60 days + of notice, a reasonable royalty and payment arrangement are not + mutually agreed upon in writing by the parties or the litigation claim + is not withdrawn, the rights granted by Participant to You under + Sections 2.1 and/or 2.2 automatically terminate at the expiration of + the 60 day notice period specified above. + + (b) any software, hardware, or device, other than such Participant's + Contributor Version, directly or indirectly infringes any patent, then + any rights granted to You by such Participant under Sections 2.1(b) + and 2.2(b) are revoked effective as of the date You first made, used, + sold, distributed, or had made, Modifications made by that + Participant. + + 8.3. If You assert a patent infringement claim against Participant + alleging that such Participant's Contributor Version directly or + indirectly infringes any patent where such claim is resolved (such as + by license or settlement) prior to the initiation of patent + infringement litigation, then the reasonable value of the licenses + granted by such Participant under Sections 2.1 or 2.2 shall be taken + into account in determining the amount or value of any payment or + license. + + 8.4. In the event of termination under Sections 8.1 or 8.2 above, + all end user license agreements (excluding distributors and resellers) + which have been validly granted by You or any distributor hereunder + prior to termination shall survive termination. + +9. LIMITATION OF LIABILITY. + + UNDER NO CIRCUMSTANCES AND UNDER NO LEGAL THEORY, WHETHER TORT + (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, SHALL YOU, THE INITIAL + DEVELOPER, ANY OTHER CONTRIBUTOR, OR ANY DISTRIBUTOR OF COVERED CODE, + OR ANY SUPPLIER OF ANY OF SUCH PARTIES, BE LIABLE TO ANY PERSON FOR + ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY + CHARACTER INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF GOODWILL, + WORK STOPPAGE, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER + COMMERCIAL DAMAGES OR LOSSES, EVEN IF SUCH PARTY SHALL HAVE BEEN + INFORMED OF THE POSSIBILITY OF SUCH DAMAGES. THIS LIMITATION OF + LIABILITY SHALL NOT APPLY TO LIABILITY FOR DEATH OR PERSONAL INJURY + RESULTING FROM SUCH PARTY'S NEGLIGENCE TO THE EXTENT APPLICABLE LAW + PROHIBITS SUCH LIMITATION. SOME JURISDICTIONS DO NOT ALLOW THE + EXCLUSION OR LIMITATION OF INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO + THIS EXCLUSION AND LIMITATION MAY NOT APPLY TO YOU. + +10. U.S. GOVERNMENT END USERS. + + The Covered Code is a "commercial item," as that term is defined in + 48 C.F.R. 2.101 (Oct. 1995), consisting of "commercial computer + software" and "commercial computer software documentation," as such + terms are used in 48 C.F.R. 12.212 (Sept. 1995). Consistent with 48 + C.F.R. 12.212 and 48 C.F.R. 227.7202-1 through 227.7202-4 (June 1995), + all U.S. Government End Users acquire Covered Code with only those + rights set forth herein. + +11. MISCELLANEOUS. + + This License represents the complete agreement concerning subject + matter hereof. If any provision of this License is held to be + unenforceable, such provision shall be reformed only to the extent + necessary to make it enforceable. This License shall be governed by + California law provisions (except to the extent applicable law, if + any, provides otherwise), excluding its conflict-of-law provisions. + With respect to disputes in which at least one party is a citizen of, + or an entity chartered or registered to do business in the United + States of America, any litigation relating to this License shall be + subject to the jurisdiction of the Federal Courts of the Northern + District of California, with venue lying in Santa Clara County, + California, with the losing party responsible for costs, including + without limitation, court costs and reasonable attorneys' fees and + expenses. The application of the United Nations Convention on + Contracts for the International Sale of Goods is expressly excluded. + Any law or regulation which provides that the language of a contract + shall be construed against the drafter shall not apply to this + License. + +12. RESPONSIBILITY FOR CLAIMS. + + As between Initial Developer and the Contributors, each party is + responsible for claims and damages arising, directly or indirectly, + out of its utilization of rights under this License and You agree to + work with Initial Developer and Contributors to distribute such + responsibility on an equitable basis. Nothing herein is intended or + shall be deemed to constitute any admission of liability. + +13. MULTIPLE-LICENSED CODE. + + Initial Developer may designate portions of the Covered Code as + "Multiple-Licensed". "Multiple-Licensed" means that the Initial + Developer permits you to utilize portions of the Covered Code under + Your choice of the MPL or the alternative licenses, if any, specified + by the Initial Developer in the file described in Exhibit A. + +EXHIBIT A -Mozilla Public License. + + ``The contents of this file are subject to the Mozilla Public License + Version 1.1 (the "License"); you may not use this file except in + compliance with the License. You may obtain a copy of the License at + http://www.mozilla.org/MPL/ + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the + License for the specific language governing rights and limitations + under the License. + + The Original Code is ______________________________________. + + The Initial Developer of the Original Code is ________________________. + Portions created by ______________________ are Copyright (C) ______ + _______________________. All Rights Reserved. + + Contributor(s): ______________________________________. + + Alternatively, the contents of this file may be used under the terms + of the _____ license (the "[___] License"), in which case the + provisions of [______] License are applicable instead of those + above. If you wish to allow use of your version of this file only + under the terms of the [____] License and not to allow others to use + your version of this file under the MPL, indicate your decision by + deleting the provisions above and replace them with the notice and + other provisions required by the [___] License. If you do not delete + the provisions above, a recipient may use your version of this file + under either the MPL or the [___] License." + + [NOTE: The text of this Exhibit A may differ slightly from the text of + the notices in the Source Code files of the Original Code. You should + use the text of this Exhibit A rather than the text found in the + Original Code Source Code for Your Modifications.] + + + + + + + + + + org.jruby + jruby-complete + JRuby Complete + http://www.jruby.org/ + + + + Common Public License Version 1.0 + http://www-128.ibm.com/developerworks/library/os-cpl.html + repo + +Copyright (c) 2007-2011 The JRuby project + + + + + + + + + org.eclipse.jdt + core + Eclipse JDT Core + http://www.eclipse.org/jdt/ + + + + Eclipse Public License v1.0 + http://www.eclipse.org/org/documents/epl-v10.php + repo + +Eclipse Public License - v 1.0 + +THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC +LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM +CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. + +1. DEFINITIONS + +"Contribution" means: + + a) in the case of the initial Contributor, the initial code and +documentation distributed under this Agreement, and + + b) in the case of each subsequent Contributor: + + i) changes to the Program, and + + ii) additions to the Program; + + where such changes and/or additions to the Program originate from and are +distributed by that particular Contributor. A Contribution 'originates' from a +Contributor if it was added to the Program by such Contributor itself or anyone +acting on such Contributor's behalf. Contributions do not include additions to +the Program which: (i) are separate modules of software distributed in +conjunction with the Program under their own license agreement, and (ii) are +not derivative works of the Program. + +"Contributor" means any person or entity that distributes the Program. + +"Licensed Patents" mean patent claims licensable by a Contributor which are +necessarily infringed by the use or sale of its Contribution alone or when +combined with the Program. + +"Program" means the Contributions distributed in accordance with this +Agreement. + +"Recipient" means anyone who receives the Program under this Agreement, +including all Contributors. + +2. GRANT OF RIGHTS + +a) Subject to the terms of this Agreement, each Contributor hereby grants +Recipient a non-exclusive, worldwide, royalty-free copyright license to +reproduce, prepare derivative works of, publicly display, publicly perform, +distribute and sublicense the Contribution of such Contributor, if any, and +such derivative works, in source code and object code form. + +b) Subject to the terms of this Agreement, each Contributor hereby grants +Recipient a non-exclusive, worldwide, royalty-free patent license under +Licensed Patents to make, use, sell, offer to sell, import and otherwise +transfer the Contribution of such Contributor, if any, in source code and +object code form. This patent license shall apply to the combination of the +Contribution and the Program if, at the time the Contribution is added by the +Contributor, such addition of the Contribution causes such combination to be +covered by the Licensed Patents. The patent license shall not apply to any +other combinations which include the Contribution. No hardware per se is +licensed hereunder. + +c) Recipient understands that although each Contributor grants the licenses to +its Contributions set forth herein, no assurances are provided by any +Contributor that the Program does not infringe the patent or other intellectual +property rights of any other entity. Each Contributor disclaims any liability +to Recipient for claims brought by any other entity based on infringement of +intellectual property rights or otherwise. As a condition to exercising the +rights and licenses granted hereunder, each Recipient hereby assumes sole +responsibility to secure any other intellectual property rights needed, if any. +For example, if a third party patent license is required to allow Recipient to +distribute the Program, it is Recipient's responsibility to acquire that +license before distributing the Program. + +d) Each Contributor represents that to its knowledge it has sufficient +copyright rights in its Contribution, if any, to grant the copyright license +set forth in this Agreement. + +3. REQUIREMENTS + +A Contributor may choose to distribute the Program in object code form under +its own license agreement, provided that: + + a) it complies with the terms and conditions of this Agreement; and + + b) its license agreement: + + i) effectively disclaims on behalf of all Contributors all warranties and +conditions, express and implied, including warranties or conditions of title +and non-infringement, and implied warranties or conditions of merchantability +and fitness for a particular purpose; + + ii) effectively excludes on behalf of all Contributors all liability for +damages, including direct, indirect, special, incidental and consequential +damages, such as lost profits; + + iii) states that any provisions which differ from this Agreement are +offered by that Contributor alone and not by any other party; and + + iv) states that source code for the Program is available from such +Contributor, and informs licensees how to obtain it in a reasonable manner on +or through a medium customarily used for software exchange. + +When the Program is made available in source code form: + + a) it must be made available under this Agreement; and + + b) a copy of this Agreement must be included with each copy of the Program. + +Contributors may not remove or alter any copyright notices contained within the +Program. + +Each Contributor must identify itself as the originator of its Contribution, if +any, in a manner that reasonably allows subsequent Recipients to identify the +originator of the Contribution. + +4. COMMERCIAL DISTRIBUTION + +Commercial distributors of software may accept certain responsibilities with +respect to end users, business partners and the like. While this license is +intended to facilitate the commercial use of the Program, the Contributor who +includes the Program in a commercial product offering should do so in a manner +which does not create potential liability for other Contributors. Therefore, if +a Contributor includes the Program in a commercial product offering, such +Contributor ("Commercial Contributor") hereby agrees to defend and indemnify +every other Contributor ("Indemnified Contributor") against any losses, damages +and costs (collectively "Losses") arising from claims, lawsuits and other legal +actions brought by a third party against the Indemnified Contributor to the +extent caused by the acts or omissions of such Commercial Contributor in +connection with its distribution of the Program in a commercial product +offering. The obligations in this section do not apply to any claims or Losses +relating to any actual or alleged intellectual property infringement. In order +to qualify, an Indemnified Contributor must: a) promptly notify the Commercial +Contributor in writing of such claim, and b) allow the Commercial Contributor +to control, and cooperate with the Commercial Contributor in, the defense and +any related settlement negotiations. The Indemnified Contributor may +participate in any such claim at its own expense. + +For example, a Contributor might include the Program in a commercial product +offering, Product X. That Contributor is then a Commercial Contributor. If that +Commercial Contributor then makes performance claims, or offers warranties +related to Product X, those performance claims and warranties are such +Commercial Contributor's responsibility alone. Under this section, the +Commercial Contributor would have to defend claims against the other +Contributors related to those performance claims and warranties, and if a court +requires any other Contributor to pay any damages as a result, the Commercial +Contributor must pay those damages. + +5. NO WARRANTY + +EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR +IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, +NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each +Recipient is solely responsible for determining the appropriateness of using +and distributing the Program and assumes all risks associated with its exercise +of rights under this Agreement , including but not limited to the risks and +costs of program errors, compliance with applicable laws, damage to or loss of +data, programs or equipment, and unavailability or interruption of operations. + +6. DISCLAIMER OF LIABILITY + +EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY +CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION LOST +PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY +WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS +GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +7. GENERAL + +If any provision of this Agreement is invalid or unenforceable under applicable +law, it shall not affect the validity or enforceability of the remainder of the +terms of this Agreement, and without further action by the parties hereto, such +provision shall be reformed to the minimum extent necessary to make such +provision valid and enforceable. + +If Recipient institutes patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that the Program itself +(excluding combinations of the Program with other software or hardware) +infringes such Recipient's patent(s), then such Recipient's rights granted +under Section 2(b) shall terminate as of the date such litigation is filed. + +All Recipient's rights under this Agreement shall terminate if it fails to +comply with any of the material terms or conditions of this Agreement and does +not cure such failure in a reasonable period of time after becoming aware of +such noncompliance. If all Recipient's rights under this Agreement terminate, +Recipient agrees to cease use and distribution of the Program as soon as +reasonably practicable. However, Recipient's obligations under this Agreement +and any licenses granted by Recipient relating to the Program shall continue +and survive. + +Everyone is permitted to copy and distribute copies of this Agreement, but in +order to avoid inconsistency the Agreement is copyrighted and may only be +modified in the following manner. The Agreement Steward reserves the right to +publish new versions (including revisions) of this Agreement from time to time. +No one other than the Agreement Steward has the right to modify this Agreement. +The Eclipse Foundation is the initial Agreement Steward. The Eclipse Foundation +may assign the responsibility to serve as the Agreement Steward to a suitable +separate entity. Each new version of the Agreement will be given a +distinguishing version number. The Program (including Contributions) may always +be distributed subject to the version of the Agreement under which it was +received. In addition, after a new version of the Agreement is published, +Contributor may elect to distribute the Program (including its Contributions) +under the new version. Except as expressly stated in Sections 2(a) and 2(b) +above, Recipient receives no rights or licenses to the intellectual property of +any Contributor under this Agreement, whether expressly, by implication, +estoppel or otherwise. All rights in the Program not expressly granted under +this Agreement are reserved. + +This Agreement is governed by the laws of the State of New York and the +intellectual property laws of the United States of America. No party to this +Agreement will bring a legal action under this Agreement more than one year +after the cause of action arose. Each party waives its rights to a jury trial +in any resulting litigation. + + + + + + diff --git a/src/docbkx/book.xml b/src/docbkx/book.xml index dbc43bf8b04a..05133bad2df8 100644 --- a/src/docbkx/book.xml +++ b/src/docbkx/book.xml @@ -1,7 +1,6 @@ HBase and Schema Design A good general introduction on the strength and weaknesses modelling on - the various non-rdbms datastores is Ian Varleys' Master thesis, + the various non-rdbms datastores is Ian Varley's Master thesis, No Relation: The Mixed Blessings of Non-Relational Databases. Recommended. Also, read for how HBase stores data internally. @@ -575,31 +590,31 @@ htable.put(put); Tables must be disabled when making ColumnFamily modifications, for example.. -Configuration config = HBaseConfiguration.create(); -HBaseAdmin admin = new HBaseAdmin(conf); +Configuration config = HBaseConfiguration.create(); +HBaseAdmin admin = new HBaseAdmin(conf); String table = "myTable"; -admin.disableTable(table); +admin.disableTable(table); HColumnDescriptor cf1 = ...; admin.addColumn(table, cf1); // adding new ColumnFamily HColumnDescriptor cf2 = ...; admin.modifyColumn(table, cf2); // modifying existing ColumnFamily -admin.enableTable(table); +admin.enableTable(table); See for more information about configuring client connections. Note: online schema changes are supported in the 0.92.x codebase, but the 0.90.x codebase requires the table to be disabled. -
Schema Updates +
Schema Updates When changes are made to either Tables or ColumnFamilies (e.g., region size, block size), these changes take effect the next time there is a major compaction and the StoreFiles get re-written. See for more information on StoreFiles.
-
+
On the number of column families @@ -610,7 +625,7 @@ admin.enableTable(table); if one column family is carrying the bulk of the data bringing on flushes, the adjacent families will also be flushed though the amount of data they carry is small. When many column families the flushing and compaction interaction can make for a bunch of needless i/o loading (To be addressed by - changing flushing and compaction to work on a per column family basis). For more information + changing flushing and compaction to work on a per column family basis). For more information on compactions, see <xref linkend="compaction"/>. </para> <para>Try to make do with one column family if you can in your schemas. Only introduce a @@ -618,9 +633,9 @@ admin.enableTable(table); i.e. you query one column family or the other but usually not both at the one time. </para> <section xml:id="number.of.cfs.card"><title>Cardinality of ColumnFamilies - Where multiple ColumnFamilies exist in a single table, be aware of the cardinality (i.e., number of rows). - If ColumnFamilyA has 1 million rows and ColumnFamilyB has 1 billion rows, ColumnFamilyA's data will likely be spread - across many, many regions (and RegionServers). This makes mass scans for ColumnFamilyA less efficient. + Where multiple ColumnFamilies exist in a single table, be aware of the cardinality (i.e., number of rows). + If ColumnFamilyA has 1 million rows and ColumnFamilyB has 1 billion rows, ColumnFamilyA's data will likely be spread + across many, many regions (and RegionServers). This makes mass scans for ColumnFamilyA less efficient.
@@ -632,7 +647,7 @@ admin.enableTable(table); In the HBase chapter of Tom White's book Hadoop: The Definitive Guide (O'Reilly) there is a an optimization note on watching out for a phenomenon where an import process walks in lock-step with all clients in concert pounding one of the table's regions (and thus, a single node), then moving onto the next region, etc. With monotonically increasing row-keys (i.e., using a timestamp), this will happen. See this comic by IKai Lan on why monotonically increasing row keys are problematic in BigTable-like datastores: monotonically increasing values are bad. The pile-up on a single region brought on - by monotonically increasing keys can be mitigated by randomizing the input records to not be in sorted order, but in general its best to avoid using a timestamp or a sequence (e.g. 1, 2, 3) as the row-key. + by monotonically increasing keys can be mitigated by randomizing the input records to not be in sorted order, but in general it's best to avoid using a timestamp or a sequence (e.g. 1, 2, 3) as the row-key. @@ -670,20 +685,20 @@ admin.enableTable(table); See for more information on HBase stores data internally to see why this is important.
Column Families Try to keep the ColumnFamily names as small as possible, preferably one character (e.g. "d" for data/default). - + See for more information on HBase stores data internally to see why this is important.
Attributes Although verbose attribute names (e.g., "myVeryImportantAttribute") are easier to read, prefer shorter attribute names (e.g., "via") to store in HBase. - + See for more information on HBase stores data internally to see why this is important.
Rowkey Length - Keep them as short as is reasonable such that they can still be useful for required data access (e.g., Get vs. Scan). + Keep them as short as is reasonable such that they can still be useful for required data access (e.g., Get vs. Scan). A short key that is useless for data access is not better than a longer key with better get/scan properties. Expect tradeoffs when designing rowkeys. - +
Byte Patterns A long is 8 bytes. You can store an unsigned number up to 18,446,744,073,709,551,615 in those eight bytes. @@ -696,28 +711,28 @@ admin.enableTable(table); long l = 1234567890L; byte[] lb = Bytes.toBytes(l); System.out.println("long bytes length: " + lb.length); // returns 8 - + String s = "" + l; byte[] sb = Bytes.toBytes(s); System.out.println("long as string length: " + sb.length); // returns 10 - -// hash + +// hash // MessageDigest md = MessageDigest.getInstance("MD5"); byte[] digest = md.digest(Bytes.toBytes(s)); System.out.println("md5 digest bytes length: " + digest.length); // returns 16 - + String sDigest = new String(digest); byte[] sbDigest = Bytes.toBytes(sDigest); -System.out.println("md5 digest as string length: " + sbDigest.length); // returns 26 - +System.out.println("md5 digest as string length: " + sbDigest.length); // returns 26 +
- +
Reverse Timestamps A common problem in database processing is quickly finding the most recent version of a value. A technique using reverse timestamps - as a part of the key can help greatly with a special case of this problem. Also found in the HBase chapter of Tom White's book Hadoop: The Definitive Guide (O'Reilly), + as a part of the key can help greatly with a special case of this problem. Also found in the HBase chapter of Tom White's book Hadoop: The Definitive Guide (O'Reilly), the technique involves appending (Long.MAX_VALUE - timestamp) to the end of any key, e.g., [key][reverse_timestamp]. The most recent value for [key] in a table can be found by performing a Scan for [key] and obtaining the first record. Since HBase keys @@ -734,11 +749,76 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret
Immutability of Rowkeys Rowkeys cannot be changed. The only way they can be "changed" in a table is if the row is deleted and then re-inserted. - This is a fairly common question on the HBase dist-list so it pays to get the rowkeys right the first time (and/or before you've + This is a fairly common question on the HBase dist-list so it pays to get the rowkeys right the first time (and/or before you've inserted a lot of data).
- +
Relationship Between RowKeys and Region Splits + If you pre-split your table, it is critical to understand how your rowkey will be distributed across + the region boundaries. As an example of why this is important, consider the example of using displayable hex characters as the + lead position of the key (e.g., ""0000000000000000" to "ffffffffffffffff"). Running those key ranges through Bytes.split + (which is the split strategy used when creating regions in HBaseAdmin.createTable(byte[] startKey, byte[] endKey, numRegions) + for 10 regions will generate the following splits... + + + +48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 // 0 +54 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 // 6 +61 -67 -67 -67 -67 -67 -67 -67 -67 -67 -67 -67 -67 -67 -67 -68 // = +68 -124 -124 -124 -124 -124 -124 -124 -124 -124 -124 -124 -124 -124 -124 -126 // D +75 75 75 75 75 75 75 75 75 75 75 75 75 75 75 72 // K +82 18 18 18 18 18 18 18 18 18 18 18 18 18 18 14 // R +88 -40 -40 -40 -40 -40 -40 -40 -40 -40 -40 -40 -40 -40 -40 -44 // X +95 -97 -97 -97 -97 -97 -97 -97 -97 -97 -97 -97 -97 -97 -97 -102 // _ +102 102 102 102 102 102 102 102 102 102 102 102 102 102 102 102 // f + + ... (note: the lead byte is listed to the right as a comment.) Given that the first split is a '0' and the last split is an 'f', + everything is great, right? Not so fast. + + The problem is that all the data is going to pile up in the first 2 regions and the last region thus creating a "lumpy" (and + possibly "hot") region problem. To understand why, refer to an ASCII Table. + '0' is byte 48, and 'f' is byte 102, but there is a huge gap in byte values (bytes 58 to 96) that will never appear in this + keyspace because the only values are [0-9] and [a-f]. Thus, the middle regions regions will + never be used. To make pre-spliting work with this example keyspace, a custom definition of splits (i.e., and not relying on the + built-in split method) is required. + + Lesson #1: Pre-splitting tables is generally a best practice, but you need to pre-split them in such a way that all the + regions are accessible in the keyspace. While this example demonstrated the problem with a hex-key keyspace, the same problem can happen + with any keyspace. Know your data. + + Lesson #2: While generally not advisable, using hex-keys (and more generally, displayable data) can still work with pre-split + tables as long as all the created regions are accessible in the keyspace. + + To conclude this example, the following is an example of how appropriate splits can be pre-created for hex-keys:. + +public static boolean createTable(HBaseAdmin admin, HTableDescriptor table, byte[][] splits) +throws IOException { + try { + admin.createTable( table, splits ); + return true; + } catch (TableExistsException e) { + logger.info("table " + table.getNameAsString() + " already exists"); + // the table already exists... + return false; + } +} + +public static byte[][] getHexSplits(String startKey, String endKey, int numRegions) { + byte[][] splits = new byte[numRegions-1][]; + BigInteger lowestKey = new BigInteger(startKey, 16); + BigInteger highestKey = new BigInteger(endKey, 16); + BigInteger range = highestKey.subtract(lowestKey); + BigInteger regionIncrement = range.divide(BigInteger.valueOf(numRegions)); + lowestKey = lowestKey.add(regionIncrement); + for(int i=0; i < numRegions-1;i++) { + BigInteger key = lowestKey.add(regionIncrement.multiply(BigInteger.valueOf(i))); + byte[] b = String.format("%016x", key).getBytes(); + splits[i] = b; + } + return splits; +} +
+
Number of Versions @@ -752,8 +832,8 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret stores different values per row by time (and qualifier). Excess versions are removed during major compactions. The number of max versions may need to be increased or decreased depending on application needs. </para> - <para>It is not recommended setting the number of max versions to an exceedingly high level (e.g., hundreds or more) unless those old values are - very dear to you because this will greatly increase StoreFile size. + <para>It is not recommended setting the number of max versions to an exceedingly high level (e.g., hundreds or more) unless those old values are + very dear to you because this will greatly increase StoreFile size. </para> </section> <section xml:id="schema.minversions"> @@ -778,24 +858,24 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret HBase supports a "bytes-in/bytes-out" interface via Put and Result, so anything that can be - converted to an array of bytes can be stored as a value. Input could be strings, numbers, complex objects, or even images as long as they can rendered as bytes. + converted to an array of bytes can be stored as a value. Input could be strings, numbers, complex objects, or even images as long as they can rendered as bytes. There are practical limits to the size of values (e.g., storing 10-50MB objects in HBase would probably be too much to ask); - search the mailling list for conversations on this topic. All rows in HBase conform to the datamodel, and - that includes versioning. Take that into consideration when making your design, as well as block size for the ColumnFamily. + search the mailling list for conversations on this topic. All rows in HBase conform to the datamodel, and + that includes versioning. Take that into consideration when making your design, as well as block size for the ColumnFamily.
Counters - One supported datatype that deserves special mention are "counters" (i.e., the ability to do atomic increments of numbers). See + One supported datatype that deserves special mention are "counters" (i.e., the ability to do atomic increments of numbers). See Increment in HTable. Synchronization on counters are done on the RegionServer, not in the client. -
+
Joins - If you have multiple tables, don't forget to factor in the potential for into the schema design. + If you have multiple tables, don't forget to factor in the potential for into the schema design.
@@ -828,22 +908,22 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret Secondary Indexes and Alternate Query Paths This section could also be titled "what if my table rowkey looks like this but I also want to query my table like that." - A common example on the dist-list is where a row-key is of the format "user-timestamp" but there are are reporting requirements on activity across users for certain + A common example on the dist-list is where a row-key is of the format "user-timestamp" but there are reporting requirements on activity across users for certain time ranges. Thus, selecting by user is easy because it is in the lead position of the key, but time is not. There is no single answer on the best way to handle this because it depends on... - Number of users + Number of users Data size and data arrival rate - Flexibility of reporting requirements (e.g., completely ad-hoc date selection vs. pre-configured ranges) - Desired execution speed of query (e.g., 90 seconds may be reasonable to some for an ad-hoc report, whereas it may be too long for others) + Flexibility of reporting requirements (e.g., completely ad-hoc date selection vs. pre-configured ranges) + Desired execution speed of query (e.g., 90 seconds may be reasonable to some for an ad-hoc report, whereas it may be too long for others) - ... and solutions are also influenced by the size of the cluster and how much processing power you have to throw at the solution. - Common techniques are in sub-sections below. This is a comprehensive, but not exhaustive, list of approaches. + ... and solutions are also influenced by the size of the cluster and how much processing power you have to throw at the solution. + Common techniques are in sub-sections below. This is a comprehensive, but not exhaustive, list of approaches. - It should not be a surprise that secondary indexes require additional cluster space and processing. + It should not be a surprise that secondary indexes require additional cluster space and processing. This is precisely what happens in an RDBMS because the act of creating an alternate index requires both space and processing cycles to update. RBDMS products - are more advanced in this regard to handle alternative index management out of the box. However, HBase scales better at larger data volumes, so this is a feature trade-off. + are more advanced in this regard to handle alternative index management out of the box. However, HBase scales better at larger data volumes, so this is a feature trade-off. Pay attention to when implementing any of these approaches. Additionally, see the David Butler response in this dist-list thread HBase, mail # user - Stargate+hbase @@ -860,7 +940,7 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret Periodic-Update Secondary Index - A secondary index could be created in an other table which is periodically updated via a MapReduce job. The job could be executed intra-day, but depending on + A secondary index could be created in an other table which is periodically updated via a MapReduce job. The job could be executed intra-day, but depending on load-strategy it could still potentially be out of sync with the main data table. See for more information.
@@ -868,7 +948,7 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret Dual-Write Secondary Index - Another strategy is to build the secondary index while publishing data to the cluster (e.g., write to data table, write to index table). + Another strategy is to build the secondary index while publishing data to the cluster (e.g., write to data table, write to index table). If this is approach is taken after a data table already exists, then bootstrapping will be needed for the secondary index with a MapReduce job (see ).
@@ -888,12 +968,12 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret
Schema Design Smackdown - This section will describe common schema design questions that appear on the dist-list. These are - general guidelines and not laws - each application must consider it's own needs. + This section will describe common schema design questions that appear on the dist-list. These are + general guidelines and not laws - each application must consider its own needs.
Rows vs. Versions A common question is whether one should prefer rows or HBase's built-in-versioning. The context is typically where there are - "a lot" of versions of a row to be retained (e.g., where it is significantly above the HBase default of 3 max versions). The + "a lot" of versions of a row to be retained (e.g., where it is significantly above the HBase default of 3 max versions). The rows-approach would require storing a timstamp in some portion of the rowkey so that they would not overwite with each successive update. Preference: Rows (generally speaking). @@ -901,18 +981,29 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret
Rows vs. Columns Another common question is whether one should prefer rows or columns. The context is typically in extreme cases of wide - tables, such as having 1 row with 1 million attributes, or 1 million rows with 1 columns apiece. + tables, such as having 1 row with 1 million attributes, or 1 million rows with 1 columns apiece. - Preference: Rows (generally speaking). To be clear, this guideline is in the context is in extremely wide cases, not in the - standard use-case where one needs to store a few dozen or hundred columns. + Preference: Rows (generally speaking). To be clear, this guideline is in the context is in extremely wide cases, not in the + standard use-case where one needs to store a few dozen or hundred columns. But there is also a middle path between these two + options, and that is "Rows as Columns."
+
Rows as Columns + The middle path between Rows vs. Columns is packing data that would be a separate row into columns, for certain rows. + OpenTSDB is the best example of this case where a single row represents a defined time-range, and then discrete events are treated as + columns. This approach is often more complex, and may require the additional complexity of re-writing your data, but has the + advantage of being I/O efficient. For an overview of this approach, see + Lessons Learned from OpenTSDB + from HBaseCon2012. + +
+
Operational and Performance Configuration Options See the Performance section for more information operational and performance schema design options, such as Bloom Filters, Table-configured regionsizes, compression, and blocksizes. -
+
Constraints HBase currently supports 'constraints' in traditional (SQL) database parlance. The advised usage for Constraints is in enforcing business rules for attributes in the table (eg. make sure values are in the range 1-10). @@ -942,9 +1033,9 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret
Custom Splitters - For those interested in implementing custom splitters, see the method getSplits in + For those interested in implementing custom splitters, see the method getSplits in TableInputFormatBase. - That is where the logic for map-task assignment resides. + That is where the logic for map-task assignment resides.
@@ -959,22 +1050,22 @@ System.out.println("md5 digest as string length: " + sbDigest.length); // ret Configuration config = HBaseConfiguration.create(); Job job = new Job(config, "ExampleRead"); job.setJarByClass(MyReadJob.class); // class that contains mapper - + Scan scan = new Scan(); scan.setCaching(500); // 1 is the default in Scan, which will be bad for MapReduce jobs scan.setCacheBlocks(false); // don't set to true for MR jobs // set other scan attrs ... - + TableMapReduceUtil.initTableMapperJob( tableName, // input HBase table name scan, // Scan instance to control CF and attribute selection MyMapper.class, // mapper - null, // mapper output key + null, // mapper output key null, // mapper output value job); job.setOutputFormatClass(NullOutputFormat.class); // because we aren't emitting anything from mapper - + boolean b = job.waitForCompletion(true); if (!b) { throw new IOException("error with job!"); @@ -987,24 +1078,24 @@ public static class MyMapper extends TableMapper<Text, Text> { public void map(ImmutableBytesWritable row, Result value, Context context) throws InterruptedException, IOException { // process data for the row from the Result instance. } -} +}
HBase MapReduce Read/Write Example - The following is an example of using HBase both as a source and as a sink with MapReduce. + The following is an example of using HBase both as a source and as a sink with MapReduce. This example will simply copy data from one table to another. Configuration config = HBaseConfiguration.create(); Job job = new Job(config,"ExampleReadWrite"); job.setJarByClass(MyReadWriteJob.class); // class that contains mapper - + Scan scan = new Scan(); scan.setCaching(500); // 1 is the default in Scan, which will be bad for MapReduce jobs scan.setCacheBlocks(false); // don't set to true for MR jobs // set other scan attrs - + TableMapReduceUtil.initTableMapperJob( sourceTable, // input table scan, // Scan instance to control CF and attribute selection @@ -1017,17 +1108,17 @@ TableMapReduceUtil.initTableReducerJob( null, // reducer class job); job.setNumReduceTasks(0); - + boolean b = job.waitForCompletion(true); if (!b) { throw new IOException("error with job!"); } - An explanation is required of what TableMapReduceUtil is doing, especially with the reducer. + An explanation is required of what TableMapReduceUtil is doing, especially with the reducer. TableOutputFormat is being used as the outputFormat class, and several parameters are being set on the config (e.g., TableOutputFormat.OUTPUT_TABLE), as well as setting the reducer output key to ImmutableBytesWritable and reducer value to Writable. - These could be set by the programmer on the job and conf, but TableMapReduceUtil tries to make things easier. + These could be set by the programmer on the job and conf, but TableMapReduceUtil tries to make things easier. The following is the example mapper, which will create a Put and matching the input Result and emit it. Note: this is what the CopyTable utility does. @@ -1038,7 +1129,7 @@ public static class MyMapper extends TableMapper<ImmutableBytesWritable, Put& // this example is just copying the data from the source table... context.write(row, resultToPut(row,value)); } - + private static Put resultToPut(ImmutableBytesWritable key, Result result) throws IOException { Put put = new Put(key.get()); for (KeyValue kv : result.raw()) { @@ -1049,9 +1140,9 @@ public static class MyMapper extends TableMapper<ImmutableBytesWritable, Put& } There isn't actually a reducer step, so TableOutputFormat takes care of sending the Put - to the target table. + to the target table. - This is just an example, developers could choose not to use TableOutputFormat and connect to the + This is just an example, developers could choose not to use TableOutputFormat and connect to the target table themselves. @@ -1063,18 +1154,18 @@ public static class MyMapper extends TableMapper<ImmutableBytesWritable, Put&
HBase MapReduce Summary to HBase Example - The following example uses HBase as a MapReduce source and sink with a summarization step. This example will + The following example uses HBase as a MapReduce source and sink with a summarization step. This example will count the number of distinct instances of a value in a table and write those summarized counts in another table. Configuration config = HBaseConfiguration.create(); Job job = new Job(config,"ExampleSummary"); job.setJarByClass(MySummaryJob.class); // class that contains mapper and reducer - + Scan scan = new Scan(); scan.setCaching(500); // 1 is the default in Scan, which will be bad for MapReduce jobs scan.setCacheBlocks(false); // don't set to true for MR jobs // set other scan attrs - + TableMapReduceUtil.initTableMapperJob( sourceTable, // input table scan, // Scan instance to control CF and attribute selection @@ -1087,20 +1178,20 @@ TableMapReduceUtil.initTableReducerJob( MyTableReducer.class, // reducer class job); job.setNumReduceTasks(1); // at least one, adjust as required - + boolean b = job.waitForCompletion(true); if (!b) { throw new IOException("error with job!"); -} +} - In this example mapper a column with a String-value is chosen as the value to summarize upon. + In this example mapper a column with a String-value is chosen as the value to summarize upon. This value is used as the key to emit from the mapper, and an IntWritable represents an instance counter. public static class MyMapper extends TableMapper<Text, IntWritable> { private final IntWritable ONE = new IntWritable(1); private Text text = new Text(); - + public void map(ImmutableBytesWritable row, Result value, Context context) throws IOException, InterruptedException { String val = new String(value.getValue(Bytes.toBytes("cf"), Bytes.toBytes("attr1"))); text.set(val); // we can only emit Writables... @@ -1112,7 +1203,7 @@ public static class MyMapper extends TableMapper<Text, IntWritable> { In the reducer, the "ones" are counted (just like any other MR example that does this), and then emits a Put. public static class MyTableReducer extends TableReducer<Text, IntWritable, ImmutableBytesWritable> { - + public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int i = 0; for (IntWritable val : values) { @@ -1131,17 +1222,17 @@ public static class MyTableReducer extends TableReducer<Text, IntWritable, Im HBase MapReduce Summary to File Example This very similar to the summary example above, with exception that this is using HBase as a MapReduce source but HDFS as the sink. The differences are in the job setup and in the reducer. The mapper remains the same. - + Configuration config = HBaseConfiguration.create(); Job job = new Job(config,"ExampleSummaryToFile"); job.setJarByClass(MySummaryFileJob.class); // class that contains mapper and reducer - + Scan scan = new Scan(); scan.setCaching(500); // 1 is the default in Scan, which will be bad for MapReduce jobs scan.setCacheBlocks(false); // don't set to true for MR jobs // set other scan attrs - + TableMapReduceUtil.initTableMapperJob( sourceTable, // input table scan, // Scan instance to control CF and attribute selection @@ -1152,22 +1243,22 @@ TableMapReduceUtil.initTableMapperJob( job.setReducerClass(MyReducer.class); // reducer class job.setNumReduceTasks(1); // at least one, adjust as required FileOutputFormat.setOutputPath(job, new Path("/tmp/mr/mySummaryFile")); // adjust directories as required - + boolean b = job.waitForCompletion(true); if (!b) { throw new IOException("error with job!"); -} +} - As stated above, the previous Mapper can run unchanged with this example. + As stated above, the previous Mapper can run unchanged with this example. As for the Reducer, it is a "generic" Reducer instead of extending TableMapper and emitting Puts. public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable> { - + public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int i = 0; for (IntWritable val : values) { i += val.get(); - } + } context.write(key, new IntWritable(i)); } } @@ -1176,11 +1267,11 @@ if (!b) {
HBase MapReduce Summary to HBase Without Reducer It is also possible to perform summaries without a reducer - if you use HBase as the reducer. - + An HBase target table would need to exist for the job summary. The HTable method incrementColumnValue - would be used to atomically increment values. From a performance perspective, it might make sense to keep a Map + would be used to atomically increment values. From a performance perspective, it might make sense to keep a Map of values with their values to be incremeneted for each map-task, and make one update per key at during the - cleanup method of the mapper. However, your milage may vary depending on the number of rows to be processed and + cleanup method of the mapper. However, your milage may vary depending on the number of rows to be processed and unique keys. In the end, the summary results are in HBase. @@ -1192,41 +1283,41 @@ if (!b) { to generate summaries directly to an RDBMS via a custom reducer. The setup method can connect to an RDBMS (the connection information can be passed via custom parameters in the context) and the cleanup method can close the connection. - + It is critical to understand that number of reducers for the job affects the summarization implementation, and you'll have to design this into your reducer. Specifically, whether it is designed to run as a singleton (one reducer) or multiple reducers. Neither is right or wrong, it depends on your use-case. Recognize that the more reducers that - are assigned to the job, the more simultaneous connections to the RDBMS will be created - this will scale, but only to a point. + are assigned to the job, the more simultaneous connections to the RDBMS will be created - this will scale, but only to a point. public static class MyRdbmsReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private Connection c = null; - + public void setup(Context context) { // create DB connection... } - + public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { // do summarization // in this example the keys are Text, but this is just an example } - + public void cleanup(Context context) { // close db connection } - + } In the end, the summary results are written to your RDBMS table/s.
- +
Accessing Other HBase Tables in a MapReduce Job Although the framework currently allows one HBase table as input to a - MapReduce job, other HBase tables can + MapReduce job, other HBase tables can be accessed as lookup tables, etc., in a MapReduce job via creating an HTable instance in the setup method of the Mapper. public class MyMapper extends TableMapper<Text, LongWritable> { @@ -1235,12 +1326,12 @@ if (!b) { public void setup(Context context) { myOtherTable = new HTable("myOtherTable"); } - + public void map(ImmutableBytesWritable row, Result value, Context context) throws IOException, InterruptedException { // process Result... // use 'myOtherTable' for lookups } - +
@@ -1253,10 +1344,13 @@ if (!b) { map-tasks which will double-write your data to HBase; this is probably not what you want. + See for more information. + - + + Architecture
@@ -1264,24 +1358,24 @@ if (!b) {
NoSQL? HBase is a type of "NoSQL" database. "NoSQL" is a general term meaning that the database isn't an RDBMS which - supports SQL as it's primary access language, but there are many types of NoSQL databases: BerkeleyDB is an + supports SQL as its primary access language, but there are many types of NoSQL databases: BerkeleyDB is an example of a local NoSQL database, whereas HBase is very much a distributed database. Technically speaking, HBase is really more a "Data Store" than "Data Base" because it lacks many of the features you find in an RDBMS, such as typed columns, secondary indexes, triggers, and advanced query languages, etc. However, HBase has many features which supports both linear and modular scaling. HBase clusters expand - by adding RegionServers that are hosted on commodity class servers. If a cluster expands from 10 to 20 + by adding RegionServers that are hosted on commodity class servers. If a cluster expands from 10 to 20 RegionServers, for example, it doubles both in terms of storage and as well as processing capacity. RDBMS can scale well, but only up to a point - specifically, the size of a single database server - and for the best performance requires specialized hardware and storage devices. HBase features of note are: - Strongly consistent reads/writes: HBase is not an "eventually consistent" DataStore. This + Strongly consistent reads/writes: HBase is not an "eventually consistent" DataStore. This makes it very suitable for tasks such as high-speed counter aggregation. Automatic sharding: HBase tables are distributed on the cluster via regions, and regions are automatically split and re-distributed as your data grows. Automatic RegionServer failover - Hadoop/HDFS Integration: HBase supports HDFS out of the box as it's distributed file system. - MapReduce: HBase supports massively parallelized processing via MapReduce for using HBase as both + Hadoop/HDFS Integration: HBase supports HDFS out of the box as its distributed file system. + MapReduce: HBase supports massively parallelized processing via MapReduce for using HBase as both source and sink. Java Client API: HBase supports an easy to use Java API for programmatic access. Thrift/REST API: HBase also supports Thrift and REST for non-Java front-ends. @@ -1289,12 +1383,12 @@ if (!b) { Operational Management: HBase provides build-in web-pages for operational insight as well as JMX metrics. -
- +
+
When Should I Use HBase? HBase isn't suitable for every problem. - First, make sure you have enough data. If you have hundreds of millions or billions of rows, then + First, make sure you have enough data. If you have hundreds of millions or billions of rows, then HBase is a good candidate. If you only have a few thousand/million rows, then using a traditional RDBMS might be a better choice due to the fact that all of your data might wind up on a single node (or two) and the rest of the cluster may be sitting idle. @@ -1302,7 +1396,7 @@ if (!b) { Second, make sure you can live without all the extra features that an RDBMS provides (e.g., typed columns, secondary indexes, transactions, advanced query languages, etc.) An application built against an RDBMS cannot be "ported" to HBase by simply changing a JDBC driver, for example. Consider moving from an RDBMS to HBase as a - complete redesign as opposed to a port. + complete redesign as opposed to a port. Third, make sure you have enough hardware. Even HDFS doesn't do well with anything less than 5 DataNodes (due to things such as HDFS block replication which has a default of 3), plus a NameNode. @@ -1313,9 +1407,9 @@ if (!b) {
What Is The Difference Between HBase and Hadoop/HDFS? - HDFS is a distributed file system that is well suited for the storage of large files. - It's documentation states that it is not, however, a general purpose file system, and does not provide fast individual record lookups in files. - HBase, on the other hand, is built on top of HDFS and provides fast record lookups (and updates) for large tables. + HDFS is a distributed file system that is well suited for the storage of large files. + It's documentation states that it is not, however, a general purpose file system, and does not provide fast individual record lookups in files. + HBase, on the other hand, is built on top of HDFS and provides fast record lookups (and updates) for large tables. This can sometimes be a point of conceptual confusion. HBase internally puts your data in indexed "StoreFiles" that exist on HDFS for high-speed lookups. See the and the rest of this chapter for more information on how HBase achieves its goals. @@ -1324,19 +1418,19 @@ if (!b) {
Catalog Tables - The catalog tables -ROOT- and .META. exist as HBase tables. They are are filtered out + The catalog tables -ROOT- and .META. exist as HBase tables. They are filtered out of the HBase shell's list command, but they are in fact tables just like any other.
ROOT - -ROOT- keeps track of where the .META. table is. The -ROOT- table structure is as follows: + -ROOT- keeps track of where the .META. table is. The -ROOT- table structure is as follows: - Key: + Key: .META. region key (.META.,,1) - Values: + Values: info:regioninfo (serialized HRegionInfo instance of .META.) @@ -1347,14 +1441,14 @@ if (!b) {
META - The .META. table keeps a list of all regions in the system. The .META. table structure is as follows: + The .META. table keeps a list of all regions in the system. The .META. table structure is as follows: - Key: + Key: Region key of the format ([table],[region start key],[region id]) - Values: + Values: info:regioninfo (serialized HRegionInfo instance for this region) @@ -1363,12 +1457,12 @@ if (!b) { info:serverstartcode (start-time of the RegionServer process containing this region) - When a table is in the process of splitting two other columns will be created, info:splitA and info:splitB + When a table is in the process of splitting two other columns will be created, info:splitA and info:splitB which represent the two daughter regions. The values for these columns are also serialized HRegionInfo instances. After the region has been split eventually this row will be deleted. Notes on HRegionInfo: the empty key is used to denote table start and table end. A region with an empty start key - is the first region in a table. If region has both an empty start and an empty end key, its the only region in the table + is the first region in a table. If region has both an empty start and an empty end key, it's the only region in the table In the (hopefully unlikely) event that programmatic processing of catalog metadata is required, see the Writables utility. @@ -1380,9 +1474,9 @@ if (!b) { For information on region-RegionServer assignment, see . -
+
- +
Client The HBase client @@ -1398,7 +1492,7 @@ if (!b) { need not go through the lookup process. Should a region be reassigned either by the master load balancer or because a RegionServer has died, the client will requery the catalog tables to determine the new - location of the user region. + location of the user region. See for more information about the impact of the Master on HBase Client communication. @@ -1406,10 +1500,11 @@ if (!b) { Administrative functions are handled through HBaseAdmin
Connections - For connection configuration information, see . + For connection configuration information, see . - HTable -instances are not thread-safe. When creating HTable instances, it is advisable to use the same HBaseConfiguration + HTable + instances are not thread-safe. Only one thread use an instance of HTable at any given + time. When creating HTable instances, it is advisable to use the same HBaseConfiguration instance. This will ensure sharing of ZooKeeper and socket instances to the RegionServers which is usually what you want. For example, this is preferred: HBaseConfiguration conf = HBaseConfiguration.create(); @@ -1425,7 +1520,19 @@ HTable table2 = new HTable(conf2, "myTable");
Connection Pooling For applications which require high-end multithreaded access (e.g., web-servers or application servers that may serve many application threads - in a single JVM), see HTablePool. + in a single JVM), one solution is HTablePool. + But as written currently, it is difficult to control client resource consumption when using HTablePool. + + + Another solution is to precreate an HConnection using + // Create a connection to the cluster. +HConnection connection = HConnectionManager.createConnection(Configuration); +HTableInterface table = connection.getTable("myTable"); +// use table as needed, the table returned is lightweight +table.close(); +// use the connection for other access to the cluster +connection.close(); + Constructing HTableInterface implementation is very lightweight and resources are controlled/shared if you go this route.
@@ -1436,9 +1543,9 @@ HTable table2 = new HTable(conf2, "myTable"); is filled. The writebuffer is 2MB by default. Before an HTable instance is discarded, either close() or flushCommits() should be invoked so Puts - will not be lost. - - Note: htable.delete(Delete); does not go in the writebuffer! This only applies to Puts. + will not be lost. + + Note: htable.delete(Delete); does not go in the writebuffer! This only applies to Puts. For additional information on write durability, review the ACID semantics page. @@ -1456,15 +1563,15 @@ HTable table2 = new HTable(conf2, "myTable"); in the client API however they are discouraged because if not managed properly these can lock up the RegionServers. - There is an oustanding ticket HBASE-2332 to + There is an oustanding ticket HBASE-2332 to remove this feature from the client.
- +
Client Request Filters Get and Scan instances can be - optionally configured with filters which are applied on the RegionServer. + optionally configured with filters which are applied on the RegionServer. Filters can be confusing because there are many different types, and it is best to approach them by understanding the groups of Filter functionality. @@ -1473,8 +1580,8 @@ HTable table2 = new HTable(conf2, "myTable"); Structural Filters contain other Filters.
FilterList FilterList - represents a list of Filters with a relationship of FilterList.Operator.MUST_PASS_ALL or - FilterList.Operator.MUST_PASS_ONE between the Filters. The following example shows an 'or' between two + represents a list of Filters with a relationship of FilterList.Operator.MUST_PASS_ALL or + FilterList.Operator.MUST_PASS_ONE between the Filters. The following example shows an 'or' between two Filters (checking for either 'my value' or 'my other value' on the same attribute). FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ONE); @@ -1521,7 +1628,7 @@ scan.setFilter(filter);
RegexStringComparator RegexStringComparator - supports regular expressions for value comparisons. + supports regular expressions for value comparisons. RegexStringComparator comp = new RegexStringComparator("my."); // any value that starts with 'my' SingleColumnValueFilter filter = new SingleColumnValueFilter( @@ -1532,7 +1639,7 @@ SingleColumnValueFilter filter = new SingleColumnValueFilter( ); scan.setFilter(filter); - See the Oracle JavaDoc for supported RegEx patterns in Java. + See the Oracle JavaDoc for supported RegEx patterns in Java.
SubstringComparator @@ -1663,36 +1770,40 @@ rs.close();
RowKey
RowFilter - It is generally a better idea to use the startRow/stopRow methods on Scan for row selection, however + It is generally a better idea to use the startRow/stopRow methods on Scan for row selection, however RowFilter can also be used.
Utility
FirstKeyOnlyFilter - This is primarily used for rowcount jobs. + This is primarily used for rowcount jobs. See FirstKeyOnlyFilter.
- +
Master HMaster is the implementation of the Master Server. The Master server is responsible for monitoring all RegionServer instances in the cluster, and is - the interface for all metadata changes. In a distributed cluster, the Master typically runs on the . + the interface for all metadata changes. In a distributed cluster, the Master typically runs on the + J Mohamed Zahoor goes into some more detail on the Master Architecture in this blog posting, HBase HMaster Architecture + . +
Startup Behavior If run in a multi-Master environment, all Masters compete to run the cluster. If the active - Master loses it's lease in ZooKeeper (or the Master shuts down), then then the remaining Masters jostle to + Master loses its lease in ZooKeeper (or the Master shuts down), then then the remaining Masters jostle to take over the Master role.
Runtime Impact A common dist-list question is what happens to an HBase cluster when the Master goes down. Because the - HBase client talks directly to the RegionServers, the cluster can still function in a "steady + HBase client talks directly to the RegionServers, the cluster can still function in a "steady state." Additionally, per ROOT and META exist as HBase tables (i.e., are - not resident in the Master). However, the Master controls critical functions such as RegionServer failover and - completing region splits. So while the cluster can still run for a time without the Master, - the Master should be restarted as soon as possible. + not resident in the Master). However, the Master controls critical functions such as RegionServer failover and + completing region splits. So while the cluster can still run for a time without the Master, + the Master should be restarted as soon as possible.
Interface @@ -1700,20 +1811,20 @@ rs.close(); Table (createTable, modifyTable, removeTable, enable, disable) - ColumnFamily (addColumn, modifyColumn, removeColumn) + ColumnFamily (addColumn, modifyColumn, removeColumn) Region (move, assign, unassign) - For example, when the HBaseAdmin method disableTable is invoked, it is serviced by the Master server. + For example, when the HBaseAdmin method disableTable is invoked, it is serviced by the Master server.
Processes The Master runs several background threads:
LoadBalancer - Periodically, and when there are not any regions in transition, - a load balancer will run and move regions around to balance cluster load. + Periodically, and when there are no regions in transition, + a load balancer will run and move regions around to balance the cluster's load. See for configuring this property. See for more information on region assignment. @@ -1726,18 +1837,18 @@ rs.close();
RegionServer HRegionServer is the RegionServer implementation. It is responsible for serving and managing regions. - In a distributed cluster, a RegionServer runs on a . + In a distributed cluster, a RegionServer runs on a .
Interface The methods exposed by HRegionRegionInterface contain both data-oriented and region-maintenance methods: Data (get, put, delete, next, etc.) - Region (splitRegion, compactRegion, etc.) + Region (splitRegion, compactRegion, etc.) For example, when the HBaseAdmin method majorCompact is invoked on a table, the client is actually iterating through - all regions for the specified table and requesting a major compaction directly to each region. + all regions for the specified table and requesting a major compaction directly to each region.
Processes @@ -1761,7 +1872,7 @@ rs.close(); posted. Documentation will eventually move to this reference guide, but the blog is the most current information available at this time.
- +
Block Cache
@@ -1849,9 +1960,9 @@ rs.close(); Purpose Each RegionServer adds updates (Puts, Deletes) to its write-ahead log (WAL) - first, and then to the for the affected . - This ensures that HBase has durable writes. Without WAL, there is the possibility of data loss in the case of a RegionServer failure - before each MemStore is flushed and new StoreFiles are written. HLog + first, and then to the for the affected . + This ensures that HBase has durable writes. Without WAL, there is the possibility of data loss in the case of a RegionServer failure + before each MemStore is flushed and new StoreFiles are written. HLog is the HBase WAL implementation, and there is one HLog instance per RegionServer. The WAL is in HDFS in /hbase/.logs/ with subdirectories per region. @@ -1875,11 +1986,11 @@ rs.close();
<varname>hbase.hlog.split.skip.errors</varname> - When set to true, the default, any error + When set to true, any error encountered splitting will be logged, the problematic WAL will be moved into the .corrupt directory under the hbase rootdir, and processing will continue. If set to - false, the exception will be propagated and the + false, the default, the exception will be propagated and the split logged as failed. See HBASE-2958 @@ -1912,10 +2023,10 @@ rs.close();
Regions Regions are the basic element of availability and - distribution for tables, and are comprised of a Store per Column Family. The heirarchy of objects + distribution for tables, and are comprised of a Store per Column Family. The heirarchy of objects is as follows: -Table (HBase table) +Table (HBase table) Region (Regions for the table) Store (Store per ColumnFamily for each Region for the table) MemStore (MemStore for each Store for each Region for the table) @@ -1924,7 +2035,7 @@ rs.close(); For a description of what HBase files look like when written to HDFS, see . - +
Region Size @@ -1936,13 +2047,13 @@ rs.close(); HBase scales by having regions across many servers. Thus if you have 2 regions for 16GB data, on a 20 node machine your data will be concentrated on just a few machines - nearly the entire - cluster will be idle. This really cant be stressed enough, since a - common problem is loading 200MB data into HBase then wondering why + cluster will be idle. This really cant be stressed enough, since a + common problem is loading 200MB data into HBase then wondering why your awesome 10 node cluster isn't doing anything. - On the other hand, high region count has been known to make things slow. + On the other hand, high region count has been known to make things slow. This is getting better with each release of HBase, but it is probably better to have 700 regions than 3000 for the same amount of data. @@ -1953,7 +2064,7 @@ rs.close(); - When starting off, its probably best to stick to the default region-size, perhaps going + When starting off, it's probably best to stick to the default region-size, perhaps going smaller for hot tables (or manually split hot regions to spread the load over the cluster), or go with larger region sizes if your cell sizes tend to be largish (100k and up). @@ -1977,10 +2088,10 @@ rs.close(); If the region assignment is still valid (i.e., if the RegionServer is still online) then the assignment is kept. - If the assignment is invalid, then the LoadBalancerFactory is invoked to assign the + If the assignment is invalid, then the LoadBalancerFactory is invoked to assign the region. The DefaultLoadBalancer will randomly assign the region to a RegionServer. - META is updated with the RegionServer assignment (if needed) and the RegionServer start codes + META is updated with the RegionServer assignment (if needed) and the RegionServer start codes (start time of the RegionServer process) upon region opening by the RegionServer. @@ -1996,7 +2107,7 @@ rs.close(); The Master will detect that the RegionServer has failed. The region assignments will be considered invalid and will be re-assigned just - like the startup sequence. + like the startup sequence. @@ -2023,14 +2134,14 @@ rs.close(); Third replica is written to a node in another rack (if sufficient nodes) - Thus, HBase eventually achieves locality for a region after a flush or a compaction. + Thus, HBase eventually achieves locality for a region after a flush or a compaction. In a RegionServer failover situation a RegionServer may be assigned regions with non-local StoreFiles (because none of the replicas are local), however as new data is written in the region, or the table is compacted and StoreFiles are re-written, they will become "local" - to the RegionServer. + to the RegionServer. For more information, see HDFS Design on Replica Placement - and also Lars George's blog on HBase and HDFS locality. + and also Lars George's blog on HBase and HDFS locality.
@@ -2048,7 +2159,7 @@ rs.close(); The default split policy can be overwritten using a custom RegionSplitPolicy (HBase 0.94+). Typically a custom split policy should extend HBase's default split policy: ConstantSizeRegionSplitPolicy. - The policy can set globally through the HBaseConfiguration used or on a per table basis: + The policy can set globally through the HBaseConfiguration used or on a per table basis: HTableDescriptor myHtd = ...; myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName()); @@ -2064,8 +2175,8 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName(
MemStore The MemStore holds in-memory modifications to the Store. Modifications are KeyValues. - When asked to flush, current memstore is moved to snapshot and is cleared. - HBase continues to serve edits out of new memstore and backing snapshot until flusher reports in that the + When asked to flush, current memstore is moved to snapshot and is cleared. + HBase continues to serve edits out of new memstore and backing snapshot until flusher reports in that the flush succeeded. At this point the snapshot is let go.
@@ -2076,7 +2187,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( The hfile file format is based on the SSTable file described in the BigTable [2006] paper and on Hadoop's tfile - (The unit test suite and the compression harness were taken directly from tfile). + (The unit test suite and the compression harness were taken directly from tfile). Schubert Zhang's blog post on HFile: A Block-Indexed File Format to Store Sorted Key-Value Pairs makes for a thorough introduction to HBase's hfile. Matteo Bertozzi has also put up a helpful description, HBase I/O: HFile. @@ -2103,7 +2214,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName(
- +
Blocks StoreFiles are composed of blocks. The blocksize is configured on a per-ColumnFamily basis. @@ -2116,7 +2227,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName(
KeyValue The KeyValue class is the heart of data storage in HBase. KeyValue wraps a byte array and takes offsets and lengths into passed array - at where to start interpreting the content as KeyValue. + at where to start interpreting the content as KeyValue. The KeyValue format inside a byte array is: @@ -2180,7 +2291,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( Compaction There are two types of compactions: minor and major. Minor compactions will usually pick up a couple of the smaller adjacent StoreFiles and rewrite them as one. Minors do not drop deletes or expired cells, only major compactions do this. Sometimes a minor compaction - will pick up all the StoreFiles in the Store and in this case it actually promotes itself to being a major compaction. + will pick up all the StoreFiles in the Store and in this case it actually promotes itself to being a major compaction. After a major compaction runs there will be a single StoreFile per Store, and this will help performance usually. Caution: major compactions rewrite all of the Stores data and on a loaded system, this may not be tenable; major compactions will usually have to be done manually on large systems. See . @@ -2189,7 +2300,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName(
Compaction File Selection - To understand the core algorithm for StoreFile selection, there is some ASCII-art in the Store source code that + To understand the core algorithm for StoreFile selection, there is some ASCII-art in the Store source code that will serve as useful reference. It has been copied below: /* normal skew: @@ -2211,16 +2322,16 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( hbase.hstore.compaction.min (.90 hbase.hstore.compactionThreshold) (files) Minimum number of StoreFiles per Store to be selected for a compaction to occur (default 2). hbase.hstore.compaction.max (files) Maximum number of StoreFiles to compact per minor compaction (default 10). - hbase.hstore.compaction.min.size (bytes) - Any StoreFile smaller than this setting with automatically be a candidate for compaction. Defaults to + hbase.hstore.compaction.min.size (bytes) + Any StoreFile smaller than this setting with automatically be a candidate for compaction. Defaults to hbase.hregion.memstore.flush.size (128 mb). - hbase.hstore.compaction.max.size (.92) (bytes) + hbase.hstore.compaction.max.size (.92) (bytes) Any StoreFile larger than this setting with automatically be excluded from compaction (default Long.MAX_VALUE). The minor compaction StoreFile selection logic is size based, and selects a file for compaction when the file <= sum(smaller_files) * hbase.hstore.compaction.ratio. - +
Minor Compaction File Selection - Example #1 (Basic Example) @@ -2228,21 +2339,21 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( hbase.store.compaction.ratio = 1.0f hbase.hstore.compaction.min = 3 (files) > - hbase.hstore.compaction.max = 5 (files) > + hbase.hstore.compaction.max = 5 (files) > hbase.hstore.compaction.min.size = 10 (bytes) > hbase.hstore.compaction.max.size = 1000 (bytes) > The following StoreFiles exist: 100, 50, 23, 12, and 12 bytes apiece (oldest to newest). With the above parameters, the files that would be selected for minor compaction are 23, 12, and 12. - + Why? 100 --> No, because sum(50, 23, 12, 12) * 1.0 = 97. 50 --> No, because sum(23, 12, 12) * 1.0 = 47. 23 --> Yes, because sum(12, 12) * 1.0 = 24. - 12 --> Yes, because the previous file has been included, and because this + 12 --> Yes, because the previous file has been included, and because this does not exceed the the max-file limit of 5 - 12 --> Yes, because the previous file had been included, and because this + 12 --> Yes, because the previous file had been included, and because this does not exceed the the max-file limit of 5. @@ -2253,19 +2364,19 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( hbase.store.compaction.ratio = 1.0f hbase.hstore.compaction.min = 3 (files) > - hbase.hstore.compaction.max = 5 (files) > + hbase.hstore.compaction.max = 5 (files) > hbase.hstore.compaction.min.size = 10 (bytes) > hbase.hstore.compaction.max.size = 1000 (bytes) > - + The following StoreFiles exist: 100, 25, 12, and 12 bytes apiece (oldest to newest). - With the above parameters, the files that would be selected for minor compaction are 23, 12, and 12. - + With the above parameters, no compaction will be started. + Why? 100 --> No, because sum(25, 12, 12) * 1.0 = 47 25 --> No, because sum(12, 12) * 1.0 = 24 - 12 --> No. Candidate because sum(12) * 1.0 = 12, there are only 2 files to compact and that is less than the threshold of 3 + 12 --> No. Candidate because sum(12) * 1.0 = 12, there are only 2 files to compact and that is less than the threshold of 3 12 --> No. Candidate because the previous StoreFile was, but there are not enough files to compact @@ -2276,13 +2387,13 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( hbase.store.compaction.ratio = 1.0f hbase.hstore.compaction.min = 3 (files) > - hbase.hstore.compaction.max = 5 (files) > + hbase.hstore.compaction.max = 5 (files) > hbase.hstore.compaction.min.size = 10 (bytes) > hbase.hstore.compaction.max.size = 1000 (bytes) > The following StoreFiles exist: 7, 6, 5, 4, 3, 2, and 1 bytes apiece (oldest to newest). - With the above parameters, the files that would be selected for minor compaction are 7, 6, 5, 4, 3. - + With the above parameters, the files that would be selected for minor compaction are 7, 6, 5, 4, 3. + Why? 7 --> Yes, because sum(6, 5, 4, 3, 2, 1) * 1.0 = 21. Also, 7 is less than the min-size @@ -2303,74 +2414,126 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( hbase.hstore.compaction.min.size. Because this limit represents the "automatic include" limit for all StoreFiles smaller than this value, this value may need to be adjusted downwards in write-heavy environments where many 1 or 2 mb StoreFiles are being flushed, because every file - will be targeted for compaction and the resulting files may still be under the min-size and require further compaction, etc. + will be targeted for compaction and the resulting files may still be under the min-size and require further compaction, etc.
- -
- Bloom Filters - Bloom filters were developed over in HBase-1200 - Add bloomfilters. - For description of the development process -- why static blooms - rather than dynamic -- and for an overview of the unique properties - that pertain to blooms in HBase, as well as possible future - directions, see the Development Process section - of the document BloomFilters - in HBase attached to HBase-1200. - - The bloom filters described here are actually version two of - blooms in HBase. In versions up to 0.19.x, HBase had a dynamic bloom - option based on work done by the European Commission One-Lab - Project 034819. The core of the HBase bloom work was later - pulled up into Hadoop to implement org.apache.hadoop.io.BloomMapFile. - Version 1 of HBase blooms never worked that well. Version 2 is a - rewrite from scratch though again it starts with the one-lab - work. - - See also and . - - -
- Bloom StoreFile footprint - Bloom filters add an entry to the StoreFile - general FileInfo data structure and then two - extra entries to the StoreFile metadata - section. - -
- BloomFilter in the <classname>StoreFile</classname> - <classname>FileInfo</classname> data structure +
- FileInfo has a - BLOOM_FILTER_TYPE entry which is set to - NONE, ROW or - ROWCOL. +
Bulk Loading +
Overview + + HBase includes several methods of loading data into tables. + The most straightforward method is to either use the TableOutputFormat + class from a MapReduce job, or use the normal client APIs; however, + these are not always the most efficient methods. + + + The bulk load feature uses a MapReduce job to output table data in HBase's internal + data format, and then directly loads the generated StoreFiles into a running + cluster. Using bulk load will use less CPU and network resources than + simply using the HBase API. + +
+
Bulk Load Architecture + + The HBase bulk load process consists of two main steps. + +
Preparing data via a MapReduce job + + The first step of a bulk load is to generate HBase data files (StoreFiles) from + a MapReduce job using HFileOutputFormat. This output format writes + out data in HBase's internal storage format so that they can be + later loaded very efficiently into the cluster. + + + In order to function efficiently, HFileOutputFormat must be + configured such that each output HFile fits within a single region. + In order to do this, jobs whose output will be bulk loaded into HBase + use Hadoop's TotalOrderPartitioner class to partition the map output + into disjoint ranges of the key space, corresponding to the key + ranges of the regions in the table. + + + HFileOutputFormat includes a convenience function, + configureIncrementalLoad(), which automatically sets up + a TotalOrderPartitioner based on the current region boundaries of a + table. +
- -
- BloomFilter entries in <classname>StoreFile</classname> - metadata - - BLOOM_FILTER_META holds Bloom Size, Hash - Function used, etc. Its small in size and is cached on - StoreFile.Reader load - BLOOM_FILTER_DATA is the actual bloomfilter - data. Obtained on-demand. Stored in the LRU cache, if it is enabled - (Its enabled by default). +
Completing the data load + + After the data has been prepared using + HFileOutputFormat, it is loaded into the cluster using + completebulkload. This command line tool iterates + through the prepared data files, and for each one determines the + region the file belongs to. It then contacts the appropriate Region + Server which adopts the HFile, moving it into its storage directory + and making the data available to clients. + + + If the region boundaries have changed during the course of bulk load + preparation, or between the preparation and completion steps, the + completebulkloads utility will automatically split the + data files into pieces corresponding to the new boundaries. This + process is not optimally efficient, so users should take care to + minimize the delay between preparing a bulk load and importing it + into the cluster, especially if other clients are simultaneously + loading data through other means. +
-
-
-
- +
Importing the prepared data using the completebulkload tool + + After a data import has been prepared, either by using the + importtsv tool with the + "importtsv.bulk.output" option or by some other MapReduce + job using the HFileOutputFormat, the + completebulkload tool is used to import the data into the + running cluster. + + + The completebulkload tool simply takes the output path + where importtsv or your MapReduce job put its results, and + the table name to import into. For example: + + $ hadoop jar hbase-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] /user/todd/myoutput mytable + + The -c config-file option can be used to specify a file + containing the appropriate hbase parameters (e.g., hbase-site.xml) if + not supplied already on the CLASSPATH (In addition, the CLASSPATH must + contain the directory that has the zookeeper configuration file if + zookeeper is NOT managed by HBase). + + + Note: If the target table does not already exist in HBase, this + tool will create the table automatically. + + This tool will run quickly, after which point the new data will be visible in + the cluster. + +
+
See Also + For more information about the referenced utilities, see and . + +
+
Advanced Usage + + Although the importtsv tool is useful in many cases, advanced users may + want to generate data programatically, or import data from other formats. To get + started doing so, dig into ImportTsv.java and check the JavaDoc for + HFileOutputFormat. + + + The import step of the bulk load can also be done programatically. See the + LoadIncrementalHFiles class for more information. + +
+
+
HDFS As HBase runs on HDFS (and each StoreFile is written as a file on HDFS), it is important to have an understanding of the HDFS Architecture @@ -2389,15 +2552,18 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( for more information.
-
- +
+ - + + + + FAQ @@ -2427,6 +2593,21 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( + + How can I find examples of NoSQL/HBase? + + See the link to the BigTable paper in in the appendix, as + well as the other papers. + + + + + What is the history of HBase? + + See . + + + Architecture @@ -2541,7 +2722,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( - EC2 issues are a special case. See Troubleshooting and Performance sections. + EC2 issues are a special case. See Troubleshooting and Performance sections. @@ -2581,6 +2762,214 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( + + hbck In Depth + HBaseFsck (hbck) is a tool for checking for region consistency and table integrity problems +and repairing a corrupted HBase. It works in two basic modes -- a read-only inconsistency +identifying mode and a multi-phase read-write repair mode. + +
+ Running hbck to identify inconsistencies +To check to see if your HBase cluster has corruptions, run hbck against your HBase cluster: + +$ ./bin/hbase hbck + + +At the end of the commands output it prints OK or tells you the number of INCONSISTENCIES +present. You may also want to run run hbck a few times because some inconsistencies can be +transient (e.g. cluster is starting up or a region is splitting). Operationally you may want to run +hbck regularly and setup alert (e.g. via nagios) if it repeatedly reports inconsistencies . +A run of hbck will report a list of inconsistencies along with a brief description of the regions and +tables affected. The using the -details option will report more details including a representative +listing of all the splits present in all the tables. + + +$ ./bin/hbase hbck -details + +If you just want to know if some tables are corrupted, you can limit hbck to identify inconsistencies +in only specific tables. For example the following command would only attempt to check table +TableFoo and TableBar. The benefit is that hbck will run in less time. + +$ ./bin/hbase hbck TableFoo TableBar + +
+
Inconsistencies + + If after several runs, inconsistencies continue to be reported, you may have encountered a +corruption. These should be rare, but in the event they occur newer versions of HBase include +the hbck tool enabled with automatic repair options. + + + There are two invariants that when violated create inconsistencies in HBase: + + + HBase’s region consistency invariant is satisfied if every region is assigned and +deployed on exactly one region server, and all places where this state kept is in +accordance. + + HBase’s table integrity invariant is satisfied if for each table, every possible row key +resolves to exactly one region. + + + +Repairs generally work in three phases -- a read-only information gathering phase that identifies +inconsistencies, a table integrity repair phase that restores the table integrity invariant, and then +finally a region consistency repair phase that restores the region consistency invariant. +Starting from version 0.90.0, hbck could detect region consistency problems report on a subset +of possible table integrity problems. It also included the ability to automatically fix the most +common inconsistency, region assignment and deployment consistency problems. This repair +could be done by using the -fix command line option. These problems close regions if they are +open on the wrong server or on multiple region servers and also assigns regions to region +servers if they are not open. + + +Starting from HBase versions 0.90.7, 0.92.2 and 0.94.0, several new command line options are +introduced to aid repairing a corrupted HBase. This hbck sometimes goes by the nickname +“uberhbck”. Each particular version of uber hbck is compatible with the HBase’s of the same +major version (0.90.7 uberhbck can repair a 0.90.4). However, versions <=0.90.6 and versions +<=0.92.1 may require restarting the master or failing over to a backup master. + +
+
Localized repairs + + When repairing a corrupted HBase, it is best to repair the lowest risk inconsistencies first. +These are generally region consistency repairs -- localized single region repairs, that only modify +in-memory data, ephemeral zookeeper data, or patch holes in the META table. +Region consistency requires that the HBase instance has the state of the region’s data in HDFS +(.regioninfo files), the region’s row in the .META. table., and region’s deployment/assignments on +region servers and the master in accordance. Options for repairing region consistency include: + + -fixAssignments (equivalent to the 0.90 -fix option) repairs unassigned, incorrectly +assigned or multiply assigned regions. + + -fixMeta which removes meta rows when corresponding regions are not present in +HDFS and adds new meta rows if they regions are present in HDFS while not in META. + + + To fix deployment and assignment problems you can run this command: + + +$ ./bin/hbase hbck -fixAssignments + +To fix deployment and assignment problems as well as repairing incorrect meta rows you can +run this command:. + +$ ./bin/hbase hbck -fixAssignments -fixMeta + +There are a few classes of table integrity problems that are low risk repairs. The first two are +degenerate (startkey == endkey) regions and backwards regions (startkey > endkey). These are +automatically handled by sidelining the data to a temporary directory (/hbck/xxxx). +The third low-risk class is hdfs region holes. This can be repaired by using the: + + -fixHdfsHoles option for fabricating new empty regions on the file system. +If holes are detected you can use -fixHdfsHoles and should include -fixMeta and -fixAssignments to make the new region consistent. + + + +$ ./bin/hbase hbck -fixAssignments -fixMeta -fixHdfsHoles + +Since this is a common operation, we’ve added a the -repairHoles flag that is equivalent to the +previous command: + +$ ./bin/hbase hbck -repairHoles + +If inconsistencies still remain after these steps, you most likely have table integrity problems +related to orphaned or overlapping regions. +
+
Region Overlap Repairs +Table integrity problems can require repairs that deal with overlaps. This is a riskier operation +because it requires modifications to the file system, requires some decision making, and may +require some manual steps. For these repairs it is best to analyze the output of a hbck -details +run so that you isolate repairs attempts only upon problems the checks identify. Because this is +riskier, there are safeguard that should be used to limit the scope of the repairs. +WARNING: This is a relatively new and have only been tested on online but idle HBase instances +(no reads/writes). Use at your own risk in an active production environment! +The options for repairing table integrity violations include: + + -fixHdfsOrphans option for “adopting” a region directory that is missing a region +metadata file (the .regioninfo file). + + -fixHdfsOverlaps ability for fixing overlapping regions + + +When repairing overlapping regions, a region’s data can be modified on the file system in two +ways: 1) by merging regions into a larger region or 2) by sidelining regions by moving data to +“sideline” directory where data could be restored later. Merging a large number of regions is +technically correct but could result in an extremely large region that requires series of costly +compactions and splitting operations. In these cases, it is probably better to sideline the regions +that overlap with the most other regions (likely the largest ranges) so that merges can happen on +a more reasonable scale. Since these sidelined regions are already laid out in HBase’s native +directory and HFile format, they can be restored by using HBase’s bulk load mechanism. +The default safeguard thresholds are conservative. These options let you override the default +thresholds and to enable the large region sidelining feature. + + -maxMerge <n> maximum number of overlapping regions to merge + + -sidelineBigOverlaps if more than maxMerge regions are overlapping, sideline attempt +to sideline the regions overlapping with the most other regions. + + -maxOverlapsToSideline <n> if sidelining large overlapping regions, sideline at most n +regions. + + + +Since often times you would just want to get the tables repaired, you can use this option to turn +on all repair options: + + -repair includes all the region consistency options and only the hole repairing table +integrity options. + + +Finally, there are safeguards to limit repairs to only specific tables. For example the following +command would only attempt to check and repair table TableFoo and TableBar. + +$ ./bin/hbase hbck -repair TableFoo TableBar + +
Special cases: Meta is not properly assigned +There are a few special cases that hbck can handle as well. +Sometimes the meta table’s only region is inconsistently assigned or deployed. In this case +there is a special -fixMetaOnly option that can try to fix meta assignments. + +$ ./bin/hbase hbck -fixMetaOnly -fixAssignments + +
+
Special cases: HBase version file is missing +HBase’s data on the file system requires a version file in order to start. If this flie is missing, you +can use the -fixVersionFile option to fabricating a new HBase version file. This assumes that +the version of hbck you are running is the appropriate version for the HBase cluster. +
+
Special case: Root and META are corrupt. +The most drastic corruption scenario is the case where the ROOT or META is corrupted and +HBase will not start. In this case you can use the OfflineMetaRepair tool create new ROOT +and META regions and tables. +This tool assumes that HBase is offline. It then marches through the existing HBase home +directory, loads as much information from region metadata files (.regioninfo files) as possible +from the file system. If the region metadata has proper table integrity, it sidelines the original root +and meta table directories, and builds new ones with pointers to the region directories and their +data. + +$ ./bin/hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair + +NOTE: This tool is not as clever as uberhbck but can be used to bootstrap repairs that uberhbck +can complete. +If the tool succeeds you should be able to start hbase and run online repairs if necessary. +
+
Special cases: Offline split parent + +Once a region is split, the offline parent will be cleaned up automatically. Sometimes, daughter regions +are split again before their parents are cleaned up. HBase can clean up parents in the right order. However, +there could be some lingering offline split parents sometimes. They are in META, in HDFS, and not deployed. +But HBase can't clean them up. In this case, you can use the -fixSplitParents option to reset +them in META to be online and not split. Therefore, hbck can merge them with other regions if fixing +overlapping regions option is used. + + +This option should not normally be used, and it is not in -fixAll. + +
+
+ + Compression In HBase<indexterm><primary>Compression</primary></indexterm> @@ -2589,9 +2978,15 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( CompressionTest Tool HBase includes a tool to test compression is set up properly. - To run it, type /bin/hbase org.apache.hadoop.hbase.util.CompressionTest. + To run it, type /bin/hbase org.apache.hadoop.hbase.util.CompressionTest. This will emit usage on how to run the tool. + You need to restart regionserver for it to pick up fixed codecs! + Be aware that the regionserver caches the result of the compression check it runs + ahead of each region open. This means + that you will have to restart the regionserver for it to notice that you have fixed + any codec issues. +
@@ -2607,7 +3002,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( hbase.regionserver.codecs to your hbase-site.xml with a value of - codecs to test on startup. For example if the + codecs to test on startup. For example if the hbase.regionserver.codecs value is lzo,gz and if lzo is not present @@ -2668,7 +3063,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( Build and install snappy on all nodes - of your cluster. + of your cluster (see below) @@ -2689,15 +3084,54 @@ hbase> describe 't1' -
+
+ + Installation + + + You will find the snappy library file under the .libs directory from your Snappy build (For example + /home/hbase/snappy-1.0.5/.libs/). The file is called libsnappy.so.1.x.x where 1.x.x is the version of the snappy + code you are building. You can either copy this file into your hbase directory under libsnappy.so name, or simply + create a symbolic link to it. + + + + The second file you need is the hadoop native library. You will find this file in your hadoop installation directory + under lib/native/Linux-amd64-64/ or lib/native/Linux-i386-32/. The file you are looking for is libhadoop.so.1.x.x. + Again, you can simply copy this file or link to it, under the name libhadoop.so. + + + At the end of the installation, you should have both libsnappy.so and libhadoop.so links or files present into + lib/native/Linux-amd64-64 or into lib/native/Linux-i386-32 + + To point hbase at snappy support, in hbase-env.sh set + export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64 + In /pathtoyourhadoop/lib/native/Linux-amd64-64 you should have something like: + + libsnappy.a + libsnappy.so + libsnappy.so.1 + libsnappy.so.1.1.2 + + +
+
+
+ Changing Compression Schemes + A frequent question on the dist-list is how to change compression schemes for ColumnFamilies. This is actually quite simple, + and can be done via an alter command. Because the compression scheme is encoded at the block-level in StoreFiles, the table does + not need to be re-created and the data does not copied somewhere else. Just make sure + the old codec is still available until you are sure that all of the old StoreFiles have been compacted. + +
<link xlink:href="https://github.com/brianfrankcooper/YCSB/">YCSB: The Yahoo! Cloud Serving Benchmark</link> and HBase TODO: Describe how YCSB is poor for putting up a decent cluster load. TODO: Describe setup of YCSB for HBase - Ted Dunning redid YCSB so its mavenized and added facility for verifying workloads. See Ted Dunning's YCSB. + Ted Dunning redid YCSB so it's mavenized and added facility for verifying workloads. See Ted Dunning's YCSB. @@ -2719,7 +3153,7 @@ hbase> describe 't1' HFile Version 1 - HFile Version 1 + HFile Version 1 @@ -2762,7 +3196,7 @@ hbase> describe 't1' HFile Version 2 - HFile Version 2 + HFile Version 2 @@ -2791,7 +3225,7 @@ hbase> describe 't1' - META – meta blocks (not used for Bloom filters in version 2 anymore) + META – meta blocks (not used for Bloom filters in version 2 anymore) @@ -2816,7 +3250,7 @@ hbase> describe 't1' - TRAILER – a fixed>size file trailer. As opposed to the above, this is not an + TRAILER – a fixed>size file trailer. As opposed to the above, this is not an HFile v2 block but a fixed>size (for each HFile version) data structure @@ -2831,7 +3265,7 @@ hbase> describe 't1' Compressed size of the block's data, not including the header (int). -Can be used for skipping the current data block when scanning HFile data. +Can be used for skipping the current data block when scanning HFile data. @@ -2961,12 +3395,12 @@ This offset may point to a data block or to a deeper>level index block. -Offset of the block referenced by this entry in the file (long) +Offset of the block referenced by this entry in the file (long) -On>disk size of the referenced block (int) +On>disk size of the referenced block (int) @@ -3207,6 +3641,8 @@ Comparator class used for Bloom filter keys, a UTF>8 encoded string stored usi HBase Wiki has a page with a number of presentations. + HBase RefCard from DZone. +
HBase Books HBase: The Definitive Guide by Lars George. @@ -3216,16 +3652,29 @@ Comparator class used for Bloom filter keys, a UTF>8 encoded string stored usi Hadoop: The Definitive Guide by Tom White.
- + + + + HBase History + + 2006: BigTable paper published by Google. + + 2006 (end of year): HBase development starts. + + 2008: HBase becomes Hadoop sub-project. + + 2010: HBase becomes Apache top-level project. + + HBase and the Apache Software Foundation HBase is a project in the Apache Software Foundation and as such there are responsibilities to the ASF to ensure a healthy project.
ASF Development Process - See the Apache Development Process page + See the Apache Development Process page for all sorts of information on how the ASF is structured (e.g., PMC, committers, contributors), to tips on contributing - and getting involved, and how open-source works at ASF. + and getting involved, and how open-source works at ASF.
ASF Board Reporting @@ -3235,6 +3684,67 @@ Comparator class used for Bloom filter keys, a UTF>8 encoded string stored usi
+ Enabling Dapper-like Tracing in HBase +HBASE-6449 added support +for tracing requests through HBase, using the open source tracing library, +HTrace. Setting up tracing is quite simple, +however it currently requires some very minor changes to your client code (it would not be very difficult to remove this requirement). + +
SpanReceivers +The tracing system works by collecting information in structs called ‘Spans’. +It is up to you to choose how you want to receive this information by implementing the +SpanReceiver interface, which defines one method: +public void receiveSpan(Span span); +This method serves as a callback whenever a span is completed. HTrace allows you to use +as many SpanReceivers as you want so you can easily send trace information to multiple destinations. + + +Configure what SpanReceivers you’d like to use by putting a comma separated list of the +fully-qualified class name of classes implementing SpanReceiver in +hbase-site.xml property: hbase.trace.spanreceiver.classes. + + +HBase includes a HBaseLocalFileSpanReceiver that writes all span +information to local files in a JSON-based format. The HBaseLocalFileSpanReceiver +looks in hbase-site.xml for a hbase.trace.spanreceiver.localfilespanreceiver.filename +property with a value describing the name of the file to which nodes should write their span information. + + +If you do not want to use the included HBaseLocalFileSpanReceiver, +you are encouraged to write your own receiver (take a look at HBaseLocalFileSpanReceiver +for an example). If you think others would benefit from your receiver, file a JIRA or send a pull request to +HTrace. + +
+
+Client Modifications +Currently, you must turn on tracing in your client code. To do this, you simply turn on tracing for +requests you think are interesting, and turn it off when the request is done. + + +For example, if you wanted to trace all of your get operations, you change this: +HTable table = new HTable(...); +Get get = new Get(...); + +into: + +Span getSpan = Trace.startSpan(“doing get”, Sampler.ALWAYS); +try { + HTable table = new HTable(...); + Get get = new Get(...); +... +} finally { + getSpan.stop(); +} + +If you wanted to trace half of your ‘get’ operations, you would pass in: +new ProbabilitySampler(0.5) in lieu of Sampler.ALWAYS to Trace.startSpan(). +See the HTrace README for more information on Samplers. + +
+ + + Index diff --git a/src/docbkx/case_studies.xml b/src/docbkx/case_studies.xml new file mode 100644 index 000000000000..2e3bba0432f8 --- /dev/null +++ b/src/docbkx/case_studies.xml @@ -0,0 +1,324 @@ + + + + Apache HBase (TM) Case Studies +
+ Overview + This chapter will describe a variety of performance and troubleshooting case studies that can + provide a useful blueprint on diagnosing Apache HBase (TM) cluster issues. + For more information on Performance and Troubleshooting, see and . + +
+ +
+ Schema Design + +
+ List Data + The following is an exchange from the user dist-list regarding a fairly common question: + how to handle per-user list data in Apache HBase. + + *** QUESTION *** + + We're looking at how to store a large amount of (per-user) list data in +HBase, and we were trying to figure out what kind of access pattern made +the most sense. One option is store the majority of the data in a key, so +we could have something like: + + + +<FixedWidthUserName><FixedWidthValueId1>:"" (no value) +<FixedWidthUserName><FixedWidthValueId2>:"" (no value) +<FixedWidthUserName><FixedWidthValueId3>:"" (no value) + + +The other option we had was to do this entirely using: + +<FixedWidthUserName><FixedWidthPageNum0>:<FixedWidthLength><FixedIdNextPageNum><ValueId1><ValueId2><ValueId3>... +<FixedWidthUserName><FixedWidthPageNum1>:<FixedWidthLength><FixedIdNextPageNum><ValueId1><ValueId2><ValueId3>... + + +where each row would contain multiple values. +So in one case reading the first thirty values would be: + + +scan { STARTROW => 'FixedWidthUsername' LIMIT => 30} + +And in the second case it would be + +get 'FixedWidthUserName\x00\x00\x00\x00' + + +The general usage pattern would be to read only the first 30 values of +these lists, with infrequent access reading deeper into the lists. Some +users would have <= 30 total values in these lists, and some users would +have millions (i.e. power-law distribution) + + + The single-value format seems like it would take up more space on HBase, +but would offer some improved retrieval / pagination flexibility. Would +there be any significant performance advantages to be able to paginate via +gets vs paginating with scans? + + + My initial understanding was that doing a scan should be faster if our +paging size is unknown (and caching is set appropriately), but that gets +should be faster if we'll always need the same page size. I've ended up +hearing different people tell me opposite things about performance. I +assume the page sizes would be relatively consistent, so for most use cases +we could guarantee that we only wanted one page of data in the +fixed-page-length case. I would also assume that we would have infrequent +updates, but may have inserts into the middle of these lists (meaning we'd +need to update all subsequent rows). + + +Thanks for help / suggestions / follow-up questions. + + *** ANSWER *** + +If I understand you correctly, you're ultimately trying to store +triples in the form "user, valueid, value", right? E.g., something +like: + + +"user123, firstname, Paul", +"user234, lastname, Smith" + + +(But the usernames are fixed width, and the valueids are fixed width). + + +And, your access pattern is along the lines of: "for user X, list the +next 30 values, starting with valueid Y". Is that right? And these +values should be returned sorted by valueid? + + +The tl;dr version is that you should probably go with one row per +user+value, and not build a complicated intra-row pagination scheme on +your own unless you're really sure it is needed. + + +Your two options mirror a common question people have when designing +HBase schemas: should I go "tall" or "wide"? Your first schema is +"tall": each row represents one value for one user, and so there are +many rows in the table for each user; the row key is user + valueid, +and there would be (presumably) a single column qualifier that means +"the value". This is great if you want to scan over rows in sorted +order by row key (thus my question above, about whether these ids are +sorted correctly). You can start a scan at any user+valueid, read the +next 30, and be done. What you're giving up is the ability to have +transactional guarantees around all the rows for one user, but it +doesn't sound like you need that. Doing it this way is generally +recommended (see +here http://hbase.apache.org/book.html#schema.smackdown). + + +Your second option is "wide": you store a bunch of values in one row, +using different qualifiers (where the qualifier is the valueid). The +simple way to do that would be to just store ALL values for one user +in a single row. I'm guessing you jumped to the "paginated" version +because you're assuming that storing millions of columns in a single +row would be bad for performance, which may or may not be true; as +long as you're not trying to do too much in a single request, or do +things like scanning over and returning all of the cells in the row, +it shouldn't be fundamentally worse. The client has methods that allow +you to get specific slices of columns. + + +Note that neither case fundamentally uses more disk space than the +other; you're just "shifting" part of the identifying information for +a value either to the left (into the row key, in option one) or to the +right (into the column qualifiers in option 2). Under the covers, +every key/value still stores the whole row key, and column family +name. (If this is a bit confusing, take an hour and watch Lars +George's excellent video about understanding HBase schema design: +http://www.youtube.com/watch?v=_HLoH_PgrLk). + + +A manually paginated version has lots more complexities, as you note, +like having to keep track of how many things are in each page, +re-shuffling if new values are inserted, etc. That seems significantly +more complex. It might have some slight speed advantages (or +disadvantages!) at extremely high throughput, and the only way to +really know that would be to try it out. If you don't have time to +build it both ways and compare, my advice would be to start with the +simplest option (one row per user+value). Start simple and iterate! :) + + +
+ + +
+ +
+ Performance/Troubleshooting + +
+ Case Study #1 (Performance Issue On A Single Node) +
Scenario + Following a scheduled reboot, one data node began exhibiting unusual behavior. Routine MapReduce + jobs run against HBase tables which regularly completed in five or six minutes began taking 30 or 40 minutes + to finish. These jobs were consistently found to be waiting on map and reduce tasks assigned to the troubled data node + (e.g., the slow map tasks all had the same Input Split). + The situation came to a head during a distributed copy, when the copy was severely prolonged by the lagging node. + +
+
Hardware + Datanodes: + + Two 12-core processors + Six Enerprise SATA disks + 24GB of RAM + Two bonded gigabit NICs + + + Network: + + 10 Gigabit top-of-rack switches + 20 Gigabit bonded interconnects between racks. + + +
+
Hypotheses +
HBase "Hot Spot" Region + We hypothesized that we were experiencing a familiar point of pain: a "hot spot" region in an HBase table, + where uneven key-space distribution can funnel a huge number of requests to a single HBase region, bombarding the RegionServer + process and cause slow response time. Examination of the HBase Master status page showed that the number of HBase requests to the + troubled node was almost zero. Further, examination of the HBase logs showed that there were no region splits, compactions, or other region transitions + in progress. This effectively ruled out a "hot spot" as the root cause of the observed slowness. + +
+
HBase Region With Non-Local Data + Our next hypothesis was that one of the MapReduce tasks was requesting data from HBase that was not local to the datanode, thus + forcing HDFS to request data blocks from other servers over the network. Examination of the datanode logs showed that there were very + few blocks being requested over the network, indicating that the HBase region was correctly assigned, and that the majority of the necessary + data was located on the node. This ruled out the possibility of non-local data causing a slowdown. + +
+
Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk + After concluding that the Hadoop and HBase were not likely to be the culprits, we moved on to troubleshooting the datanode's hardware. + Java, by design, will periodically scan its entire memory space to do garbage collection. If system memory is heavily overcommitted, the Linux + kernel may enter a vicious cycle, using up all of its resources swapping Java heap back and forth from disk to RAM as Java tries to run garbage + collection. Further, a failing hard disk will often retry reads and/or writes many times before giving up and returning an error. This can manifest + as high iowait, as running processes wait for reads and writes to complete. Finally, a disk nearing the upper edge of its performance envelope will + begin to cause iowait as it informs the kernel that it cannot accept any more data, and the kernel queues incoming data into the dirty write pool in memory. + However, using vmstat(1) and free(1), we could see that no swap was being used, and the amount of disk IO was only a few kilobytes per second. + +
+
Slowness Due To High Processor Usage + Next, we checked to see whether the system was performing slowly simply due to very high computational load. top(1) showed that the system load + was higher than normal, but vmstat(1) and mpstat(1) showed that the amount of processor being used for actual computation was low. + +
+
Network Saturation (The Winner) + Since neither the disks nor the processors were being utilized heavily, we moved on to the performance of the network interfaces. The datanode had two + gigabit ethernet adapters, bonded to form an active-standby interface. ifconfig(8) showed some unusual anomalies, namely interface errors, overruns, framing errors. + While not unheard of, these kinds of errors are exceedingly rare on modern hardware which is operating as it should: + +$ /sbin/ifconfig bond0 +bond0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 +inet addr:10.x.x.x Bcast:10.x.x.255 Mask:255.255.255.0 +UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 +RX packets:2990700159 errors:12 dropped:0 overruns:1 frame:6 <--- Look Here! Errors! +TX packets:3443518196 errors:0 dropped:0 overruns:0 carrier:0 +collisions:0 txqueuelen:0 +RX bytes:2416328868676 (2.4 TB) TX bytes:3464991094001 (3.4 TB) + + + These errors immediately lead us to suspect that one or more of the ethernet interfaces might have negotiated the wrong line speed. This was confirmed both by running an ICMP ping + from an external host and observing round-trip-time in excess of 700ms, and by running ethtool(8) on the members of the bond interface and discovering that the active interface + was operating at 100Mbs/, full duplex. + +$ sudo ethtool eth0 +Settings for eth0: +Supported ports: [ TP ] +Supported link modes: 10baseT/Half 10baseT/Full + 100baseT/Half 100baseT/Full + 1000baseT/Full +Supports auto-negotiation: Yes +Advertised link modes: 10baseT/Half 10baseT/Full + 100baseT/Half 100baseT/Full + 1000baseT/Full +Advertised pause frame use: No +Advertised auto-negotiation: Yes +Link partner advertised link modes: Not reported +Link partner advertised pause frame use: No +Link partner advertised auto-negotiation: No +Speed: 100Mb/s <--- Look Here! Should say 1000Mb/s! +Duplex: Full +Port: Twisted Pair +PHYAD: 1 +Transceiver: internal +Auto-negotiation: on +MDI-X: Unknown +Supports Wake-on: umbg +Wake-on: g +Current message level: 0x00000003 (3) +Link detected: yes + + + In normal operation, the ICMP ping round trip time should be around 20ms, and the interface speed and duplex should read, "1000MB/s", and, "Full", respectively. + +
+
+
Resolution + After determining that the active ethernet adapter was at the incorrect speed, we used the ifenslave(8) command to make the standby interface + the active interface, which yielded an immediate improvement in MapReduce performance, and a 10 times improvement in network throughput: + + On the next trip to the datacenter, we determined that the line speed issue was ultimately caused by a bad network cable, which was replaced. + +
+
+
+ Case Study #2 (Performance Research 2012) + Investigation results of a self-described "we're not sure what's wrong, but it seems slow" problem. + http://gbif.blogspot.com/2012/03/hbase-performance-evaluation-continued.html + +
+ +
+ Case Study #3 (Performance Research 2010)) + + Investigation results of general cluster performance from 2010. Although this research is on an older version of the codebase, this writeup + is still very useful in terms of approach. + http://hstack.org/hbase-performance-testing/ + +
+ +
+ Case Study #4 (xcievers Config) + Case study of configuring xceivers, and diagnosing errors from mis-configurations. + http://www.larsgeorge.com/2012/03/hadoop-hbase-and-xceivers.html + + See also . + +
+ +
+ + diff --git a/src/docbkx/community.xml b/src/docbkx/community.xml new file mode 100644 index 000000000000..2c09908aed98 --- /dev/null +++ b/src/docbkx/community.xml @@ -0,0 +1,109 @@ + + + + Community +
+ Decisions +
+ Feature Branches + Feature Branches are easy to make. You do not have to be a committer to make one. Just request the name of your branch be added to JIRA up on the + developer's mailing list and a committer will add it for you. Thereafter you can file issues against your feature branch in Apache HBase (TM) JIRA. Your code you + keep elsewhere -- it should be public so it can be observed -- and you can update dev mailing list on progress. When the feature is ready for commit, + 3 +1s from committers will get your feature mergedSee HBase, mail # dev - Thoughts about large feature dev branches + +
+
+ Patch +1 Policy + +The below policy is something we put in place 09/2012. It is a +suggested policy rather than a hard requirement. We want to try it +first to see if it works before we cast it in stone. + + +Apache HBase is made of +components. +Components have one or more s. See the 'Description' field on the +components +JIRA page for who the current owners are by component. + + +Patches that fit within the scope of a single Apache HBase component require, +at least, a +1 by one of the component's owners before commit. If +owners are absent -- busy or otherwise -- two +1s by non-owners will +suffice. + + +Patches that span components need at least two +1s before they can be +committed, preferably +1s by owners of components touched by the +x-component patch (TODO: This needs tightening up but I think fine for +first pass). + + +Any -1 on a patch by anyone vetos a patch; it cannot be committed +until the justification for the -1 is addressed. + +
+
+
+ Community Roles +
+ Component Owner + +Component owners are listed in the description field on this Apache HBase JIRA components +page. The owners are listed in the 'Description' field rather than in the 'Component +Lead' field because the latter only allows us list one individual +whereas it is encouraged that components have multiple owners. + + +Owners are volunteers who are (usually, but not necessarily) expert in +their component domain and may have an agenda on how they think their +Apache HBase component should evolve. + + +Duties include: + + + +Owners will try and review patches that land within their component's scope. + + + + +If applicable, if an owner has an agenda, they will publish their +goals or the design toward which they are driving their component + + + + + +If you would like to be volunteer as a component owner, just write the +dev list and we'll sign you up. Owners do not need to be committers. + +
+
+ diff --git a/src/docbkx/configuration.xml b/src/docbkx/configuration.xml index 44936e19e98e..2d182fa6a600 100644 --- a/src/docbkx/configuration.xml +++ b/src/docbkx/configuration.xml @@ -26,14 +26,16 @@ * limitations under the License. */ --> - Configuration - This chapter is the Not-So-Quick start guide to HBase configuration. - Please read this chapter carefully and ensure that all requirements have + Apache HBase (TM) Configuration + This chapter is the Not-So-Quick start guide to Apache HBase (TM) configuration. It goes + over system requirements, Hadoop setup, the different Apache HBase run modes, and the + various configurations in HBase. Please read this chapter carefully. At a mimimum + ensure that all have been satisfied. Failure to do so will cause you (and us) grief debugging strange errors and/or data loss. - + - HBase uses the same configuration system as Hadoop. + Apache HBase uses the same configuration system as Apache Hadoop. To configure a deploy, edit a file of environment variables in conf/hbase-env.sh -- this configuration is used mostly by the launcher shell scripts getting the cluster @@ -55,17 +57,20 @@ to ensure well-formedness of your document after an edit session. content of the conf directory to all nodes of the cluster. HBase will not do this for you. Use rsync. - + +
+ Basic Prerequisites + This section lists required services and some required system configuration. + +
Java - - Just like Hadoop, HBase requires java 6 from Oracle. Usually - you'll want to use the latest version available except the problematic - u18 (u24 is the latest version as of this writing). + Just like Hadoop, HBase requires at least java 6 from + Oracle.
+
- Operating System + Operating System
ssh @@ -73,14 +78,20 @@ to ensure well-formedness of your document after an edit session. sshd must be running to use Hadoop's scripts to manage remote Hadoop and HBase daemons. You must be able to ssh to all nodes, including your local node, using passwordless login (Google - "ssh passwordless login"). + "ssh passwordless login"). If on mac osx, see the section, + SSH: Setting up Remote Desktop and Enabling Self-Login + on the hadoop wiki.
DNS - HBase uses the local hostname to self-report it's IP address. - Both forward and reverse DNS resolving should work. + HBase uses the local hostname to self-report its IP address. + Both forward and reverse DNS resolving must work in versions of + HBase previous to 0.92.0 + The hadoop-dns-checker tool can be used to verify + DNS is working correctly on the cluster. The project README file provides detailed instructions on usage. +. If your machine has multiple interfaces, HBase will use the interface that the primary hostname resolves to. @@ -97,15 +108,7 @@ to ensure well-formedness of your document after an edit session.
Loopback IP - HBase expects the loopback IP address to be 127.0.0.1. Ubuntu and some other distributions, - for example, will default to 127.0.1.1 and this will cause problems for you. - - /etc/hosts should look something like this: - - 127.0.0.1 localhost - 127.0.0.1 ubuntu.ubuntu-domain ubuntu - - + HBase expects the loopback IP address to be 127.0.0.1. See
@@ -132,7 +135,7 @@ to ensure well-formedness of your document after an edit session. - HBase is a database. It uses a lot of files all at the same time. + Apache HBase is a database. It uses a lot of files all at the same time. The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems is insufficient (On mac os x its 256). Any significant amount of loading will lead you to . @@ -141,9 +144,9 @@ to ensure well-formedness of your document after an edit session. 2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901 Do yourself a favor and change the upper bound on the number of file descriptors. Set it to north of 10k. The math runs roughly as follows: per ColumnFamily - there is at least one StoreFile and possibly up to 5 or 6 if the region is under load. Multiply the + there is at least one StoreFile and possibly up to 5 or 6 if the region is under load. Multiply the average number of StoreFiles per ColumnFamily times the number of regions per RegionServer. For example, assuming - that a schema had 3 ColumnFamilies per region with an average of 3 StoreFiles per ColumnFamily, + that a schema had 3 ColumnFamilies per region with an average of 3 StoreFiles per ColumnFamily, and there are 100 regions per RegionServer, the JVM will open 3 * 3 * 100 = 900 file descriptors (not counting open jar files, config files, etc.) @@ -153,7 +156,7 @@ to ensure well-formedness of your document after an edit session. See Jack Levin's major hdfs issues note up on the user list. The requirement that a database requires upping of system limits - is not peculiar to HBase. See for example the section + is not peculiar to Apache HBase. See for example the section Setting Shell Limits for the Oracle User in Short Guide to install Oracle 10 on Linux.. @@ -198,7 +201,7 @@ to ensure well-formedness of your document after an edit session.
Windows - HBase has been little tested running on Windows. Running a + Apache HBase has been little tested running on Windows. Running a production install of HBase on top of Windows is not recommended. @@ -206,32 +209,61 @@ to ensure well-formedness of your document after an edit session. xlink:href="http://cygwin.com/">Cygwin to have a *nix-like environment for the shell scripts. The full details are explained in the Windows - Installation guide. Also + Installation guide. Also search our user mailing list to pick up latest fixes figured by Windows users.
- +
<link xlink:href="http://hadoop.apache.org">Hadoop</link><indexterm> <primary>Hadoop</primary> </indexterm> - Please read all of this section - Please read this section to the end. Up front we - wade through the weeds of Hadoop versions. Later we talk of what you must do in HBase - to make it work w/ a particular Hadoop version. - - - - HBase will lose data unless it is running on an HDFS that has a durable - sync implementation. Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0 - DO NOT have this attribute. - Currently only Hadoop versions 0.20.205.x or any release in excess of this - version -- this includes hadoop 1.0.0 -- have a working, durable sync + Selecting a Hadoop version is critical for your HBase deployment. Below table shows some information about what versions of Hadoop are supported by various HBase versions. Based on the version of HBase, you should select the most appropriate version of Hadoop. We are not in the Hadoop distro selection business. You can use Hadoop distributions from Apache, or learn about vendor distributions of Hadoop at + + + Hadoop version support matrix + + + HBase-0.92.xHBase-0.94.xHBase-0.96 + + Hadoop-0.20.205S X X + Hadoop-0.22.x S X X + Hadoop-1.0.x S S S + Hadoop-1.1.x NT S S + Hadoop-0.23.x X S NT + Hadoop-2.x X S S +
+ + Where + + S = supported and tested, + X = not supported, + NT = it should run, but not tested enough. + + + + Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its lib directory. The bundled jar is ONLY for use in standalone mode. In distributed mode, it is critical that the version of Hadoop that is out on your cluster match what is under HBase. Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop version mismatch issues have various manifestations but often all looks like its hung up. + +
+ Apache HBase 0.92 and 0.94 + HBase 0.92 and 0.94 versions can work with Hadoop versions, 0.20.205, 0.22.x, 1.0.x, and 1.1.x. HBase-0.94 can additionally work with Hadoop-0.23.x and 2.x, but you may have to recompile the code using the specific maven profile (see top level pom.xml) +
+ +
+ Apache HBase 0.96 + Apache HBase 0.96.0 requires Apache Hadoop 1.x at a minimum, and it can run equally well on hadoop-2.0. + As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required. We will no longer run properly on older Hadoops such as 0.20.205 or branch-0.20-append. Do not move to Apache HBase 0.96.x if you cannot upgrade your HadoopSee HBase, mail # dev - DISCUSS: Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?. +
+ +
+ Hadoop versions 0.20.x - 1.x + + HBase will lose data unless it is running on an HDFS that has a durable + sync implementation. DO NOT use Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0 which DO NOT have this attribute. Currently only Hadoop versions 0.20.205.x or any release in excess of this version -- this includes hadoop-1.0.0 -- have a working, durable sync - On Hadoop Versions The Cloudera blog post An update on Apache Hadoop 1.0 by Charles Zedlweski has a nice exposition on how all the Hadoop versions relate. Its worth checking out if you are having trouble making sense of the @@ -250,57 +282,18 @@ to ensure well-formedness of your document after an edit session. You will have to restart your cluster after making this edit. Ignore the chicken-little comment you'll find in the hdfs-default.xml in the - description for the dfs.support.append configuration; it says it is not enabled because there - are ... bugs in the 'append code' and is not supported in any production - cluster.. This comment is stale, from another era, and while I'm sure there - are bugs, the sync/append code has been running - in production at large scale deploys and is on - by default in the offerings of hadoop by commercial vendors - Until recently only the - branch-0.20-append - branch had a working sync but no official release was ever made from this branch. - You had to build it yourself. Michael Noll wrote a detailed blog, - Building - an Hadoop 0.20.x version for HBase 0.90.2, on how to build an - Hadoop from branch-0.20-append. Recommended. - Praveen Kumar has written - a complimentary article, - Building Hadoop and HBase for HBase Maven application development. -Cloudera have dfs.support.append set to true by default.. - -Or use the - Cloudera or - MapR distributions. - Cloudera' CDH3 - is Apache Hadoop 0.20.x plus patches including all of the - branch-0.20-append - additions needed to add a durable sync. Use the released, most recent version of CDH3. - - MapR - includes a commercial, reimplementation of HDFS. - It has a durable sync as well as some other interesting features that are not - yet in Apache Hadoop. Their M3 - product is free to use and unlimited. - - - Because HBase depends on Hadoop, it bundles an instance of the - Hadoop jar under its lib directory. The bundled jar is ONLY for use in standalone mode. - In distributed mode, it is critical that the version of Hadoop that is out - on your cluster match what is under HBase. Replace the hadoop jar found in the HBase - lib directory with the hadoop jar you are running on - your cluster to avoid version mismatch issues. Make sure you - replace the jar in HBase everywhere on your cluster. Hadoop version - mismatch issues have various manifestations but often all looks like - its hung up. - + description for the dfs.support.append configuration. + +
- Hadoop Security - HBase will run on any Hadoop 0.20.x that incorporates Hadoop - security features -- e.g. Y! 0.20S or CDH3B3 -- as long as you do as + Apache HBase on Secure Hadoop + Apache HBase will run on any Hadoop 0.20.x that incorporates Hadoop + security features as long as you do as suggested above and replace the Hadoop jar that ships with HBase - with the secure version. + with the secure version. If you want to read more about how to setup + Secure HBase, see .
- +
<varname>dfs.datanode.max.xcievers</varname><indexterm> <primary>xcievers</primary> @@ -331,9 +324,12 @@ to ensure well-formedness of your document after an edit session. java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry...</code> <footnote><para>See <link xlink:href="http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html">Hadoop HDFS: Deceived by Xciever</link> for an informative rant on xceivering.</para></footnote></para> + <para>See also <xref linkend="casestudies.xceivers"/> + </para> </section> - + </section> <!-- hadoop --> + </section> <section xml:id="standalone_dist"> <title>HBase run modes: Standalone and Distributed @@ -376,7 +372,7 @@ to ensure well-formedness of your document after an edit session. Distributed modes require an instance of the Hadoop Distributed File System (HDFS). See the Hadoop + xlink:href="http://hadoop.apache.org/common/docs/r1.1.1/api/overview-summary.html#overview_description"> requirements and instructions for how to set up a HDFS. Before proceeding, ensure you have an appropriate, working HDFS. @@ -395,57 +391,92 @@ to ensure well-formedness of your document after an edit session. HBase. Do not use this configuration for production nor for evaluating HBase performance. - Once you have confirmed your HDFS setup, edit - conf/hbase-site.xml. This is the file into + First, setup your HDFS in pseudo-distributed mode. + + Next, configure HBase. Below is an example conf/hbase-site.xml. + This is the file into which you add local customizations and overrides for - and . Point HBase at the running Hadoop HDFS - instance by setting the hbase.rootdir property. - This property points HBase at the Hadoop filesystem instance to use. - For example, adding the properties below to your - hbase-site.xml says that HBase should use the - /hbase directory in the HDFS whose namenode is - at port 8020 on your local machine, and that it should run with one - replica only (recommended for pseudo-distributed mode): + and . + Note that the hbase.rootdir property points to the + local HDFS instance. + - + Now skip to for how to start and verify your + pseudo-distributed install. + See Pseudo-distributed + mode extras for notes on how to start extra Masters and + RegionServers when running pseudo-distributed. + + + + Let HBase create the hbase.rootdir + directory. If you don't, you'll get warning saying HBase needs a + migration run because the directory is missing files expected by + HBase (it'll create them if you let it). + + +
+ Pseudo-distributed Configuration File + Below is a sample pseudo-distributed file for the node h-24-30.example.com. +hbase-site.xml + <configuration> ... <property> <name>hbase.rootdir</name> - <value>hdfs://localhost:8020/hbase</value> - <description>The directory shared by RegionServers. - </description> + <value>hdfs://h-24-30.sfo.stumble.net:8020/hbase</value> </property> <property> - <name>dfs.replication</name> - <value>1</value> - <description>The replication count for HLog and HFile storage. Should not be greater than HDFS datanode count. - </description> + <name>hbase.cluster.distributed</name> + <value>true</value> + </property> + <property> + <name>hbase.zookeeper.quorum</name> + <value>h-24-30.sfo.stumble.net</value> </property> ... </configuration> + - - Let HBase create the hbase.rootdir - directory. If you don't, you'll get warning saying HBase needs a - migration run because the directory is missing files expected by - HBase (it'll create them if you let it). - +
- - Above we bind to localhost. This means - that a remote client cannot connect. Amend accordingly, if you - want to connect from a remote location. - +
+ Pseudo-distributed Extras + +
+ Startup + To start up the initial HBase cluster... + % bin/start-hbase.sh + + To start up an extra backup master(s) on the same server run... + % bin/local-master-backup.sh start 1 + ... the '1' means use ports 60001 & 60011, and this backup master's logfile will be at logs/hbase-${USER}-1-master-${HOSTNAME}.log. + + To startup multiple backup masters run... % bin/local-master-backup.sh start 2 3 You can start up to 9 backup masters (10 total). + + To start up more regionservers... + % bin/local-regionservers.sh start 1 + where '1' means use ports 60201 & 60301 and its logfile will be at logs/hbase-${USER}-1-regionserver-${HOSTNAME}.log. + + To add 4 more regionservers in addition to the one you just started by running... % bin/local-regionservers.sh start 2 3 4 5 + This supports up to 99 extra regionservers (100 total). + +
+
+ Stop + Assuming you want to stop master backup # 1, run... + % cat /tmp/hbase-${USER}-1-master.pid |xargs kill -9 + Note that bin/local-master-backup.sh stop 1 will try to stop the cluster along with the master. + + To stop an individual regionserver, run... + % bin/local-regionservers.sh stop 1 + + +
+ +
- Now skip to for how to start and verify your - pseudo-distributed install. - See Pseudo-distributed - mode extras for notes on how to start extra Masters and - RegionServers when running pseudo-distributed. -
@@ -542,7 +573,7 @@ to ensure well-formedness of your document after an edit session.
Running and Confirming Your Installation - + Make sure HDFS is running first. Start and stop the Hadoop HDFS daemons by running bin/start-hdfs.sh over in the @@ -552,31 +583,31 @@ to ensure well-formedness of your document after an edit session. not normally use the mapreduce daemons. These do not need to be started. - + If you are managing your own ZooKeeper, start it and confirm its running else, HBase will start up ZooKeeper for you as part of its start process. - + Start HBase with the following command: - + bin/start-hbase.sh - Run the above from the + Run the above from the HBASE_HOME - directory. + directory. You should now have a running HBase instance. HBase logs can be found in the logs subdirectory. Check them out especially if HBase had trouble starting. - + HBase also puts up a UI listing vital attributes. By default its deployed on the Master host at port 60010 (HBase RegionServers listen @@ -586,13 +617,13 @@ to ensure well-formedness of your document after an edit session. Master's homepage you'd point your browser at http://master.example.org:60010. - + Once HBase has started, see the for how to create tables, add data, scan your insertions, and finally disable and drop your tables. - + To stop HBase after exiting the HBase shell enter $ ./bin/stop-hbase.sh @@ -602,574 +633,15 @@ stopping hbase............... Shutdown can take a moment to until HBase has shut down completely before stopping the Hadoop daemons. - +
- -
- ZooKeeper<indexterm> - <primary>ZooKeeper</primary> - </indexterm> - - A distributed HBase depends on a running ZooKeeper cluster. - All participating nodes and clients need to be able to access the - running ZooKeeper ensemble. HBase by default manages a ZooKeeper - "cluster" for you. It will start and stop the ZooKeeper ensemble - as part of the HBase start/stop process. You can also manage the - ZooKeeper ensemble independent of HBase and just point HBase at - the cluster it should use. To toggle HBase management of - ZooKeeper, use the HBASE_MANAGES_ZK variable in - conf/hbase-env.sh. This variable, which - defaults to true, tells HBase whether to - start/stop the ZooKeeper ensemble servers as part of HBase - start/stop. - - When HBase manages the ZooKeeper ensemble, you can specify - ZooKeeper configuration using its native - zoo.cfg file, or, the easier option is to - just specify ZooKeeper options directly in - conf/hbase-site.xml. A ZooKeeper - configuration option can be set as a property in the HBase - hbase-site.xml XML configuration file by - prefacing the ZooKeeper option name with - hbase.zookeeper.property. For example, the - clientPort setting in ZooKeeper can be changed - by setting the - hbase.zookeeper.property.clientPort property. - For all default values used by HBase, including ZooKeeper - configuration, see . Look for the - hbase.zookeeper.property prefix - For the full list of ZooKeeper configurations, see - ZooKeeper's zoo.cfg. HBase does not ship - with a zoo.cfg so you will need to browse - the conf directory in an appropriate - ZooKeeper download. - - - You must at least list the ensemble servers in - hbase-site.xml using the - hbase.zookeeper.quorum property. This property - defaults to a single ensemble member at - localhost which is not suitable for a fully - distributed HBase. (It binds to the local machine only and remote - clients will not be able to connect). - How many ZooKeepers should I run? - - You can run a ZooKeeper ensemble that comprises 1 node - only but in production it is recommended that you run a - ZooKeeper ensemble of 3, 5 or 7 machines; the more members an - ensemble has, the more tolerant the ensemble is of host - failures. Also, run an odd number of machines. In ZooKeeper, - an even number of peers is supported, but it is normally not used - because an even sized ensemble requires, proportionally, more peers - to form a quorum than an odd sized ensemble requires. For example, an - ensemble with 4 peers requires 3 to form a quorum, while an ensemble with - 5 also requires 3 to form a quorum. Thus, an ensemble of 5 allows 2 peers to - fail, and thus is more fault tolerant than the ensemble of 4, which allows - only 1 down peer. - - Give each ZooKeeper server around 1GB of RAM, and if possible, its own - dedicated disk (A dedicated disk is the best thing you can do - to ensure a performant ZooKeeper ensemble). For very heavily - loaded clusters, run ZooKeeper servers on separate machines - from RegionServers (DataNodes and TaskTrackers). - - - For example, to have HBase manage a ZooKeeper quorum on - nodes rs{1,2,3,4,5}.example.com, bound to - port 2222 (the default is 2181) ensure - HBASE_MANAGE_ZK is commented out or set to - true in conf/hbase-env.sh - and then edit conf/hbase-site.xml and set - hbase.zookeeper.property.clientPort and - hbase.zookeeper.quorum. You should also set - hbase.zookeeper.property.dataDir to other than - the default as the default has ZooKeeper persist data under - /tmp which is often cleared on system - restart. In the example below we have ZooKeeper persist to - /user/local/zookeeper. - <configuration> - ... - <property> - <name>hbase.zookeeper.property.clientPort</name> - <value>2222</value> - <description>Property from ZooKeeper's config zoo.cfg. - The port at which the clients will connect. - </description> - </property> - <property> - <name>hbase.zookeeper.quorum</name> - <value>rs1.example.com,rs2.example.com,rs3.example.com,rs4.example.com,rs5.example.com</value> - <description>Comma separated list of servers in the ZooKeeper Quorum. - For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". - By default this is set to localhost for local and pseudo-distributed modes - of operation. For a fully-distributed setup, this should be set to a full - list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh - this is the list of servers which we will start/stop ZooKeeper on. - </description> - </property> - <property> - <name>hbase.zookeeper.property.dataDir</name> - <value>/usr/local/zookeeper</value> - <description>Property from ZooKeeper's config zoo.cfg. - The directory where the snapshot is stored. - </description> - </property> - ... - </configuration> - -
- Using existing ZooKeeper ensemble - - To point HBase at an existing ZooKeeper cluster, one that - is not managed by HBase, set HBASE_MANAGES_ZK - in conf/hbase-env.sh to false - - ... - # Tell HBase whether it should manage it's own instance of Zookeeper or not. - export HBASE_MANAGES_ZK=false Next set ensemble locations - and client port, if non-standard, in - hbase-site.xml, or add a suitably - configured zoo.cfg to HBase's - CLASSPATH. HBase will prefer the - configuration found in zoo.cfg over any - settings in hbase-site.xml. - - When HBase manages ZooKeeper, it will start/stop the - ZooKeeper servers as a part of the regular start/stop scripts. - If you would like to run ZooKeeper yourself, independent of - HBase start/stop, you would do the following - - -${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper - - - Note that you can use HBase in this manner to spin up a - ZooKeeper cluster, unrelated to HBase. Just make sure to set - HBASE_MANAGES_ZK to false - if you want it to stay up across HBase restarts so that when - HBase shuts down, it doesn't take ZooKeeper down with it. - - For more information about running a distinct ZooKeeper - cluster, see the ZooKeeper Getting - Started Guide. Additionally, see the ZooKeeper Wiki or the - ZooKeeper documentation - for more information on ZooKeeper sizing. - -
- - -
- SASL Authentication with ZooKeeper - Newer releases of HBase (>= 0.92) will - support connecting to a ZooKeeper Quorum that supports - SASL authentication (which is available in Zookeeper - versions 3.4.0 or later). - - This describes how to set up HBase to mutually - authenticate with a ZooKeeper Quorum. ZooKeeper/HBase - mutual authentication (HBASE-2418) - is required as part of a complete secure HBase configuration - (HBASE-3025). - - For simplicity of explication, this section ignores - additional configuration required (Secure HDFS and Coprocessor - configuration). It's recommended to begin with an - HBase-managed Zookeeper configuration (as opposed to a - standalone Zookeeper quorum) for ease of learning. - - -
Operating System Prerequisites
- - - You need to have a working Kerberos KDC setup. For - each $HOST that will run a ZooKeeper - server, you should have a principle - zookeeper/$HOST. For each such host, - add a service key (using the kadmin or - kadmin.local tool's ktadd - command) for zookeeper/$HOST and copy - this file to $HOST, and make it - readable only to the user that will run zookeeper on - $HOST. Note the location of this file, - which we will use below as - $PATH_TO_ZOOKEEPER_KEYTAB. - - - - Similarly, for each $HOST that will run - an HBase server (master or regionserver), you should - have a principle: hbase/$HOST. For each - host, add a keytab file called - hbase.keytab containing a service - key for hbase/$HOST, copy this file to - $HOST, and make it readable only to the - user that will run an HBase service on - $HOST. Note the location of this file, - which we will use below as - $PATH_TO_HBASE_KEYTAB. - - - - Each user who will be an HBase client should also be - given a Kerberos principal. This principal should - usually have a password assigned to it (as opposed to, - as with the HBase servers, a keytab file) which only - this user knows. The client's principal's - maxrenewlife should be set so that it can - be renewed enough so that the user can complete their - HBase client processes. For example, if a user runs a - long-running HBase client process that takes at most 3 - days, we might create this user's principal within - kadmin with: addprinc -maxrenewlife - 3days. The Zookeeper client and server - libraries manage their own ticket refreshment by - running threads that wake up periodically to do the - refreshment. - - - On each host that will run an HBase client - (e.g. hbase shell), add the following - file to the HBase home directory's conf - directory: - - - Client { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=false - useTicketCache=true; - }; - - - We'll refer to this JAAS configuration file as - $CLIENT_CONF below. - -
- HBase-managed Zookeeper Configuration - - On each node that will run a zookeeper, a - master, or a regionserver, create a JAAS - configuration file in the conf directory of the node's - HBASE_HOME directory that looks like the - following: - - - Server { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=true - keyTab="$PATH_TO_ZOOKEEPER_KEYTAB" - storeKey=true - useTicketCache=false - principal="zookeeper/$HOST"; - }; - Client { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=true - useTicketCache=false - keyTab="$PATH_TO_HBASE_KEYTAB" - principal="hbase/$HOST"; - }; - - - where the $PATH_TO_HBASE_KEYTAB and - $PATH_TO_ZOOKEEPER_KEYTAB files are what - you created above, and $HOST is the hostname for that - node. - - The Server section will be used by - the Zookeeper quorum server, while the - Client section will be used by the HBase - master and regionservers. The path to this file should - be substituted for the text $HBASE_SERVER_CONF - in the hbase-env.sh - listing below. - - - The path to this file should be substituted for the - text $CLIENT_CONF in the - hbase-env.sh listing below. - - - Modify your hbase-env.sh to include the - following: - - - export HBASE_OPTS="-Djava.security.auth.login.config=$CLIENT_CONF" - export HBASE_MANAGES_ZK=true - export HBASE_ZOOKEEPER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" - export HBASE_MASTER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" - export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" - - - where $HBASE_SERVER_CONF and - $CLIENT_CONF are the full paths to the - JAAS configuration files created above. - - Modify your hbase-site.xml on each node - that will run zookeeper, master or regionserver to contain: - - - - hbase.zookeeper.quorum - $ZK_NODES - - - hbase.cluster.distributed - true - - - hbase.zookeeper.property.authProvider.1 - org.apache.zookeeper.server.auth.SASLAuthenticationProvider - - - hbase.zookeeper.property.kerberos.removeHostFromPrincipal - true - - - hbase.zookeeper.property.kerberos.removeRealmFromPrincipal - true - - - ]]> - - where $ZK_NODES is the - comma-separated list of hostnames of the Zookeeper - Quorum hosts. - - Start your hbase cluster by running one or more - of the following set of commands on the appropriate - hosts: - - - - bin/hbase zookeeper start - bin/hbase master start - bin/hbase regionserver start - - -
- -
External Zookeeper Configuration - Add a JAAS configuration file that looks like: - - - Client { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=true - useTicketCache=false - keyTab="$PATH_TO_HBASE_KEYTAB" - principal="hbase/$HOST"; - }; - - - where the $PATH_TO_HBASE_KEYTAB is the keytab - created above for HBase services to run on this host, and $HOST is the - hostname for that node. Put this in the HBase home's - configuration directory. We'll refer to this file's - full pathname as $HBASE_SERVER_CONF below. - - Modify your hbase-env.sh to include the following: - - - export HBASE_OPTS="-Djava.security.auth.login.config=$CLIENT_CONF" - export HBASE_MANAGES_ZK=false - export HBASE_MASTER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" - export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" - - - - Modify your hbase-site.xml on each node - that will run a master or regionserver to contain: - - - - hbase.zookeeper.quorum - $ZK_NODES - - - hbase.cluster.distributed - true - - - ]]> - - - where $ZK_NODES is the - comma-separated list of hostnames of the Zookeeper - Quorum hosts. - - - Add a zoo.cfg for each Zookeeper Quorum host containing: - - authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider - kerberos.removeHostFromPrincipal=true - kerberos.removeRealmFromPrincipal=true - - - Also on each of these hosts, create a JAAS configuration file containing: - - - Server { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=true - keyTab="$PATH_TO_ZOOKEEPER_KEYTAB" - storeKey=true - useTicketCache=false - principal="zookeeper/$HOST"; - }; - - - where $HOST is the hostname of each - Quorum host. We will refer to the full pathname of - this file as $ZK_SERVER_CONF below. - - - - - Start your Zookeepers on each Zookeeper Quorum host with: - - - SERVER_JVMFLAGS="-Djava.security.auth.login.config=$ZK_SERVER_CONF" bin/zkServer start - - - - - - Start your HBase cluster by running one or more of the following set of commands on the appropriate nodes: - - - - bin/hbase master start - bin/hbase regionserver start - - - -
- -
- Zookeeper Server Authentication Log Output - If the configuration above is successful, - you should see something similar to the following in - your Zookeeper server logs: - -11/12/05 22:43:39 INFO zookeeper.Login: successfully logged in. -11/12/05 22:43:39 INFO server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 -11/12/05 22:43:39 INFO zookeeper.Login: TGT refresh thread started. -11/12/05 22:43:39 INFO zookeeper.Login: TGT valid starting at: Mon Dec 05 22:43:39 UTC 2011 -11/12/05 22:43:39 INFO zookeeper.Login: TGT expires: Tue Dec 06 22:43:39 UTC 2011 -11/12/05 22:43:39 INFO zookeeper.Login: TGT refresh sleeping until: Tue Dec 06 18:36:42 UTC 2011 -.. -11/12/05 22:43:59 INFO auth.SaslServerCallbackHandler: - Successfully authenticated client: authenticationID=hbase/ip-10-166-175-249.us-west-1.compute.internal@HADOOP.LOCALDOMAIN; - authorizationID=hbase/ip-10-166-175-249.us-west-1.compute.internal@HADOOP.LOCALDOMAIN. -11/12/05 22:43:59 INFO auth.SaslServerCallbackHandler: Setting authorizedID: hbase -11/12/05 22:43:59 INFO server.ZooKeeperServer: adding SASL authorization for authorizationID: hbase - - - - -
- -
- Zookeeper Client Authentication Log Output - On the Zookeeper client side (HBase master or regionserver), - you should see something similar to the following: - - -11/12/05 22:43:59 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-166-175-249.us-west-1.compute.internal:2181 sessionTimeout=180000 watcher=master:60000 -11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.166.175.249:2181 -11/12/05 22:43:59 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 14851@ip-10-166-175-249 -11/12/05 22:43:59 INFO zookeeper.Login: successfully logged in. -11/12/05 22:43:59 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. -11/12/05 22:43:59 INFO zookeeper.Login: TGT refresh thread started. -11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-166-175-249.us-west-1.compute.internal/10.166.175.249:2181, initiating session -11/12/05 22:43:59 INFO zookeeper.Login: TGT valid starting at: Mon Dec 05 22:43:59 UTC 2011 -11/12/05 22:43:59 INFO zookeeper.Login: TGT expires: Tue Dec 06 22:43:59 UTC 2011 -11/12/05 22:43:59 INFO zookeeper.Login: TGT refresh sleeping until: Tue Dec 06 18:30:37 UTC 2011 -11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-166-175-249.us-west-1.compute.internal/10.166.175.249:2181, sessionid = 0x134106594320000, negotiated timeout = 180000 - - -
- -
- Configuration from Scratch - - This has been tested on the current standard Amazon - Linux AMI. First setup KDC and principals as - described above. Next checkout code and run a sanity - check. - - - git clone git://git.apache.org/hbase.git - cd hbase - mvn -Psecurity,localTests clean test -Dtest=TestZooKeeperACL - - - Then configure HBase as described above. - Manually edit target/cached_classpath.txt (see below).. - - - bin/hbase zookeeper & - bin/hbase master & - bin/hbase regionserver & - -
- - -
- Future improvements - -
Fix target/cached_classpath.txt - - You must override the standard hadoop-core jar file from the - target/cached_classpath.txt - file with the version containing the HADOOP-7070 fix. You can use the following script to do this: - - - echo `find ~/.m2 -name "*hadoop-core*7070*SNAPSHOT.jar"` ':' `cat target/cached_classpath.txt` | sed 's/ //g' > target/tmp.txt - mv target/tmp.txt target/cached_classpath.txt - - - - -
- -
- Set JAAS configuration - programmatically - - - This would avoid the need for a separate Hadoop jar - that fixes HADOOP-7070. -
- -
- Elimination of - <code>kerberos.removeHostFromPrincipal</code> and - <code>kerberos.removeRealmFromPrincipal</code> -
- -
- - -
- - -
- - -
+
Configuration Files - +
<filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename> Just as in Hadoop where you add site-specific HDFS configuration @@ -1197,7 +669,7 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper The generated file is a docbook section with a glossary in it--> + href="../../src/main/resources/hbase-default.xml" />
@@ -1242,8 +714,17 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper used by tests). - Minimally, a client of HBase needs the hbase, hadoop, log4j, commons-logging, commons-lang, - and ZooKeeper jars in its CLASSPATH connecting to a cluster. + Minimally, a client of HBase needs several libraries in its CLASSPATH when connecting to a cluster, including: + +commons-configuration (commons-configuration-1.6.jar) +commons-lang (commons-lang-2.5.jar) +commons-logging (commons-logging-1.1.1.jar) +hadoop-core (hadoop-core-1.0.0.jar) +hbase (hbase-0.92.0.jar) +log4j (log4j-1.2.16.jar) +slf4j-api (slf4j-api-1.5.8.jar) +slf4j-log4j (slf4j-log4j12-1.5.8.jar) +zookeeper (zookeeper-3.4.2.jar) An example basic hbase-site.xml for client only @@ -1261,7 +742,7 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper ]]> - +
Java client configuration The configuration used by a Java client is kept @@ -1270,15 +751,15 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper on invocation, will read in the content of the first hbase-site.xml found on the client's CLASSPATH, if one is present (Invocation will also factor in any hbase-default.xml found; - an hbase-default.xml ships inside the hbase.X.X.X.jar). + an hbase-default.xml ships inside the hbase.X.X.X.jar). It is also possible to specify configuration directly without having to read from a hbase-site.xml. For example, to set the ZooKeeper ensemble for the cluster programmatically do as follows: Configuration config = HBaseConfiguration.create(); -config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zookeeper locally +config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zookeeper locally If multiple ZooKeeper instances make up your ZooKeeper ensemble, they may be specified in a comma-separated list (just as in the hbase-site.xml file). - This populated Configuration instance can then be passed to an + This populated Configuration instance can then be passed to an HTable, and so on. @@ -1286,7 +767,7 @@ config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zooke
- +
Example Configurations @@ -1378,7 +859,7 @@ config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zooke 1G. - + $ git diff hbase-env.sh diff --git a/conf/hbase-env.sh b/conf/hbase-env.sh index e70ebc6..96f8c27 100644 @@ -1386,11 +867,11 @@ index e70ebc6..96f8c27 100644 +++ b/conf/hbase-env.sh @@ -31,7 +31,7 @@ export JAVA_HOME=/usr/lib//jvm/java-6-sun/ # export HBASE_CLASSPATH= - + # The maximum amount of heap to use, in MB. Default is 1000. -# export HBASE_HEAPSIZE=1000 +export HBASE_HEAPSIZE=4096 - + # Extra Java runtime options. # Below are what we set by default. May only work with SUN JVM. @@ -1402,8 +883,8 @@ index e70ebc6..96f8c27 100644
- - + +
The Important Configurations Below we list what the important @@ -1415,9 +896,23 @@ index e70ebc6..96f8c27 100644
Required Configurations Review the and sections. +
Big Cluster Configurations + If a cluster with a lot of regions, it is possible if an eager beaver + regionserver checks in soon after master start while all the rest in the + cluster are laggardly, this first server to checkin will be assigned all + regions. If lots of regions, this first server could buckle under the + load. To prevent the above scenario happening up the + hbase.master.wait.on.regionservers.mintostart from its + default value of 1. See + HBASE-6389 Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments + for more detail. + +
Recommended Configurations +
+ ZooKeeper Configuration
<varname>zookeeper.session.timeout</varname> The default timeout is three minutes (specified in milliseconds). This means that if a server crashes, it will be three minutes before the Master notices @@ -1427,7 +922,7 @@ index e70ebc6..96f8c27 100644 configuration under control otherwise, a long garbage collection that lasts beyond the ZooKeeper session timeout will take out your RegionServer (You might be fine with this -- you probably want recovery to start - on the server if a RegionServer has been in GC for a long period of time). + on the server if a RegionServer has been in GC for a long period of time). To change this configuration, edit hbase-site.xml, copy the changed file around the cluster and restart. @@ -1443,6 +938,18 @@ index e70ebc6..96f8c27 100644
Number of ZooKeeper Instances See . +
+
+
+ HDFS Configurations +
+ dfs.datanode.failed.volumes.tolerated + This is the "...number of volumes that are allowed to fail before a datanode stops offering service. By default + any volume failure will cause a datanode to shutdown" from the hdfs-default.xml + description. If you have > three or four disks, you might want to set this to 1 or if you have many disks, + two or more. + +
<varname>hbase.regionserver.handler.count</varname> @@ -1503,7 +1010,7 @@ index e70ebc6..96f8c27 100644 cluster (You can always later manually split the big Regions should one prove hot and you want to spread the request load over the cluster). A lower number of regions is preferred, generally in the range of 20 to low-hundreds - per RegionServer. Adjust the regionsize as appropriate to achieve this number. + per RegionServer. Adjust the regionsize as appropriate to achieve this number. For the 0.90.x codebase, the upper-bound of regionsize is about 4Gb, with a default of 256Mb. For 0.92.x codebase, due to the HFile v2 change much larger regionsizes can be supported (e.g., 20Gb). @@ -1511,10 +1018,58 @@ index e70ebc6..96f8c27 100644 You may need to experiment with this setting based on your hardware configuration and application needs. Adjust hbase.hregion.max.filesize in your hbase-site.xml. - RegionSize can also be set on a per-table basis via + RegionSize can also be set on a per-table basis via HTableDescriptor. - +
+ How many regions per RegionServer? + + Typically you want to keep your region count low on HBase for numerous reasons. + Usually right around 100 regions per RegionServer has yielded the best results. + Here are some of the reasons below for keeping region count low: + + + MSLAB requires 2mb per memstore (that's 2mb per family per region). + 1000 regions that have 2 families each is 3.9GB of heap used, and it's not even storing data yet. NB: the 2MB value is configurable. + + If you fill all the regions at somewhat the same rate, the global memory usage makes it that it forces tiny + flushes when you have too many regions which in turn generates compactions. + Rewriting the same data tens of times is the last thing you want. + An example is filling 1000 regions (with one family) equally and let's consider a lower bound for global memstore + usage of 5GB (the region server would have a big heap). + Once it reaches 5GB it will force flush the biggest region, + at that point they should almost all have about 5MB of data so + it would flush that amount. 5MB inserted later, it would flush another + region that will now have a bit over 5MB of data, and so on. + A basic formula for the amount of regions to have per region server would + look like this: + Heap * upper global memstore limit = amount of heap devoted to memstore + then the amount of heap devoted to memstore / (Number of regions per RS * CFs). + This will give you the rough memstore size if everything is being written to. + A more accurate formula is + Heap * upper global memstore limit = amount of heap devoted to memstore then the + amount of heap devoted to memstore / (Number of actively written regions per RS * CFs). + This can allot you a higher region count from the write perspective if you know how many + regions you will be writing to at one time. + + The master as is is allergic to tons of regions, and will + take a lot of time assigning them and moving them around in batches. + The reason is that it's heavy on ZK usage, and it's not very async + at the moment (could really be improved -- and has been imporoved a bunch + in 0.96 hbase). + + + In older versions of HBase (pre-v2 hfile, 0.90 and previous), tons of regions + on a few RS can cause the store file index to rise raising heap usage and can + create memory pressure or OOME on the RSs + + + + Another issue is the effect of the number of regions on mapreduce jobs. + Keeping 5 regions per RS would be too low for a job, whereas 1000 will generate too many maps. + +
+
Managed Splitting @@ -1567,23 +1122,30 @@ of all regions.
Managed Compactions - A common administrative technique is to manage major compactions manually, rather than letting + A common administrative technique is to manage major compactions manually, rather than letting HBase do it. By default, HConstants.MAJOR_COMPACTION_PERIOD is one day and major compactions may kick in when you least desire it - especially on a busy system. To turn off automatic major compactions set - the value to 0. + the value to 0. It is important to stress that major compactions are absolutely necessary for StoreFile cleanup, the only variant is when - they occur. They can be administered through the HBase shell, or via + they occur. They can be administered through the HBase shell, or via HBaseAdmin. For more information about compactions and the compaction file selection process, see
- + +
Speculative Execution + Speculative Execution of MapReduce tasks is on by default, and for HBase clusters it is generally advised to turn off + Speculative Execution at a system-level unless you need it for a specific case, where it can be configured per-job. + Set the properties mapred.map.tasks.speculative.execution and + mapred.reduce.tasks.speculative.execution to false. + +
Other Configurations
Balancer - The balancer is periodic operation run on the master to redistribute regions on the cluster. It is configured via + The balancer is a periodic operation which is run on the master to redistribute regions on the cluster. It is configured via hbase.balancer.period and defaults to 300000 (5 minutes). See for more information on the LoadBalancer. @@ -1596,38 +1158,18 @@ of all regions. on the size you need by surveying regionserver UIs; you'll see index block size accounted near the top of the webpage).
-
- -
- -
- Bloom Filter Configuration -
- <varname>io.hfile.bloom.enabled</varname> global kill - switch - - io.hfile.bloom.enabled in - Configuration serves as the kill switch in case - something goes wrong. Default = true. -
- -
- <varname>io.hfile.bloom.error.rate</varname> +
+ <link xlink:href="http://en.wikipedia.org/wiki/Nagle's_algorithm">Nagle's</link> or the small package problem + If a big 40ms or so occasional delay is seen in operations against HBase, + try the Nagles' setting. For example, see the user mailing list thread, + Inconsistent scan performance with caching set to 1 + and the issue cited therein where setting notcpdelay improved scan speeds. You might also + see the graphs on the tail of HBASE-7008 Set scanner caching to a better default + where our Lars Hofhansl tries various data sizes w/ Nagle's on and off measuring the effect. +
- io.hfile.bloom.error.rate = average false - positive rate. Default = 1%. Decrease rate by ½ (e.g. to .5%) == +1 - bit per bloom entry. -
+
-
- <varname>io.hfile.bloom.max.fold</varname> +
- io.hfile.bloom.max.fold = guaranteed minimum - fold rate. Most people should leave this alone. Default = 7, or can - collapse to at least 1/128th of original size. See the - Development Process section of the document BloomFilters - in HBase for more on what this option means. -
-
diff --git a/src/docbkx/customization.xsl b/src/docbkx/customization.xsl index d80a2b5abd61..a5065a48ff93 100644 --- a/src/docbkx/customization.xsl +++ b/src/docbkx/customization.xsl @@ -20,15 +20,29 @@ * See the License for the specific language governing permissions and * limitations under the License. */ - -This stylesheet is used making an html version of hbase-default.xml. --> + +
+ + +comments powered by Disqus diff --git a/src/docbkx/developer.xml b/src/docbkx/developer.xml index 0c139b843ed0..854b6f6038fe 100644 --- a/src/docbkx/developer.xml +++ b/src/docbkx/developer.xml @@ -26,126 +26,266 @@ * limitations under the License. */ --> - Building and Developing HBase - This chapter will be of interest only to those building and developing HBase (i.e., as opposed to + Building and Developing Apache HBase (TM) + This chapter will be of interest only to those building and developing Apache HBase (TM) (i.e., as opposed to just downloading the latest distribution).
- HBase Repositories + Apache HBase Repositories + There are two different repositories for Apache HBase: Subversion (SVN) and Git. The former is the system of record for committers, but the latter is easier to work with to build and contribute. SVN updates get automatically propagated to the Git repo.
SVN -svn co http://svn.apache.org/repos/asf/hbase/trunk hbase-core-trunk +svn co http://svn.apache.org/repos/asf/hbase/trunk hbase-core-trunk -
+
Git git clone git://git.apache.org/hbase.git -
-
- -
+
+
+ +
IDEs
Eclipse
Code Formatting - See HBASE-3678 Add Eclipse-based Apache Formatter to HBase Wiki - for an Eclipse formatter to help ensure your code conforms to HBase'y coding convention. - The issue includes instructions for loading the attached formatter. + Under the dev-support folder, you will find hbase_eclipse_formatter.xml. + We encourage you to have this formatter in place in eclipse when editing HBase code. To load it into eclipse: + +Go to Eclipse->Preferences... +In Preferences, Go to Java->Code Style->Formatter +Import... hbase_eclipse_formatter.xml +Click Apply +Still in Preferences, Go to Java->Editor->Save Actions +Check the following: + +Perform the selected actions on save +Format source code +Format edited lines + + +Click Apply + + + In addition to the automatic formatting, make sure you follow the style guidelines explained in Also, no @author tags - that's a rule. Quality Javadoc comments are appreciated. And include the Apache license. -
+
Subversive Plugin Download and install the Subversive plugin. Set up an SVN Repository target from , then check out the code. -
+
+
+ Git Plugin + If you cloned the project via git, download and install the Git plugin (EGit). Attach to your local git repo (via the Git Repositories window) and you'll be able to see file revision history, generate patches, etc. +
- HBase Project Setup - To set up your Eclipse environment for HBase, close Eclipse and execute... - -mvn eclipse:eclipse - - ... from your local HBase project directory in your workspace to generate some new .project - and .classpathfiles. Then reopen Eclipse. -
-
- Maven Plugin - Download and install the Maven plugin. For example, Help -> Install New Software -> (search for Maven Plugin) -
+ HBase Project Setup in Eclipse + The easiest way is to use the m2eclipse plugin for Eclipse. Eclipse Indigo or newer has m2eclipse built-in, or it can be found here:http://www.eclipse.org/m2e/. M2Eclipse provides Maven integration for Eclipse - it even lets you use the direct Maven commands from within Eclipse to compile and test your project. + To import the project, you merely need to go to File->Import...Maven->Existing Maven Projects and then point Eclipse at the HBase root directory; m2eclipse will automatically find all the hbase modules for you. + If you install m2eclipse and import HBase in your workspace, you will have to fix your eclipse Build Path. + Remove target folder, add target/generated-jamon + and target/generated-sources/java folders. You may also remove from your Build Path + the exclusions on the src/main/resources and src/test/resources + to avoid error message in the console 'Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project hbase: + 'An Ant BuildException has occured: Replace: source file .../target/classes/hbase-default.xml doesn't exist'. This will also + reduce the eclipse build cycles and make your life easier when developing. +
+
+ Import into eclipse with the command line + For those not inclined to use m2eclipse, you can generate the Eclipse files from the command line. First, run (you should only have to do this once): + mvn clean install -DskipTests + and then close Eclipse and execute... + mvn eclipse:eclipse + ... from your local HBase project directory in your workspace to generate some new .project + and .classpathfiles. Then reopen Eclipse, or refresh your eclipse project (F5), and import + the .project file in the HBase directory to a workspace. + +
Maven Classpath Variable - The M2_REPO classpath variable needs to be set up for the project. This needs to be set to + The M2_REPO classpath variable needs to be set up for the project. This needs to be set to your local Maven repository, which is usually ~/.m2/repository If this classpath variable is not configured, you will see compile errors in Eclipse like this... Description Resource Path Location Type -The project cannot be built until build path errors are resolved hbase Unknown Java Problem +The project cannot be built until build path errors are resolved hbase Unknown Java Problem Unbound classpath variable: 'M2_REPO/asm/asm/3.1/asm-3.1.jar' in project 'hbase' hbase Build path Build Path Problem -Unbound classpath variable: 'M2_REPO/com/github/stephenc/high-scale-lib/high-scale-lib/1.1.1/high-scale-lib-1.1.1.jar' in project 'hbase' hbase Build path Build Path Problem +Unbound classpath variable: 'M2_REPO/com/github/stephenc/high-scale-lib/high-scale-lib/1.1.1/high-scale-lib-1.1.1.jar' in project 'hbase' hbase Build path Build Path Problem Unbound classpath variable: 'M2_REPO/com/google/guava/guava/r09/guava-r09.jar' in project 'hbase' hbase Build path Build Path Problem Unbound classpath variable: 'M2_REPO/com/google/protobuf/protobuf-java/2.3.0/protobuf-java-2.3.0.jar' in project 'hbase' hbase Build path Build Path Problem Unbound classpath variable: - +
-
- Import via m2eclipse - If you install the m2eclipse and import the HBase pom.xml in your workspace, you will have to fix your eclipse Build Path. - Remove target folder, add target/generated-jamon - and target/generated-sources/java folders. You may also remove from your Build Path - the exclusions on the src/main/resources and src/test/resources - to avoid error message in the console 'Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project hbase: - 'An Ant BuildException has occured: Replace: source file .../target/classes/hbase-default.xml doesn't exist'. This will also - reduce the eclipse build cycles and make your life easier when developing. -
Eclipse Known Issues Eclipse will currently complain about Bytes.java. It is not possible to turn these errors off. - + Description Resource Path Location Type Access restriction: The method arrayBaseOffset(Class) from the type Unsafe is not accessible due to restriction on required library /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar Bytes.java /hbase/src/main/java/org/apache/hadoop/hbase/util line 1061 Java Problem Access restriction: The method arrayIndexScale(Class) from the type Unsafe is not accessible due to restriction on required library /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar Bytes.java /hbase/src/main/java/org/apache/hadoop/hbase/util line 1064 Java Problem Access restriction: The method getLong(Object, long) from the type Unsafe is not accessible due to restriction on required library /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar Bytes.java /hbase/src/main/java/org/apache/hadoop/hbase/util line 1111 Java Problem - +
Eclipse - More Information - For additional information on setting up Eclipse for HBase development on Windows, see + For additional information on setting up Eclipse for HBase development on Windows, see Michael Morello's blog on the topic.
-
- + +
- Building HBase - This section will be of interest only to those building HBase from source. - + Building Apache HBase +
+ Basic Compile + Thanks to maven, building HBase is pretty easy. You can read about the various maven commands in , but the simplest command to compile HBase from its java source code is: + +mvn package -DskipTests + + Or, to clean up before compiling: + +mvn clean package -DskipTests + + With Eclipse set up as explained above in , you can also simply use the build command in Eclipse. To create the full installable HBase package takes a little bit more work, so read on. + +
Building in snappy compression support Pass -Dsnappy to trigger the snappy maven profile for building - snappy native libs into hbase. + snappy native libs into hbase. See also
Building the HBase tarball Do the following to build the HBase tarball. - Passing the -Drelease will generate javadoc and run the RAT plugin to verify licenses on source. - % MAVEN_OPTS="-Xmx2g" mvn clean site install assembly:single -Dmaven.test.skip -Prelease + Passing the -Prelease will generate javadoc and run the RAT plugin to verify licenses on source. + % MAVEN_OPTS="-Xmx2g" mvn clean site install assembly:assembly -DskipTests -Prelease
+ +
Build Gotchas + If you see Unable to find resource 'VM_global_library.vm', ignore it. + Its not an error. It is officially ugly though. + +
+
- Adding an HBase release to Apache's Maven Repository + Adding an Apache HBase release to Apache's Maven Repository Follow the instructions at - Publishing Maven Artifacts. - The 'trick' to making it all work is answering the questions put to you by the mvn release plugin properly, - making sure it is using the actual branch AND before doing the mvn release:perform step, - VERY IMPORTANT, hand edit the release.properties file that was put under ${HBASE_HOME} - by the previous step, release:perform. You need to edit it to make it point at - right locations in SVN. + Publishing Maven Artifacts after + reading the below miscellaney. + + You must use maven 3.0.x (Check by running mvn -version). + + Let me list out the commands I used first. The sections that follow dig in more + on what is going on. In this example, we are releasing the 0.92.2 jar to the apache + maven repository. + + # First make a copy of the tag we want to release; presumes the release has been tagged already + # We do this because we need to make some commits for the mvn release plugin to work. + 853 svn copy -m "Publishing 0.92.2 to mvn" https://svn.apache.org/repos/asf/hbase/tags/0.92.2 https://svn.apache.org/repos/asf/hbase/tags/0.92.2mvn + 857 svn checkout https://svn.apache.org/repos/asf/hbase/tags/0.92.2mvn + 858 cd 0.92.2mvn/ + # Edit the version making it release version with a '-SNAPSHOT' suffix (See below for more on this) + 860 vi pom.xml + 861 svn commit -m "Add SNAPSHOT to the version" pom.xml + 862 ~/bin/mvn/bin/mvn release:clean + 865 ~/bin/mvn/bin/mvn release:prepare + 866 # Answer questions and then ^C to kill the build after the last question. See below for more on this. + 867 vi release.properties + # Change the references to trunk svn to be 0.92.2mvn; the release plugin presumes trunk + # Then restart the release:prepare -- it won't ask questions + # because the properties file exists. + 868 ~/bin/mvn/bin/mvn release:prepare + # The apache-release profile comes from the apache parent pom and does signing of artifacts published + 869 ~/bin/mvn/bin/mvn release:perform -Papache-release + # When done copying up to apache staging repository, + # browse to repository.apache.org, login and finish + # the release as according to the above + # "Publishing Maven Artifacts. + + + Below is more detail on the commmands listed above. + At the mvn release:perform step, before starting, if you are for example + releasing hbase 0.92.2, you need to make sure the pom.xml version is 0.92.2-SNAPSHOT. This needs + to be checked in. Since we do the maven release after actual release, I've been doing this + checkin into a copy of the release tag rather than into the actual release tag itself (presumes the release has been properly tagged in svn). + So, say we released hbase 0.92.2 and now we want to do the release to the maven repository, in svn, the 0.92.2 + release will be tagged 0.92.2. Making the maven release, copy the 0.92.2 tag to 0.92.2mvn. + Check out this tag and change the version therein and commit. + + + Currently, the mvn release wants to go against trunk. I haven't figured how to tell it to do otherwise + so I do the below hack. The hack comprises answering the questions put to you by the mvn release plugin properly, + then immediately control-C'ing the build after the last question asked as the build release step starts to run. + After control-C'ing it, You'll notice a release.properties in your build dir. Review it. + Make sure it is using the proper branch -- it tends to use trunk rather than the 0.92.2mvn or whatever + that you want it to use -- so hand edit the release.properties file that was put under ${HBASE_HOME} + by the release:perform invocation. When done, resstart the + release:perform. + Here is how I'd answer the questions at release:prepare time: + What is the release version for "HBase"? (org.apache.hbase:hbase) 0.92.2: : +What is SCM release tag or label for "HBase"? (org.apache.hbase:hbase) hbase-0.92.2: : 0.92.2mvn +What is the new development version for "HBase"? (org.apache.hbase:hbase) 0.92.3-SNAPSHOT: : +[INFO] Transforming 'HBase'... + + When you run release:perform, pass -Papache-release + else it will not 'sign' the artifacts it uploads. + + A strange issue I ran into was the one where the upload into the apache + repository was being sprayed across multiple apache machines making it so I could + not release. See INFRA-4482 Why is my upload to mvn spread across multiple repositories?. + + Here is my ~/.m2/settings.xml. + This is read by the release plugin. The apache-release profile will pick up your + gpg key setup from here if you've specified it into the file. The password + can be maven encrypted as suggested in the "Publishing Maven Artifacts" but plain + text password works too (just don't let anyone see your local settings.xml). + <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 + http://maven.apache.org/xsd/settings-1.0.0.xsd"> + <servers> + <!- To publish a snapshot of some part of Maven --> + <server> + <id>apache.snapshots.https</id> + <username>YOUR_APACHE_ID + </username> + <password>YOUR_APACHE_PASSWORD + </password> + </server> + <!-- To publish a website using Maven --> + <!-- To stage a release of some part of Maven --> + <server> + <id>apache.releases.https</id> + <username>YOUR_APACHE_ID + </username> + <password>YOUR_APACHE_PASSWORD + </password> + </server> + </servers> + <profiles> + <profile> + <id>apache-release</id> + <properties> + <gpg.keyname>YOUR_KEYNAME</gpg.keyname> + <!--Keyname is something like this ... 00A5F21E... do gpg --list-keys to find it--> + <gpg.passphrase>YOUR_KEY_PASSWORD + </gpg.passphrase> + </properties> + </profile> + </profiles> +</settings> + + + If you see run into the below, its because you need to edit version in the pom.xml and add -SNAPSHOT to the version (and commit). [INFO] Scanning for projects... @@ -168,73 +308,163 @@ Access restriction: The method getLong(Object, long) from the type Unsafe is not [INFO] -----------------------------------------------------------------------
-
Build Gotchas - If you see Unable to find resource 'VM_global_library.vm', ignore it. - Its not an error. It is officially ugly though. - +
+ Generating the HBase Reference Guide + The manual is marked up using docbook. + We then use the docbkx maven plugin + to transform the markup to html. This plugin is run when you specify the site + goal as in when you run mvn site or you can call the plugin explicitly to + just generate the manual by doing mvn docbkx:generate-html + (TODO: It looks like you have to run mvn site first because docbkx wants to + include a transformed hbase-default.xml. Fix). + When you run mvn site, we do the document generation twice, once to generate the multipage + manual and then again for the single page manual (the single page version is easier to search). +
-
- +
+ Updating hbase.apache.org +
+ Contributing to hbase.apache.org + The Apache HBase apache web site (including this reference guide) is maintained as part of the main Apache HBase source tree, under /src/docbkx and /src/site. The former is this reference guide; the latter, in most cases, are legacy pages that are in the process of being merged into the docbkx tree. + To contribute to the reference guide, edit these files and submit them as a patch (see ). Your Jira should contain a summary of the changes in each section (see HBASE-6081 for an example). + To generate the site locally while you're working on it, run: + mvn site + Then you can load up the generated HTML files in your browser (file are under /target/site). +
+
+ Publishing hbase.apache.org + As of INFRA-5680 Migrate apache hbase website, + to publish the website, build it, and then deploy it over a checkout of https://svn.apache.org/repos/asf/hbase/hbase.apache.org/trunk, + and then check it in. For example, if trunk is checked out out at /Users/stack/checkouts/trunk + and hbase.apache.org is checked out at /Users/stack/checkouts/hbase.apache.org/trunk, to update + the site, do the following: + + # Build the site and deploy it to the checked out directory + # Getting the javadoc into site is a little tricky. You have to build it independent, then + # 'aggregate' it at top-level so the pre-site site lifecycle step can find it; that is + # what the javadoc:javadoc and javadoc:aggregate is about. + $ MAVEN_OPTS=" -Xmx3g" mvn clean -DskipTests javadoc:javadoc javadoc:aggregate site site:stage -DstagingDirectory=/Users/stack/checkouts/hbase.apache.org/trunk + # Check the deployed site by viewing in a brower. + # If all is good, commit it and it will show up at http://hbase.apache.org + # + $ cd /Users/stack/checkouts/hbase.apache.org/trunk + $ svn commit -m 'Committing latest version of website...' + + +
+
Tests -HBase tests are divided into two groups: and -. -Unit tests are run by the Apache Continuous Integration server and by developers -when they are verifying a fix does not cause breakage elsewhere in the code base. -Integration tests are generally long-running tests that are invoked out-of-bound of -the CI server when you want to do more intensive testing beyond the unit test set. -Integration tests, for example, are run proving a release candidate or a production -deploy. Below we go into more detail on each of these test types. Developers at a -minimum should familiarize themselves with the unit test detail; unit tests in -HBase have a character not usually seen in other projects. + Developers, at a minimum, should familiarize themselves with the unit test detail; unit tests in +HBase have a character not usually seen in other projects. + +
+Apache HBase Modules +As of 0.96, Apache HBase is split into multiple modules which creates "interesting" rules for +how and where tests are written. If you are writting code for hbase-server, see + for how to write your tests; these tests can spin +up a minicluster and will need to be categorized. For any other module, for example +hbase-common, the tests must be strict unit tests and just test the class +under test - no use of the HBaseTestingUtility or minicluster is allowed (or even possible +given the dependency tree). +
+ Running Tests in other Modules + If the module you are developing in has no other dependencies on other HBase modules, then + you can cd into that module and just run: + mvn test + which will just run the tests IN THAT MODULE. If there are other dependencies on other modules, + then you will have run the command from the ROOT HBASE DIRECTORY. This will run the tests in the other + modules, unless you specify to skip the tests in that module. For instance, to skip the tests in the hbase-server module, + you would run: + mvn clean test -PskipServerTests + from the top level directory to run all the tests in modules other than hbase-server. Note that you + can specify to skip tests in multiple modules as well as just for a single module. For example, to skip + the tests in hbase-server and hbase-common, you would run: + mvn clean test -PskipServerTests -PskipCommonTests + Also, keep in mind that if you are running tests in the hbase-server module you will need to + apply the maven profiles discussed in to get the tests to run properly. +
+
Unit Tests -HBase unit tests are subdivided into three categories: small, medium and large, with -corresponding JUnit categories: +Apache HBase unit tests are subdivided into four categories: small, medium, large, and +integration with corresponding JUnit categories: SmallTests, MediumTests, -LargeTests. JUnit categories are denoted using java annotations -and look like this in your unit test code. +LargeTests, IntegrationTests. +JUnit categories are denoted using java annotations and look like this in your unit test code. ... @Category(SmallTests.class) public class TestHRegionInfo { - @Test public void testCreateHRegionInfoName() throws Exception { // ... } } -The above example shows how to mark a test as belonging to the small category. +The above example shows how to mark a unit test as belonging to the small category. +All unit tests in HBase have a categorization. +The first three categories, small, medium, and large are for tests run when +you type $ mvn test; i.e. these three categorizations are for +HBase unit tests. The integration category is for not for unit tests but for integration +tests. These are run when you invoke $ mvn verify. Integration tests +are described in integration tests section and will not be discussed further +in this section on HBase unit tests. + +Apache HBase uses a patched maven surefire plugin and maven profiles to implement +its unit test characterizations. + +Read the below to figure which annotation of the set small, medium, and large to +put on your new HBase unit test. + + +
+Small Tests<indexterm><primary>SmallTests</primary></indexterm> + Small tests are executed in a shared JVM. We put in this category all the tests that can -be executed quickly in a shared JVM. The maximum execution time for a test is 15 seconds, -and they do not use a cluster. Medium tests represent tests that must be executed +be executed quickly in a shared JVM. The maximum execution time for a small test is 15 seconds, +and small tests should not use a (mini)cluster. +
+ +
+Medium Tests<indexterm><primary>MediumTests</primary></indexterm> +Medium tests represent tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They are designed to last less than 50 seconds individually. They can use a cluster, and each of them is executed in a separate JVM. -Large tests are everything else. They are typically integration-like -tests (yes, some large tests should be moved out to be HBase ), -regression tests for specific bugs, timeout tests, performance tests. + +
+ +
+Large Tests<indexterm><primary>LargeTests</primary></indexterm> +Large tests are everything else. They are typically large-scale +tests, regression tests for specific bugs, timeout tests, performance tests. They are executed before a commit on the pre-integration machines. They can be run on the developer machine as well. -HBase uses a patched maven surefire plugin and maven profiles to implement its -unit test characterizations. +
+
+Integration Tests<indexterm><primary>IntegrationTests</primary></indexterm> +Integration tests are system level tests. See +integration tests section for more info. + +
+
Running tests -Below we describe how to run the HBase junit categories. +Below we describe how to run the Apache HBase junit categories.
Default: small and medium category tests -Running mvn test will execute all small tests in a single JVM and medium tests in a separate JVM for -each test instance. Medium tests are NOT executed if there is an error in a small test. +Running mvn test will execute all small tests in a single JVM +(no fork) and then medium tests in a separate JVM for each test instance. +Medium tests are NOT executed if there is an error in a small test. Large tests are NOT executed. There is one report for small tests, and one report for -medium tests if they are executed. To run small and medium tests with the security -profile enabled, do mvn test -P security +medium tests if they are executed.
@@ -244,42 +474,69 @@ profile enabled, do mvn test -P security will execute small tests in a single JVM then medium and large tests in a separate JVM for each test. Medium and large tests are NOT executed if there is an error in a small test. Large tests are NOT executed if there is an error in a small or medium test. -There is one report for small tests, and one report for medium and large tests if they are executed +There is one report for small tests, and one report for medium and large tests if they are executed.
Running a single test or all tests in a package To run an individual test, e.g. MyTest, do -mvn test -P localTests -Dtest=MyTest You can also +mvn test -Dtest=MyTest You can also pass multiple, individual tests as a comma-delimited list: -mvn test -P localTests -Dtest=MyTest1,MyTest2,MyTest3 +mvn test -Dtest=MyTest1,MyTest2,MyTest3 You can also pass a package, which will run all tests under the package: -mvn test -P localTests -Dtest=org.apache.hadoop.hbase.client.* -To run a single test with the security profile enabled: -mvn test -P security,localTests -Dtest=TestGet +mvn test -Dtest=org.apache.hadoop.hbase.client.* -The -P localTests will remove the JUnit category effect (without this specific profile, -the profiles are taken into account). It will actually use the official release of surefire -and the old connector (The HBase build uses a patched version of the maven surefire plugin). -junit tests are executed in separated JVM. You will see a new message at the end of the -report: "[INFO] Tests are skipped". It's harmless. +When -Dtest is specified, localTests profile will be used. It will use the official release +of maven surefire, rather than our custom surefire plugin, and the old connector (The HBase build uses a patched +version of the maven surefire plugin). Each junit tests is executed in a separate JVM (A fork per test class). +There is no parallelization when tests are running in this mode. You will see a new message at the end of the +-report: "[INFO] Tests are skipped". It's harmless. While you need to make sure the sum of Tests run: in +the Results : section of test reports matching the number of tests you specified because no +error will be reported when a non-existent test case is specified.
Other test invocation permutations -Running mvn test -P runSmallTests will execute small tests only, in a single JVM. +Running mvn test -P runSmallTests will execute "small" tests only, using a single JVM. + +Running mvn test -P runMediumTests will execute "medium" tests only, launching a new JVM for each test-class. + +Running mvn test -P runLargeTests will execute "large" tests only, launching a new JVM for each test-class. -Running mvn test -P runMediumTests will execute medium tests in a single JVM. +For convenience, you can run mvn test -P runDevTests to execute both small and medium tests, using a single JVM. -Running mvn test -P runLargeTests execute medium tests in a single JVM. +
+ +
+Running tests faster + +By default, $ mvn test -P runAllTests runs 5 tests in parallel. +It can be increased on a developer's machine. Allowing that you can have 2 +tests in parallel per core, and you need about 2Gb of memory per test (at the +extreme), if you have an 8 core, 24Gb box, you can have 16 tests in parallel. +but the memory available limits it to 12 (24/2), To run all tests with 12 tests +in parallell, do this: +mvn test -P runAllTests -Dsurefire.secondPartThreadCount=12. +To increase the speed, you can as well use a ramdisk. You will need 2Gb of memory +to run all tests. You will also need to delete the files between two test run. +The typical way to configure a ramdisk on Linux is: +$ sudo mkdir /ram2G +sudo mount -t tmpfs -o size=2048M tmpfs /ram2G +You can then use it to run all HBase tests with the command: +mvn test -P runAllTests -Dsurefire.secondPartThreadCount=12 -Dtest.build.data.basedirectory=/ram2G +
+
+<command>hbasetests.sh</command> It's also possible to use the script hbasetests.sh. This script runs the medium and -large tests in parallel with two maven instances, and provide a single report. +large tests in parallel with two maven instances, and provides a single report. This script does not use +the hbase version of surefire so no parallelization is being done other than the two maven instances the +script sets up. It must be executed from the directory which contains the pom.xml. For example running ./dev-support/hbasetests.sh will execute small and medium tests. @@ -288,6 +545,26 @@ Running ./dev-support/hbasetests.sh replayFailed
+
+Test Resource Checker<indexterm><primary>Test Resource Checker</primary></indexterm> + +A custom Maven SureFire plugin listener checks a number of resources before +and after each HBase unit test runs and logs its findings at the end of the test +output files which can be found in target/surefire-reports +per Maven module (Tests write test reports named for the test class into this directory. +Check the *-out.txt files). The resources counted are the number +of threads, the number of file descriptors, etc. If the number has increased, it adds +a LEAK? comment in the logs. As you can have an HBase instance +running in the background, some threads can be deleted/created without any specific +action in the test. However, if the test does not work as expected, or if the test +should not impact these resources, it's worth checking these log lines +...hbase.ResourceChecker(157): before... and +...hbase.ResourceChecker(157): after.... For example: + +2012-09-26 09:22:15,315 INFO [pool-1-thread-1] hbase.ResourceChecker(157): after: regionserver.TestColumnSeeking#testReseeking Thread=65 (was 65), OpenFileDescriptor=107 (was 107), MaxFileDescriptor=10240 (was 10240), ConnectionCount=1 (was 1) + + +
@@ -307,8 +584,12 @@ Tests should not overlog. More than 100 lines/second makes the logs complex to r Tests can be written with HBaseTestingUtility. This class offers helper functions to create a temp directory and do the cleanup, or to start a cluster. -Categories and execution time + +
+
+Categories and execution time + All tests must be categorized, if not they could be skipped. @@ -345,30 +626,60 @@ As most as possible, tests should use the default settings for the cluster. When
-
Integration Tests -HBase integration Tests are tests that are beyond HBase unit tests. They +HBase integration/system tests are tests that are beyond HBase unit tests. They are generally long-lasting, sizeable (the test can be asked to 1M rows or 1B rows), targetable (they can take configuration that will point them at the ready-made cluster they are to run against; integration tests do not include cluster start/stop code), and verifying success, integration tests rely on public APIs only; they do not -attempt to examine server internals asserring success/fail. Integration tests +attempt to examine server internals asserting success/fail. Integration tests are what you would run when you need to more elaborate proofing of a release candidate beyond what unit tests can do. They are not generally run on the Apache Continuous Integration -build server. +build server, however, some sites opt to run integration tests as a part of their +continuous testing on an actual cluster. -Integration tests currently live under the src/test directory and -will match the regex: **/IntegrationTest*.java. +Integration tests currently live under the src/test directory +in the hbase-it submodule and will match the regex: **/IntegrationTest*.java. +All integration tests are also annotated with @Category(IntegrationTests.class). + + +Integration tests can be run in two modes: using a mini cluster, or against an actual distributed cluster. +Maven failsafe is used to run the tests using the mini cluster. IntegrationTestsDriver class is used for +executing the tests against a distributed cluster. Integration tests SHOULD NOT assume that they are running against a +mini cluster, and SHOULD NOT use private API's to access cluster state. To interact with the distributed or mini +cluster uniformly, IntegrationTestingUtility, and HBaseCluster classes, +and public client API's can be used. + + + +On a distributed cluster, integration tests that use ChaosMonkey or otherwise manipulate services thru cluster manager (e.g. restart regionservers) use SSH to do it. +To run these, test process should be able to run commands on remote end, so ssh should be configured accordingly (for example, if HBase runs under hbase +user in your cluster, you can set up passwordless ssh for that user and run the test also under it). To facilitate that, hbase.it.clustermanager.ssh.user, +hbase.it.clustermanager.ssh.opts and hbase.it.clustermanager.ssh.cmd configuration settings can be used. "User" is the remote user that cluster manager should use to perform ssh commands. +"Opts" contains additional options that are passed to SSH (for example, "-i /tmp/my-key"). +Finally, if you have some custom environment setup, "cmd" is the override format for the entire tunnel (ssh) command. The default string is {/usr/bin/ssh %1$s %2$s%3$s%4$s "%5$s"} and is a good starting point. This is a standard Java format string with 5 arguments that is used to execute the remote command. The argument 1 (%1$s) is SSH options set the via opts setting or via environment variable, 2 is SSH user name, 3 is "@" if username is set or "" otherwise, 4 is the target host name, and 5 is the logical command to execute (that may include single quotes, so don't use them). For example, if you run the tests under non-hbase user and want to ssh as that user and change to hbase on remote machine, you can use {/usr/bin/ssh %1$s %2$s%3$s%4$s "su hbase - -c \"%5$s\""}. That way, to kill RS (for example) integration tests may run {/usr/bin/ssh some-hostname "su hbase - -c \"ps aux | ... | kill ...\""}. +The command is logged in the test logs, so you can verify it is correct for your environment. + + +
+Running integration tests against mini cluster HBase 0.92 added a verify maven target. Invoking it, for example by doing mvn verify, will run all the phases up to and including the verify phase via the maven failsafe plugin, running all the above mentioned HBase unit tests as well as tests that are in the HBase integration test group. -If you just want to run the integration tests, you need to run two commands. First: +After you have completed + mvn install -DskipTests +You can run just the integration tests by invoking: + +cd hbase-it +mvn verify + +If you just want to run the integration tests in top-level, you need to run two commands. First: mvn failsafe:integration-test This actually runs ALL the integration tests. This command will always output BUILD SUCCESS even if there are test failures. @@ -379,75 +690,170 @@ This actually runs ALL the integration tests.
Running a subset of Integration tests - This is very similar to how you specify running a subset of unit tests (see above). + This is very similar to how you specify running a subset of unit tests (see above), but use the property + it.test instead of test. To just run IntegrationTestClassXYZ.java, use: - mvn failsafe:integration-test -Dtest=IntegrationTestClassXYZ - Pretty similar, right? + mvn failsafe:integration-test -Dit.test=IntegrationTestClassXYZ The next thing you might want to do is run groups of integration tests, say all integration tests that are named IntegrationTestClassX*.java: - mvn failsafe:integration-test -Dtest=*ClassX* + mvn failsafe:integration-test -Dit.test=*ClassX* This runs everything that is an integration test that matches *ClassX*. This means anything matching: "**/IntegrationTest*ClassX*". You can also run multiple groups of integration tests using comma-delimited lists (similar to unit tests). Using a list of matches still supports full regex matching for each of the groups.This would look something like: - mvn failsafe:integration-test -Dtest=*ClassX*, *ClassY + mvn failsafe:integration-test -Dit.test=*ClassX*, *ClassY
-
+
+
+Running integration tests against distributed cluster + +If you have an already-setup HBase cluster, you can launch the integration tests by invoking the class IntegrationTestsDriver. You may have to +run test-compile first. The configuration will be picked by the bin/hbase script. +mvn test-compile +Then launch the tests with: +bin/hbase [--config config_dir] org.apache.hadoop.hbase.IntegrationTestsDriver [-test=class_regex] + +This execution will launch the tests under hbase-it/src/test, having @Category(IntegrationTests.class) annotation, +and a name starting with IntegrationTests. If specified, class_regex will be used to filter test classes. The regex is checked against full class name; so, part of class name can be used. +IntegrationTestsDriver uses Junit to run the tests. Currently there is no support for running integration tests against a distributed cluster using maven (see HBASE-6201). + + + +The tests interact with the distributed cluster by using the methods in the DistributedHBaseCluster (implementing HBaseCluster) class, which in turn uses a pluggable ClusterManager. Concrete implementations provide actual functionality for carrying out deployment-specific and environment-dependent tasks (SSH, etc). The default ClusterManager is HBaseClusterManager, which uses SSH to remotely execute start/stop/kill/signal commands, and assumes some posix commands (ps, etc). Also assumes the user running the test has enough "power" to start/stop servers on the remote machines. By default, it picks up HBASE_SSH_OPTS, HBASE_HOME, HBASE_CONF_DIR from the env, and uses bin/hbase-daemon.sh to carry out the actions. Currently tarball deployments, deployments which uses hbase-daemons.sh, and Apache Ambari deployments are supported. /etc/init.d/ scripts are not supported for now, but it can be easily added. For other deployment options, a ClusterManager can be implemented and plugged in. + +
+ +
+Destructive integration / system tests + + In 0.96, a tool named ChaosMonkey has been introduced. It is modeled after the same-named tool by Netflix. +Some of the tests use ChaosMonkey to simulate faults in the running cluster in the way of killing random servers, +disconnecting servers, etc. ChaosMonkey can also be used as a stand-alone tool to run a (misbehaving) policy while you +are running other tests. + + + +ChaosMonkey defines Action's and Policy's. Actions are sequences of events. We have at least the following actions: + +Restart active master (sleep 5 sec) +Restart random regionserver (sleep 5 sec) +Restart random regionserver (sleep 60 sec) +Restart META regionserver (sleep 5 sec) +Restart ROOT regionserver (sleep 5 sec) +Batch restart of 50% of regionservers (sleep 5 sec) +Rolling restart of 100% of regionservers (sleep 5 sec) + + +Policies on the other hand are responsible for executing the actions based on a strategy. +The default policy is to execute a random action every minute based on predefined action +weights. ChaosMonkey executes predefined named policies until it is stopped. More than one +policy can be active at any time. + + + + To run ChaosMonkey as a standalone tool deploy your HBase cluster as usual. ChaosMonkey uses the configuration +from the bin/hbase script, thus no extra configuration needs to be done. You can invoke the ChaosMonkey by running: +bin/hbase org.apache.hadoop.hbase.util.ChaosMonkey + +This will output smt like: + +12/11/19 23:21:57 INFO util.ChaosMonkey: Using ChaosMonkey Policy: class org.apache.hadoop.hbase.util.ChaosMonkey$PeriodicRandomActionPolicy, period:60000 +12/11/19 23:21:57 INFO util.ChaosMonkey: Sleeping for 26953 to add jitter +12/11/19 23:22:24 INFO util.ChaosMonkey: Performing action: Restart active master +12/11/19 23:22:24 INFO util.ChaosMonkey: Killing master:master.example.com,60000,1353367210440 +12/11/19 23:22:24 INFO hbase.HBaseCluster: Aborting Master: master.example.com,60000,1353367210440 +12/11/19 23:22:24 INFO hbase.ClusterManager: Executing remote command: ps aux | grep master | grep -v grep | tr -s ' ' | cut -d ' ' -f2 | xargs kill -s SIGKILL , hostname:master.example.com +12/11/19 23:22:25 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output: +12/11/19 23:22:25 INFO hbase.HBaseCluster: Waiting service:master to stop: master.example.com,60000,1353367210440 +12/11/19 23:22:25 INFO hbase.ClusterManager: Executing remote command: ps aux | grep master | grep -v grep | tr -s ' ' | cut -d ' ' -f2 , hostname:master.example.com +12/11/19 23:22:25 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output: +12/11/19 23:22:25 INFO util.ChaosMonkey: Killed master server:master.example.com,60000,1353367210440 +12/11/19 23:22:25 INFO util.ChaosMonkey: Sleeping for:5000 +12/11/19 23:22:30 INFO util.ChaosMonkey: Starting master:master.example.com +12/11/19 23:22:30 INFO hbase.HBaseCluster: Starting Master on: master.example.com +12/11/19 23:22:30 INFO hbase.ClusterManager: Executing remote command: /homes/enis/code/hbase-0.94/bin/../bin/hbase-daemon.sh --config /homes/enis/code/hbase-0.94/bin/../conf start master , hostname:master.example.com +12/11/19 23:22:31 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output:starting master, logging to /homes/enis/code/hbase-0.94/bin/../logs/hbase-enis-master-master.example.com.out +.... +12/11/19 23:22:33 INFO util.ChaosMonkey: Started master: master.example.com,60000,1353367210440 +12/11/19 23:22:33 INFO util.ChaosMonkey: Sleeping for:51321 +12/11/19 23:23:24 INFO util.ChaosMonkey: Performing action: Restart random region server +12/11/19 23:23:24 INFO util.ChaosMonkey: Killing region server:rs3.example.com,60020,1353367027826 +12/11/19 23:23:24 INFO hbase.HBaseCluster: Aborting RS: rs3.example.com,60020,1353367027826 +12/11/19 23:23:24 INFO hbase.ClusterManager: Executing remote command: ps aux | grep regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2 | xargs kill -s SIGKILL , hostname:rs3.example.com +12/11/19 23:23:25 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output: +12/11/19 23:23:25 INFO hbase.HBaseCluster: Waiting service:regionserver to stop: rs3.example.com,60020,1353367027826 +12/11/19 23:23:25 INFO hbase.ClusterManager: Executing remote command: ps aux | grep regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2 , hostname:rs3.example.com +12/11/19 23:23:25 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output: +12/11/19 23:23:25 INFO util.ChaosMonkey: Killed region server:rs3.example.com,60020,1353367027826. Reported num of rs:6 +12/11/19 23:23:25 INFO util.ChaosMonkey: Sleeping for:60000 +12/11/19 23:24:25 INFO util.ChaosMonkey: Starting region server:rs3.example.com +12/11/19 23:24:25 INFO hbase.HBaseCluster: Starting RS on: rs3.example.com +12/11/19 23:24:25 INFO hbase.ClusterManager: Executing remote command: /homes/enis/code/hbase-0.94/bin/../bin/hbase-daemon.sh --config /homes/enis/code/hbase-0.94/bin/../conf start regionserver , hostname:rs3.example.com +12/11/19 23:24:26 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output:starting regionserver, logging to /homes/enis/code/hbase-0.94/bin/../logs/hbase-enis-regionserver-rs3.example.com.out + +12/11/19 23:24:27 INFO util.ChaosMonkey: Started region server:rs3.example.com,60020,1353367027826. Reported num of rs:6 + + +As you can see from the log, ChaosMonkey started the default PeriodicRandomActionPolicy, which is configured with all the available actions, and ran RestartActiveMaster and RestartRandomRs actions. ChaosMonkey tool, if run from command line, will keep on running until the process is killed. + +
+ - -
+ +
Maven Build Commands All commands executed from the local HBase project directory. Note: use Maven 3 (Maven 2 may work but we suggest you use Maven 3). -
+
Compile mvn compile -
+
-
+
Running all or individual Unit Tests See the section above in -
- -
- Running all or individual Integration Tests - See - -
+
-
- To build against hadoop 0.22.x or 0.23.x - -mvn -Dhadoop.profile=22 ... - -That is, designate build with hadoop.profile 22. Pass 23 for hadoop.profile to build against hadoop 0.23. -Tests do not all pass as of this writing so you may need ot pass -DskipTests unless you are inclined -to fix the failing tests. - +
+ Building against various hadoop versions. + As of 0.96, Apache HBase supports building against Apache Hadoop versions: 1.0.3, 2.0.0-alpha and 3.0.0-SNAPSHOT. + By default, we will build with Hadoop-1.0.3. To change the version to run with Hadoop-2.0.0-alpha, you would run: + mvn -Dhadoop.profile=2.0 ... + + That is, designate build with hadoop.profile 2.0. Pass 2.0 for hadoop.profile to build against hadoop 2.0. + Tests may not all pass as of this writing so you may need to pass -DskipTests unless you are inclined + to fix the failing tests. + + Similarly, for 3.0, you would just replace the profile value. Note that Hadoop-3.0.0-SNAPSHOT does not currently have a deployed maven artificat - you will need to build and install your own in your local maven repository if you want to run against this profile. + + + In earilier verions of Apache HBase, you can build against older versions of Apache Hadoop, notably, Hadoop 0.22.x and 0.23.x. + If you are running, for example HBase-0.94 and wanted to build against Hadoop 0.23.x, you would run with: + mvn -Dhadoop.profile=22 ...
- -
+ +
Getting Involved - HBase gets better only when people contribute! + Apache HBase gets better only when people contribute! - As HBase is an Apache Software Foundation project, see for more information about how the ASF functions. + As Apache HBase is an Apache Software Foundation project, see for more information about how the ASF functions.
Mailing Lists - Sign up for the dev-list and the user-list. See the + Sign up for the dev-list and the user-list. See the mailing lists page. - Posing questions - and helping to answer other people's questions - is encouraged! - There are varying levels of experience on both lists so patience and politeness are encouraged (and please - stay on topic.) + Posing questions - and helping to answer other people's questions - is encouraged! + There are varying levels of experience on both lists so patience and politeness are encouraged (and please + stay on topic.)
Jira - Check for existing issues in Jira. + Check for existing issues in Jira. If it's either a new feature request, enhancement, or a bug, file a ticket.
Jira Priorities @@ -457,10 +863,10 @@ to fix the failing tests. Critical: The issue described can cause data loss or cluster instability in some cases. Major: Important but not tragic issues, like updates to the client API that will add a lot of much-needed functionality or significant bugs that need to be fixed but that don't cause data loss. - Minor: Useful enhancements and annoying but not damaging bugs. - Trivial: Useful enhancements but generally cosmetic. - - + Minor: Useful enhancements and annoying but not damaging bugs. + Trivial: Useful enhancements but generally cosmetic. + +
Code Blocks in Jira Comments @@ -475,15 +881,15 @@ to fix the failing tests.
- +
Developing
Codelines Most development is done on TRUNK. However, there are branches for minor releases (e.g., 0.90.1, 0.90.2, and 0.90.3 are on the 0.90 branch). If you have any questions on this just send an email to the dev dist-list.
- -
+ +
Unit Tests In HBase we use JUnit 4. If you need to run miniclusters of HDFS, ZooKeeper, HBase, or MapReduce testing, @@ -506,30 +912,82 @@ to fix the failing tests.
+
+
Code Standards See and . -
-
+ Also, please pay attention to the interface stability/audience classifications that you + will see all over our code base. They look like this at the head of the class: + @InterfaceAudience.Public +@InterfaceStability.Stable + + If the InterfaceAudience is Private, + we can change the class (and we do not need to include a InterfaceStability mark). + If a class is marked Public but its InterfaceStability + is marked Unstable, we can change it. If it's + marked Public/Evolving, we're allowed to change it + but should try not to. If it's Public and Stable + we can't change it without a deprecation path or with a really GREAT reason. + When you add new classes, mark them with the annotations above if publically accessible. + If you are not cleared on how to mark your additions, ask up on the dev list. + + This convention comes from our parent project Hadoop. +
+ +
+ Invariants + We don't have many but what we have we list below. All are subject to challenge of + course but until then, please hold to the rules of the road. + +
+ No permanent state in ZooKeeper + ZooKeeper state should transient (treat it like memory). If deleted, hbase + should be able to recover and essentially be in the same stateThere are currently + a few exceptions that we need to fix around whether a table is enabled or disabled. + +
+ +
+ +
+ Running In-Situ + If you are developing Apache HBase, frequently it is useful to test your changes against a more-real cluster than what you find in unit tests. In this case, HBase can be run directly from the source in local-mode. + All you need to do is run: + + ${HBASE_HOME}/bin/start-hbase.sh + + This will spin up a full local-cluster, just as if you had packaged up HBase and installed it on your machine. + + Keep in mind that you will need to have installed HBase into your local maven repository for the in-situ cluster to work properly. That is, you will need to run: + mvn clean install -DskipTests + to ensure that maven can find the correct classpath and dependencies. Generally, the above command + is just a good thing to try running first, if maven is acting oddly. +
Submitting Patches + If you are new to submitting patches to open source or new to submitting patches to Apache, + I'd suggest you start by reading the On Contributing Patches + page from Apache Commons Project. Its a nice overview that + applies equally to the Apache HBase Project.
Create Patch - Patch files can be easily generated from Eclipse, for example by selecting "Team -> Create Patch". + See the aforementioned Apache Commons link for how to make patches against a checked out subversion + repository. Patch files can also be easily generated from Eclipse, for example by selecting "Team -> Create Patch". Patches can also be created by git diff and svn diff. - Please submit one patch-file per Jira. For example, if multiple files are changed make sure the + Please submit one patch-file per Jira. For example, if multiple files are changed make sure the selected resource when generating the patch is a directory. Patch files can reflect changes in multiple files. Make sure you review for code style.
Patch File Naming - The patch file should have the HBase Jira ticket in the name. For example, if a patch was submitted for Foo.java, then - a patch file called Foo_HBASE_XXXX.patch would be acceptable where XXXX is the HBase Jira number. + The patch file should have the Apache HBase Jira ticket in the name. For example, if a patch was submitted for Foo.java, then + a patch file called Foo_HBASE_XXXX.patch would be acceptable where XXXX is the Apache HBase Jira number. If you generating from a branch, then including the target branch in the filename is advised, e.g., HBASE-XXXX-0.90.patch. @@ -539,26 +997,30 @@ to fix the failing tests. Yes, please. Please try to include unit tests with every code patch (and especially new classes and large changes). Make sure unit tests pass locally before submitting the patch. Also, see . + If you are creating a new unit test class, notice how other unit test classes have classification/sizing + annotations at the top and a static method on the end. Be sure to include these in any new unit test files + you generate. See for more on how the annotations work. +
Attach Patch to Jira The patch should be attached to the associated Jira ticket "More Actions -> Attach Files". Make sure you click the ASF license inclusion, otherwise the patch can't be considered for inclusion. - Once attached to the ticket, click "Submit Patch" and + Once attached to the ticket, click "Submit Patch" and the status of the ticket will change. Committers will review submitted patches for inclusion into the codebase. Please understand that not every patch may get committed, and that feedback will likely be provided on the patch. Fear not, though, - because the HBase community is helpful! + because the Apache HBase community is helpful!
- +
Common Patch Feedback The following items are representative of common patch feedback. Your patch process will go faster if these are taken into account before submission. - See the Java coding standards + See the Java coding standards for more information on coding conventions in Java.
@@ -567,7 +1029,7 @@ to fix the failing tests. if ( foo.equals( bar ) ) { // don't do this - ... do this instead... + ... do this instead... if (foo.equals(bar)) { @@ -576,9 +1038,9 @@ if (foo.equals(bar)) { foo = barArray[ i ]; // don't do this - ... do this instead... + ... do this instead... -foo = barArray[i]; +foo = barArray[i];
@@ -589,7 +1051,7 @@ foo = barArray[i]; public void readFields(DataInput arg0) throws IOException { // don't do this foo = arg0.readUTF(); // don't do this - ... do this instead ... + ... do this instead ... public void readFields(DataInput di) throws IOException { foo = di.readUTF(); @@ -600,19 +1062,14 @@ foo = barArray[i];
Long Lines - Keep lines less than 80 characters. + Keep lines less than 100 characters. -Bar bar = foo.veryLongMethodWithManyArguments(argument1, argument2, argument3, argument4, argument5); // don't do this +Bar bar = foo.veryLongMethodWithManyArguments(argument1, argument2, argument3, argument4, argument5, argument6, argument7, argument8, argument9); // don't do this - ... do this instead ... - -Bar bar = foo.veryLongMethodWithManyArguments(argument1, - argument2, argument3,argument4, argument5); - - ... or this, whichever looks better ... + ... do something like this instead ... Bar bar = foo.veryLongMethodWithManyArguments( - argument1, argument2, argument3,argument4, argument5); + argument1, argument2, argument3,argument4, argument5, argument6, argument7, argument8, argument9);
@@ -624,11 +1081,17 @@ Bar bar = foo.veryLongMethodWithManyArguments( Bar bar = foo.getBar(); <--- imagine there's an extra space(s) after the semicolon instead of a line break. Make sure there's a line-break after the end of your code, and also avoid lines that have nothing - but whitespace. + but whitespace. -
+
Implementing Writable + + Applies pre-0.96 only + In 0.96, HBase moved to protobufs. The below section on Writables + applies to 0.94.x and previous, not to 0.96 and beyond. + + Every class returned by RegionServers must implement Writable. If you are creating a new class that needs to implement this interface, don't forget the default constructor. @@ -636,39 +1099,60 @@ Bar bar = foo.getBar(); <--- imagine there's an extra space(s) after the
Javadoc This is also a very common feedback item. Don't forget Javadoc! + Javadoc warnings are checked during precommit. If the precommit tool gives you a '-1', + please fix the javadoc issue. Your patch won't be committed if it adds such warnings. +
+
+ Findbugs + + Findbugs is used to detect common bugs pattern. As Javadoc, it is checked during + the precommit build up on Apache's Jenkins, and as with Javadoc, please fix them. + You can run findbugs locally with 'mvn findbugs:findbugs': it will generate the + findbugs files locally. Sometimes, you may have to write code smarter than + Findbugs. You can annotate your code to tell Findbugs you know what you're + doing, by annotating your class with: + @edu.umd.cs.findbugs.annotations.SuppressWarnings( + value="HE_EQUALS_USE_HASHCODE", + justification="I know what I'm doing") + + + Note that we're using the apache licensed version of the annotations. + +
+
Javadoc - Useless Defaults Don't just leave the @param arguments the way your IDE generated them. Don't do this... /** - * + * * @param bar <---- don't do this!!!! * @return <---- or this!!!! */ public Foo getFoo(Bar bar); - - ... either add something descriptive to the @param and @return lines, or just remove them. - But the preference is to add something descriptive and useful. + + ... either add something descriptive to the @param and @return lines, or just remove them. + But the preference is to add something descriptive and useful.
One Thing At A Time, Folks If you submit a patch for one thing, don't do auto-reformatting or unrelated reformatting of code on a completely - different area of code. + different area of code. - Likewise, don't add unrelated cleanup or refactorings outside the scope of your Jira. + Likewise, don't add unrelated cleanup or refactorings outside the scope of your Jira.
Ambigious Unit Tests - Make sure that you're clear about what you are testing in your unit tests and why. + Make sure that you're clear about what you are testing in your unit tests and why.
- +
ReviewBoard Larger patches should go through ReviewBoard. @@ -676,16 +1160,29 @@ Bar bar = foo.getBar(); <--- imagine there's an extra space(s) after the For more information on how to use ReviewBoard, see the ReviewBoard documentation. -
+
Committing Patches - Committers do this. See How To Commit in the HBase wiki. + Committers do this. See How To Commit in the Apache HBase wiki. Commiters will also resolve the Jira, typically after the patch passes a build. +
+ Committers are responsible for making sure commits do not break the build or tests + + If a committer commits a patch it is their responsibility + to make sure it passes the test suite. It is helpful + if contributors keep an eye out that their patch + does not break the hbase build and/or tests but ultimately, + a contributor cannot be expected to be up on the + particular vagaries and interconnections that occur + in a project like hbase. A committer should. + +
- + + diff --git a/src/docbkx/external_apis.xml b/src/docbkx/external_apis.xml index 155a964862f3..6380b6e7b801 100644 --- a/src/docbkx/external_apis.xml +++ b/src/docbkx/external_apis.xml @@ -26,31 +26,34 @@ * limitations under the License. */ --> - External APIs - This chapter will cover access to HBase either through non-Java languages, or through custom protocols. - + Apache HBase (TM) External APIs + This chapter will cover access to Apache HBase (TM) either through non-Java languages, or through custom protocols. +
Non-Java Languages Talking to the JVM - Currently the documentation on this topic in the - HBase Wiki. + Currently the documentation on this topic in the + Apache HBase Wiki. + See also the Thrift API Javadoc.
REST - Currently most of the documentation on REST exists in the - HBase Wiki on REST. + Currently most of the documentation on REST exists in the + Apache HBase Wiki on REST (The REST gateway used to be + called 'Stargate'). There are also a nice set of blogs on How-to: Use the Apache HBase REST Interface + by Jesse Anderson.
Thrift - Currently most of the documentation on Thrift exists in the - HBase Wiki on Thrift. + Currently most of the documentation on Thrift exists in the + Apache HBase Wiki on Thrift.
Filter Language
Use Case - Note: this feature was introduced in HBase 0.92 + Note: this feature was introduced in Apache HBase 0.92 This allows the user to perform server-side filtering when accessing HBase over Thrift. The user specifies a filter via a string. The string is parsed on the server to construct the filter
@@ -407,10 +410,15 @@
- +
- + +
+ C/C++ Apache HBase Client + FB's Chip Turner wrote a pure C/C++ client. Check it out. + +
diff --git a/src/docbkx/getting_started.xml b/src/docbkx/getting_started.xml index 3aa392b810bf..e1c4344ef074 100644 --- a/src/docbkx/getting_started.xml +++ b/src/docbkx/getting_started.xml @@ -32,9 +32,8 @@ Introduction will get you up and - running on a single-node instance of HBase using the local filesystem. - describes setup - of HBase in distributed mode running on top of HDFS. + running on a single-node instance of HBase using the local filesystem. +
@@ -45,17 +44,31 @@ rows via the HBase shell, and then cleaning up and shutting down your standalone HBase instance. The below exercise should take no more than ten minutes (not including download time). + Before we proceed, make sure you are good on the below loopback prerequisite. + + Loopback IP + HBase expects the loopback IP address to be 127.0.0.1. Ubuntu and some other distributions, + for example, will default to 127.0.1.1 and this will cause problems for you. + + /etc/hosts should look something like this: + + 127.0.0.1 localhost + 127.0.0.1 ubuntu.ubuntu-domain ubuntu + + + +
Download and unpack the latest stable release. Choose a download site from this list of Apache Download - Mirrors. Click on suggested top link. This will take you to a + Mirrors. Click on the suggested top link. This will take you to a mirror of HBase Releases. Click on the folder named stable and then download the file that ends in .tar.gz to your local filesystem; e.g. - hbase-.tar.gz. + hbase-0.94.2.tar.gz. Decompress and untar your download and then change into the unpacked directory. @@ -65,24 +78,27 @@ $ cd hbase- At this point, you are ready to start HBase. But before starting - it, you might want to edit conf/hbase-site.xml and - set the directory you want HBase to write to, - hbase.rootdir. - -<?xml version="1.0"?> + it, edit conf/hbase-site.xml, the file you write + your site-specific configurations into. Set + hbase.rootdir, the directory HBase writes data to, + and hbase.zookeeper.property.dataDir, the director + ZooKeeper writes its data too: +<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hbase.rootdir</name> <value>file:///DIRECTORY/hbase</value> </property> -</configuration> - - Replace DIRECTORY in the above with a - path to a directory where you want HBase to store its data. By default, - hbase.rootdir is set to - /tmp/hbase-${user.name} which means you'll lose all - your data whenever your server reboots (Most operating systems clear + <property> + <name>hbase.zookeeper.property.dataDir</name> + <value>/DIRECTORY/zookeeper</value> + </property> +</configuration> Replace DIRECTORY in the above with the + path to the directory you would have HBase and ZooKeeper write their data. By default, + hbase.rootdir is set to /tmp/hbase-${user.name} + and similarly so for the default ZooKeeper data location which means you'll lose all + your data whenever your server reboots unless you change it (Most operating systems clear /tmp on restart).
@@ -96,19 +112,19 @@ starting Master, logging to logs/hbase-user-master-example.org.outlogs subdirectory. Check them out especially if - HBase had trouble starting. + it seems HBase had trouble starting. Is <application>java</application> installed? All of the above presumes a 1.6 version of Oracle java is installed on your machine and - available on your path; i.e. when you type + available on your path (See ); i.e. when you type java, you see output that describes the options the java program takes (HBase requires java 6). If this is not the case, HBase will not start. Install java, edit conf/hbase-env.sh, uncommenting the - JAVA_HOME line pointing it to your java install. Then, + JAVA_HOME line pointing it to your java install, then, retry the steps above.
@@ -154,9 +170,7 @@ hbase(main):006:0> put 'test', 'row3', 'cf:c', 'value3' cf in this example -- followed by a colon and then a column qualifier suffix (a in this case). - Verify the data insert. - - Run a scan of the table by doing the following + Verify the data insert by running a scan of the table as follows hbase(main):007:0> scan 'test' ROW COLUMN+CELL @@ -165,7 +179,7 @@ row2 column=cf:b, timestamp=1288380738440, value=value2 row3 column=cf:c, timestamp=1288380747365, value=value3 3 row(s) in 0.0590 seconds - Get a single row as follows + Get a single row hbase(main):008:0> get 'test', 'row1' COLUMN CELL @@ -198,9 +212,9 @@ stopping hbase............... Where to go next The above described standalone setup is good for testing and - experiments only. Next move on to where we'll go into - depth on the different HBase run modes, requirements and critical - configurations needed setting up a distributed HBase deploy. + experiments only. In the next chapter, , + we'll go into depth on the different HBase run modes, system requirements + running HBase, and critical configurations setting up a distributed HBase deploy. diff --git a/src/docbkx/ops_mgt.xml b/src/docbkx/ops_mgt.xml index 3dbd718a89c2..0009ab42bc09 100644 --- a/src/docbkx/ops_mgt.xml +++ b/src/docbkx/ops_mgt.xml @@ -26,16 +26,35 @@ * limitations under the License. */ --> - HBase Operational Management - This chapter will cover operational tools and practices required of a running HBase cluster. + Apache HBase (TM) Operational Management + This chapter will cover operational tools and practices required of a running Apache HBase cluster. The subject of operations is related to the topics of , , - and but is a distinct topic in itself. - + and but is a distinct topic in itself. +
HBase Tools and Utilities Here we list HBase tools for administration, analysis, fixup, and debugging. +
Driver + There is a Driver class that is executed by the HBase jar can be used to invoke frequently accessed utilities. For example, +HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar + +... will return... + +An example program must be given as the first argument. +Valid program names are: + completebulkload: Complete a bulk data load. + copytable: Export a table from local cluster to peer cluster + export: Write table data to HDFS. + import: Import data written by Export. + importtsv: Import data in TSV format. + rowcounter: Count rows in HBase table + verifyrep: Compare the data from tables in two different clusters. WARNING: It doesn't work for incrementColumnValues'd cells since the timestamp is chan + +... for allowable program names. + +
HBase <application>hbck</application> An fsck for your HBase install @@ -50,6 +69,8 @@ Passing -fix may correct the inconsistency (This latter is an experimental feature). + For more information, see . +
HFile Tool See . @@ -72,23 +93,28 @@ Similarly you can force a split of a log file directory by doing: $ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.HLog --split hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/ + +
+ <classname>HLogPrettyPrinter</classname> + HLogPrettyPrinter is a tool with configurable options to print the contents of an HLog. + +
+
Compression Tool - See . + See .
CopyTable CopyTable is a utility that can copy part or of all of a table, either to the same cluster or another cluster. The usage is as follows: -$ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable [--rs.class=CLASS] [--rs.impl=IMPL] [--starttime=X] [--endtime=Y] [--new.name=NEW] [--peer.adr=ADR] tablename +$ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable [--starttime=X] [--endtime=Y] [--new.name=NEW] [--peer.adr=ADR] tablename Options: - rs.class hbase.regionserver.class of the peer cluster. Specify if different from current cluster. - rs.impl hbase.regionserver.impl of the peer cluster. starttime Beginning of the time range. Without endtime means starttime to forever. endtime End of the time range. Without endtime means starttime to forever. versions Number of cell versions to copy. @@ -104,12 +130,15 @@ Example of copying 'TestTable' to a cluster that uses replication for a 1 hour window: $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable ---rs.class=org.apache.hadoop.hbase.ipc.ReplicationRegionInterface ---rs.impl=org.apache.hadoop.hbase.regionserver.replication.ReplicationRegionServer --starttime=1265875194289 --endtime=1265878794289 --peer.adr=server1,server2,server3:2181:/hbase TestTable - Note: caching for the input Scan is configured via hbase.client.scanner.caching in the job configuration. + Scanner Caching + Caching for the input Scan is configured via hbase.client.scanner.caching in the job configuration. + + + + See Jonathan Hsieh's Online HBase Backups with CopyTable blog post for more on CopyTable.
@@ -128,17 +157,156 @@
+
+ ImportTsv + ImportTsv is a utility that will load data in TSV format into HBase. It has two distinct usages: loading data from TSV format in HDFS + into HBase via Puts, and preparing StoreFiles to be loaded via the completebulkload. + + To load data via Puts (i.e., non-bulk loading): +$ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c <tablename> <hdfs-inputdir> + + + To generate StoreFiles for bulk-loading: +$ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c -Dimporttsv.bulk.output=hdfs://storefile-outputdir <tablename> <hdfs-data-inputdir> + + + These generated StoreFiles can be loaded into HBase via . + +
ImportTsv Options + Running ImportTsv with no arguments prints brief usage information: + +Usage: importtsv -Dimporttsv.columns=a,b,c <tablename> <inputdir> + +Imports the given input directory of TSV data into the specified table. + +The column names of the TSV data must be specified using the -Dimporttsv.columns +option. This option takes the form of comma-separated column names, where each +column name is either a simple column family, or a columnfamily:qualifier. The special +column name HBASE_ROW_KEY is used to designate that this column should be used +as the row key for each imported record. You must specify exactly one column +to be the row key, and you must specify a column name for every column that exists in the +input data. + +By default importtsv will load data directly into HBase. To instead generate +HFiles of data to prepare for a bulk data load, pass the option: + -Dimporttsv.bulk.output=/path/for/output + Note: the target table will be created with default column family descriptors if it does not already exist. + +Other options that may be specified with -D include: + -Dimporttsv.skip.bad.lines=false - fail if encountering an invalid line + '-Dimporttsv.separator=|' - eg separate on pipes instead of tabs + -Dimporttsv.timestamp=currentTimeAsLong - use the specified timestamp for the import + -Dimporttsv.mapper.class=my.Mapper - A user-defined Mapper to use instead of org.apache.hadoop.hbase.mapreduce.TsvImporterMapper + +
+
ImportTsv Example + For example, assume that we are loading data into a table called 'datatsv' with a ColumnFamily called 'd' with two columns "c1" and "c2". + + Assume that an input file exists as follows: + +row1 c1 c2 +row2 c1 c2 +row3 c1 c2 +row4 c1 c2 +row5 c1 c2 +row6 c1 c2 +row7 c1 c2 +row8 c1 c2 +row9 c1 c2 +row10 c1 c2 + + + For ImportTsv to use this imput file, the command line needs to look like this: + + HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,d:c1,d:c2 -Dimporttsv.bulk.output=hdfs://storefileoutput datatsv hdfs://inputfile + + ... and in this example the first column is the rowkey, which is why the HBASE_ROW_KEY is used. The second and third columns in the file will be imported as "d:c1" and "d:c2", respectively. + +
+
ImportTsv Warning + If you have preparing a lot of data for bulk loading, make sure the target HBase table is pre-split appropriately. + +
+
See Also + For more information about bulk-loading HFiles into HBase, see +
+
+ +
+ CompleteBulkLoad + The completebulkload utility will move generated StoreFiles into an HBase table. This utility is often used + in conjunction with output from . + + There are two ways to invoke this utility, with explicit classname and via the driver: +$ bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles <hdfs://storefileoutput> <tablename> + +.. and via the Driver.. +HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar completebulkload <hdfs://storefileoutput> <tablename> + + +
CompleteBulkLoad Warning + Data generated via MapReduce is often created with file permissions that are not compatible with the running HBase process. Assuming you're running HDFS with permissions enabled, those permissions will need to be updated before you run CompleteBulkLoad. + +
+ For more information about bulk-loading HFiles into HBase, see . + +
+
+ WALPlayer + WALPlayer is a utility to replay WAL files into HBase. + + The WAL can be replayed for a set of tables or all tables, and a + timerange can be provided (in milliseconds). The WAL is filtered to + this set of tables. The output can optionally be mapped to another set of tables. + + WALPlayer can also generate HFiles for later bulk importing, in that case + only a single table and no mapping can be specified. + + Invoke via: +$ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer [options] <wal inputdir> <tables> [<tableMappings>]> + + + For example: +$ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer /backuplogdir oldTable1,oldTable2 newTable1,newTable2 + + + + WALPlayer, by default, runs as a mapreduce job. To NOT run WALPlayer as a mapreduce job on your cluster, + force it to run all in the local process by adding the flags -Dmapred.job.tracker=local on the command line. + +
- RowCounter - RowCounter is a utility that will count all the rows of a table. This is a good utility to use - as a sanity check to ensure that HBase can read all the blocks of a table if there are any concerns of metadata inconsistency. + RowCounter and CellCounter + RowCounter is a + mapreduce job to count all the rows of a table. This is a good utility to use as a sanity check to ensure that HBase can read + all the blocks of a table if there are any concerns of metadata inconsistency. It will run the mapreduce all in a single + process but it will run faster if you have a MapReduce cluster in place for it to exploit. $ bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter <tablename> [<column1> <column2>...] - Note: caching for the input Scan is configured via hbase.client.scanner.caching in the job configuration. + Note: caching for the input Scan is configured via hbase.client.scanner.caching in the job configuration. + + HBase ships another diagnostic mapreduce job called + CellCounter. Like + RowCounter, it gathers more fine-grained statistics about your table. The statistics gathered by RowCounter are more fine-grained + and include: + + Total number of rows in the table. + Total number of CFs across all rows. + Total qualifiers across all rows. + Total occurrence of each CF. + Total occurrence of each qualifier. + Total number of versions of each qualifier. + + + The program allows you to limit the scope of the run. Provide a row regex or prefix to limit the rows to analyze. Use + hbase.mapreduce.scan.column.family to specify scanning a single column family. + $ bin/hbase org.apache.hadoop.hbase.mapreduce.CellCounter <tablename> <outputDir> [regex or prefix] + Note: just like RowCounter, caching for the input Scan is configured via hbase.client.scanner.caching in the + job configuration.
- +
@@ -148,7 +316,7 @@ Major compactions can be requested via the HBase shell or HBaseAdmin.majorCompact. Note: major compactions do NOT do region merges. See for more information about compactions. - +
@@ -157,16 +325,16 @@ $ bin/hbase org.apache.hbase.util.Merge <tablename> <region1> <region2> If you feel you have too many regions and want to consolidate them, Merge is the utility you need. Merge must - run be done when the cluster is down. + run be done when the cluster is down. See the O'Reilly HBase Book for an example of usage. - Additionally, there is a Ruby script attached to HBASE-1621 + Additionally, there is a Ruby script attached to HBASE-1621 for region merging.
- +
Node Management
Node Decommission You can stop an individual RegionServer by running the following @@ -189,10 +357,10 @@ A downside to the above stop of a RegionServer is that regions could be offline for a good period of time. Regions are closed in order. If many regions on the server, the first region to close may not be back online until all regions close and after the master - notices the RegionServer's znode gone. In HBase 0.90.2, we added facility for having - a node gradually shed its load and then shutdown itself down. HBase 0.90.2 added the + notices the RegionServer's znode gone. In Apache HBase 0.90.2, we added facility for having + a node gradually shed its load and then shutdown itself down. Apache HBase 0.90.2 added the graceful_stop.sh script. Here is its usage: - $ ./bin/graceful_stop.sh + $ ./bin/graceful_stop.sh Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] [--thrift] [--rest] &hostname> thrift If we should stop/start thrift before/after the hbase stop/start rest If we should stop/start rest before/after the hbase stop/start @@ -205,7 +373,7 @@ Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] [--thri To decommission a loaded RegionServer, run the following: $ ./bin/graceful_stop.sh HOSTNAME where HOSTNAME is the host carrying the RegionServer - you would decommission. + you would decommission. On <varname>HOSTNAME</varname> The HOSTNAME passed to graceful_stop.sh must match the hostname that hbase is using to identify RegionServers. @@ -227,7 +395,7 @@ Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] [--thri and because the RegionServer went down cleanly, there will be no WAL logs to split. Load Balancer - + It is assumed that the Region Load Balancer is disabled while the graceful_stop script runs (otherwise the balancer and the decommission script will end up fighting over region deployments). @@ -239,10 +407,31 @@ This turns the balancer OFF. To reenable, do: hbase(main):001:0> balance_switch true false 0 row(s) in 0.3590 seconds - + -
+
+ Bad or Failing Disk + It is good having set if you have a decent number of disks + per machine for the case where a disk plain dies. But usually disks do the "John Wayne" -- i.e. take a while + to go down spewing errors in dmesg -- or for some reason, run much slower than their + companions. In this case you want to decommission the disk. You have two options. You can + decommission the datanode + or, less disruptive in that only the bad disks data will be rereplicated, can stop the datanode, + unmount the bad volume (You can't umount a volume while the datanode is using it), and then restart the + datanode (presuming you have set dfs.datanode.failed.volumes.tolerated > 0). The regionserver will + throw some errors in its logs as it recalibrates where to get its data from -- it will likely + roll its WAL log too -- but in general but for some latency spikes, it should keep on chugging. + + If you are doing short-circuit reads, you will have to move the regions off the regionserver + before you stop the datanode; when short-circuiting reading, though chmod'd so regionserver cannot + have access, because it already has the files open, it will be able to keep reading the file blocks + from the bad disk even though the datanode is down. Move the regions back after you restart the + datanode. + + +
+
Rolling Restart @@ -300,7 +489,7 @@ false
- Metrics + HBase Metrics
Metric Setup See Metrics for @@ -381,8 +570,37 @@ false
HBase Monitoring - TODO - +
+ Overview + The following metrics are arguably the most important to monitor for each RegionServer for + "macro monitoring", preferably with a system like OpenTSDB. + If your cluster is having performance issues it's likely that you'll see something unusual with + this group. + + HBase: + + Requests + Compactions queue + + + OS: + + IO Wait + User CPU + + + Java: + + GC + + + + + + For more information on HBase metrics, see . + +
+
Slow Query Log The HBase slow query log consists of parseable JSON structures describing the properties of those client operations (Gets, Puts, Deletes, etc.) that either took too long to run, or produced too much output. The thresholds for "too long to run" and "too much output" are configurable, as described below. The output is produced inline in the main region server logs so that it is easy to discover further details from context with other logged events. It is also prepended with identifying tags (responseTooSlow), (responseTooLarge), (operationTooSlow), and (operationTooLarge) in order to enable easy filtering with grep, in case the user desires to see only slow queries. @@ -429,7 +647,7 @@ false
- +
Cluster Replication See Cluster Replication. @@ -437,8 +655,8 @@ false
HBase Backup - There are two broad strategies for performing HBase backups: backing up with a full cluster shutdown, and backing up on a live cluster. - Each approach has pros and cons. + There are two broad strategies for performing HBase backups: backing up with a full cluster shutdown, and backing up on a live cluster. + Each approach has pros and cons. For additional information, see HBase Backup Options over on the Sematext Blog. @@ -452,27 +670,27 @@ false
Distcp - Distcp could be used to either copy the contents of the HBase directory in HDFS to either the same cluster in another directory, or + Distcp could be used to either copy the contents of the HBase directory in HDFS to either the same cluster in another directory, or to a different cluster. - Note: Distcp works in this situation because the cluster is down and there are no in-flight edits to files. + Note: Distcp works in this situation because the cluster is down and there are no in-flight edits to files. Distcp-ing of files in the HBase directory is not generally recommended on a live cluster.
Restore (if needed) - The backup of the hbase directory from HDFS is copied onto the 'real' hbase directory via distcp. The act of copying these files + The backup of the hbase directory from HDFS is copied onto the 'real' hbase directory via distcp. The act of copying these files creates new HDFS metadata, which is why a restore of the NameNode edits from the time of the HBase backup isn't required for this kind of restore, because it's a restore (via distcp) of a specific HDFS directory (i.e., the HBase part) not the entire HDFS file-system.
Live Cluster Backup - Replication - This approach assumes that there is a second cluster. + This approach assumes that there is a second cluster. See the HBase page on replication for more information.
Live Cluster Backup - CopyTable - The utility could either be used to copy data from one table to another on the + The utility could either be used to copy data from one table to another on the same cluster, or to copy data to another table on another cluster. Since the cluster is up, there is a risk that edits could be missed in the copy process. @@ -486,6 +704,106 @@ false
+ +
+ HBase Snapshots + HBase Snapshots allow you to take a snapshot of a table without too much impact on Region Servers. + Snapshot, Clone and restore operations don't involve data copying. + Also, Exporting the snapshot to another cluster doesn't have impact on the Region Servers. + + Prior to version 0.94.6, the only way to backup or to clone a table is to use CopyTable/ExportTable, + or to copy all the hfiles in HDFS after disabling the table. + The disadvantages of these methods are that you can degrade region server performance + (Copy/Export Table) or you need to disable the table, that means no reads or writes; + and this is usually unacceptable. + +
Configuration + To turn on the snapshot support just set the + hbase.snapshot.enabled property to true. + (Snapshots are enabled by default in 0.95+ and off by default in 0.94.6+) + + <property> + <name>hbase.snapshot.enabled</name> + <value>true</value> + </property> + + +
+
Take a Snapshot + You can take a snapshot of a table regardless of whether it is enabled or disabled. + The snapshot operation doesn't involve any data copying. + + $ ./bin/hbase shell + hbase> snapshot 'myTable', 'myTableSnapshot-122112' + + +
+
Listing Snapshots + List all snapshots taken (by printing the names and relative information). + + $ ./bin/hbase shell + hbase> list_snapshots + + +
+
Deleting Snapshots + You can remove a snapshot, and the files retained for that snapshot will be removed + if no longer needed. + + $ ./bin/hbase shell + hbase> delete_snapshot 'myTableSnapshot-122112' + + +
+
Clone a table from snapshot + From a snapshot you can create a new table (clone operation) with the same data + that you had when the snapshot was taken. + The clone operation, doesn't involve data copies, and a change to the cloned table + doesn't impact the snapshot or the original table. + + $ ./bin/hbase shell + hbase> clone_snapshot 'myTableSnapshot-122112', 'myNewTestTable' + + +
+
Restore a snapshot + The restore operation requires the table to be disabled, and the table will be + restored to the state at the time when the snapshot was taken, + changing both data and schema if required. + + $ ./bin/hbase shell + hbase> disable 'myTable' + hbase> restore_snapshot 'myTableSnapshot-122112' + + + + Since Replication works at log level and snapshots at file-system level, + after a restore, the replicas will be in a different state from the master. + If you want to use restore, you need to stop replication and redo the bootstrap. + + + In case of partial data-loss due to misbehaving client, instead of a full restore + that requires the table to be disabled, you can clone the table from the snapshot + and use a Map-Reduce job to copy the data that you need, from the clone to the main one. + +
+
Snapshots operations and ACLs + If you are using security with the AccessController Coprocessor (See ), + only a global administrator can take, clone, or restore a snapshot, and these actions do not capture the ACL rights. + This means that restoring a table preserves the ACL rights of the existing table, + while cloning a table creates a new table that has no ACL rights until the administrator adds them. +
+
Export to another cluster + The ExportSnapshot tool copies all the data related to a snapshot (hfiles, logs, snapshot metadata) to another cluster. + The tool executes a Map-Reduce job, similar to distcp, to copy files between the two clusters, + and since it works at file-system level the hbase cluster does not have to be online. + To copy a snapshot called MySnapshot to an HBase cluster srv2 (hdfs:///srv2:8082/hbase) using 16 mappers: +$ bin/hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot MySnapshot -copy-to hdfs:///srv2:8082/hbase -mappers 16 + + +
+
+
Capacity Planning
Storage A common question for HBase administrators is estimating how much storage will be required for an HBase cluster. @@ -493,10 +811,10 @@ false with a solid understanding of how HBase handles data internally (KeyValue).
KeyValue - HBase storage will be dominated by KeyValues. See and for - how HBase stores data internally. + HBase storage will be dominated by KeyValues. See and for + how HBase stores data internally. - It is critical to understand that there is a KeyValue instance for every attribute stored in a row, and the + It is critical to understand that there is a KeyValue instance for every attribute stored in a row, and the rowkey-length, ColumnFamily name-length and attribute lengths will drive the size of the database more than any other factor. diff --git a/src/docbkx/performance.xml b/src/docbkx/performance.xml index 3ae843232698..8fb9559bd9f9 100644 --- a/src/docbkx/performance.xml +++ b/src/docbkx/performance.xml @@ -26,7 +26,7 @@ * limitations under the License. */ --> - Performance Tuning + Apache HBase (TM) Performance Tuning
Operating System @@ -47,7 +47,7 @@ Network Perhaps the most important factor in avoiding network issues degrading Hadoop and HBbase performance is the switching hardware - that is used, decisions made early in the scope of the project can cause major problems when you double or triple the size of your cluster (or more). + that is used, decisions made early in the scope of the project can cause major problems when you double or triple the size of your cluster (or more). Important items to consider: @@ -59,15 +59,15 @@
Single Switch - The single most important factor in this configuration is that the switching capacity of the hardware is capable of + The single most important factor in this configuration is that the switching capacity of the hardware is capable of handling the traffic which can be generated by all systems connected to the switch. Some lower priced commodity hardware - can have a slower switching capacity than could be utilized by a full switch. + can have a slower switching capacity than could be utilized by a full switch.
Multiple Switches Multiple switches are a potential pitfall in the architecture. The most common configuration of lower priced hardware is a - simple 1Gbps uplink from one switch to another. This often overlooked pinch point can easily become a bottleneck for cluster communication. + simple 1Gbps uplink from one switch to another. This often overlooked pinch point can easily become a bottleneck for cluster communication. Especially with MapReduce jobs that are both reading and writing a lot of data the communication across this uplink could be saturated. Mitigation of this issue is fairly simple and can be accomplished in multiple ways: @@ -85,22 +85,27 @@ Poor switch capacity performance Insufficient uplink to another rack - If the the switches in your rack have appropriate switching capacity to handle all the hosts at full speed, the next most likely issue will be caused by homing + If the the switches in your rack have appropriate switching capacity to handle all the hosts at full speed, the next most likely issue will be caused by homing more of your cluster across racks. The easiest way to avoid issues when spanning multiple racks is to use port trunking to create a bonded uplink to other racks. The downside of this method however, is in the overhead of ports that could potentially be used. An example of this is, creating an 8Gbps port channel from rack - A to rack B, using 8 of your 24 ports to communicate between racks gives you a poor ROI, using too few however can mean you're not getting the most out of your cluster. + A to rack B, using 8 of your 24 ports to communicate between racks gives you a poor ROI, using too few however can mean you're not getting the most out of your cluster. Using 10Gbe links between racks will greatly increase performance, and assuming your switches support a 10Gbe uplink or allow for an expansion card will allow you to save your ports for machines as opposed to uplinks. - +
+
+ Network Interfaces + Are all the network interfaces functioning correctly? Are you sure? See the Troubleshooting Case Study in . +
+
Java
- The Garbage Collector and HBase + The Garbage Collector and Apache HBase
Long GC pauses @@ -117,13 +122,20 @@ threshold, the more GCing is done, the more CPU used). To address the second fragmentation issue, Todd added an experimental facility, MSLAB, that - must be explicitly enabled in HBase 0.90.x (Its defaulted to be on in - 0.92.x HBase). See hbase.hregion.memstore.mslab.enabled + must be explicitly enabled in Apache HBase 0.90.x (Its defaulted to be on in + Apache 0.92.x HBase). See hbase.hregion.memstore.mslab.enabled to true in your Configuration. See the cited slides for background and detailThe latest jvms do better regards fragmentation so make sure you are running a recent release. Read down in the message, - Identifying concurrent mode failures caused by fragmentation.. + Identifying concurrent mode failures caused by fragmentation.. + Be aware that when enabled, each MemStore instance will occupy at least + an MSLAB instance of memory. If you have thousands of regions or lots + of regions each with many column families, this allocation of MSLAB + may be responsible for a good portion of your heap allocation and in + an extreme case cause you to OOME. Disable MSLAB in this case, or + lower the amount of memory it uses or float less regions per server. + For more information about GC logs, see .
@@ -135,6 +147,7 @@ See . +
Number of Regions @@ -153,41 +166,52 @@
<varname>hbase.regionserver.handler.count</varname> - See . + See .
<varname>hfile.block.cache.size</varname> - See . + See . A memory setting for the RegionServer process. -
+
<varname>hbase.regionserver.global.memstore.upperLimit</varname> - See . + See . This memory setting is often adjusted for the RegionServer process depending on needs. -
+
<varname>hbase.regionserver.global.memstore.lowerLimit</varname> - See . + See . This memory setting is often adjusted for the RegionServer process depending on needs.
<varname>hbase.hstore.blockingStoreFiles</varname> - See . + See . If there is blocking in the RegionServer logs, increasing this can help.
<varname>hbase.hregion.memstore.block.multiplier</varname> - See . - If there is enough RAM, increasing this can help. + See . + If there is enough RAM, increasing this can help. + +
+
+ <varname>hbase.regionserver.checksum.verify</varname> + Have HBase write the checksum into the datablock and save + having to do the checksum seek whenever you read. See the + release note on HBASE-5074 support checksums in HBase block cache.
+ + + +
ZooKeeper See for information on configuring ZooKeeper, and see the part @@ -196,19 +220,19 @@
Schema Design - +
Number of Column Families See .
Key and Attribute Lengths - See . See also for + See . See also for compression caveats.
Table RegionSize The regionsize can be set on a per-table basis via setFileSize on - HTableDescriptor in the + HTableDescriptor in the event where certain tables require different regionsizes than the configured default regionsize. See for more information. @@ -224,22 +248,23 @@ on each insert. If ROWCOL, the hash of the row + column family + column family qualifier will be added to the bloom on each key insert. - See HColumnDescriptor and - for more information. + See HColumnDescriptor and + for more information or this answer up in quora, +How are bloom filters used in HBase?.
ColumnFamily BlockSize - The blocksize can be configured for each ColumnFamily in a table, and this defaults to 64k. Larger cell values require larger blocksizes. + The blocksize can be configured for each ColumnFamily in a table, and this defaults to 64k. Larger cell values require larger blocksizes. There is an inverse relationship between blocksize and the resulting StoreFile indexes (i.e., if the blocksize is doubled then the resulting indexes should be roughly halved). - See HColumnDescriptor + See HColumnDescriptor and for more information.
In-Memory ColumnFamilies - ColumnFamilies can optionally be defined as in-memory. Data is still persisted to disk, just like any other ColumnFamily. + ColumnFamilies can optionally be defined as in-memory. Data is still persisted to disk, just like any other ColumnFamily. In-memory blocks have the highest priority in the , but it is not a guarantee that the entire table will be in memory. @@ -251,24 +276,24 @@ Production systems should use compression with their ColumnFamily definitions. See for more information.
However... - Compression deflates data on disk. When it's in-memory (e.g., in the + Compression deflates data on disk. When it's in-memory (e.g., in the MemStore) or on the wire (e.g., transferring between RegionServer and Client) it's inflated. So while using ColumnFamily compression is a best practice, but it's not going to completely eliminate - the impact of over-sized Keys, over-sized ColumnFamily names, or over-sized Column names. + the impact of over-sized Keys, over-sized ColumnFamily names, or over-sized Column names. See on for schema design tips, and for more information on HBase stores data internally. - +
- +
Writing to HBase
Batch Loading Use the bulk load tool if you can. See - Bulk Loads. + . Otherwise, pay attention to the below.
@@ -278,35 +303,27 @@ Table Creation: Pre-Creating Regions -Tables in HBase are initially created with one region by default. For bulk imports, this means that all clients will write to the same region until it is large enough to split and become distributed across the cluster. A useful pattern to speed up the bulk import process is to pre-create empty regions. Be somewhat conservative in this, because too-many regions can actually degrade performance. An example of pre-creation using hex-keys is as follows (note: this example may need to be tweaked to the individual applications keys): +Tables in HBase are initially created with one region by default. For bulk imports, this means that all clients will write to the same region +until it is large enough to split and become distributed across the cluster. A useful pattern to speed up the bulk import process is to pre-create empty regions. + Be somewhat conservative in this, because too-many regions can actually degrade performance. + There are two different approaches to pre-creating splits. The first approach is to rely on the default HBaseAdmin strategy + (which is implemented in Bytes.split)... + + +byte[] startKey = ...; // your lowest keuy +byte[] endKey = ...; // your highest key +int numberOfRegions = ...; // # of regions to create +admin.createTable(table, startKey, endKey, numberOfRegions); + + And the other approach is to define the splits yourself... + + +byte[][] splits = ...; // create your own splits +admin.createTable(table, splits); + -public static boolean createTable(HBaseAdmin admin, HTableDescriptor table, byte[][] splits) -throws IOException { - try { - admin.createTable( table, splits ); - return true; - } catch (TableExistsException e) { - logger.info("table " + table.getNameAsString() + " already exists"); - // the table already exists... - return false; - } -} - -public static byte[][] getHexSplits(String startKey, String endKey, int numRegions) { - byte[][] splits = new byte[numRegions-1][]; - BigInteger lowestKey = new BigInteger(startKey, 16); - BigInteger highestKey = new BigInteger(endKey, 16); - BigInteger range = highestKey.subtract(lowestKey); - BigInteger regionIncrement = range.divide(BigInteger.valueOf(numRegions)); - lowestKey = lowestKey.add(regionIncrement); - for(int i=0; i < numRegions-1;i++) { - BigInteger key = lowestKey.add(regionIncrement.multiply(BigInteger.valueOf(i))); - byte[] b = String.format("%016x", key).getBytes(); - splits[i] = b; - } - return splits; -} + See for issues related to understanding your keyspace and pre-creating regions.
@@ -314,7 +331,7 @@ public static byte[][] getHexSplits(String startKey, String endKey, int numRegio Table Creation: Deferred Log Flush -The default behavior for Puts using the Write Ahead Log (WAL) is that HLog edits will be written immediately. If deferred log flush is used, +The default behavior for Puts using the Write Ahead Log (WAL) is that HLog edits will be written immediately. If deferred log flush is used, WAL edits are kept in memory until the flush period. The benefit is aggregated and asynchronous HLog- writes, but the potential downside is that if the RegionServer goes down the yet-to-be-flushed edits are lost. This is safer, however, than not using WAL at all with Puts. @@ -322,7 +339,7 @@ WAL edits are kept in memory until the flush period. The benefit is aggregated Deferred log flush can be configured on tables via HTableDescriptor. The default value of hbase.regionserver.optionallogflushinterval is 1000ms. -
+
HBase Client: AutoFlush @@ -348,25 +365,25 @@ Deferred log flush can be configured on tables via In general, it is best to use WAL for Puts, and where loading throughput - is a concern to use bulk loading techniques instead. + is a concern to use bulk loading techniques instead.
HBase Client: Group Puts by RegionServer - In addition to using the writeBuffer, grouping Puts by RegionServer can reduce the number of client RPC calls per writeBuffer flush. + In addition to using the writeBuffer, grouping Puts by RegionServer can reduce the number of client RPC calls per writeBuffer flush. There is a utility HTableUtil currently on TRUNK that does this, but you can either copy that or implement your own verison for those still on 0.90.x or earlier. -
+
MapReduce: Skip The Reducer When writing a lot of data to an HBase table from a MR job (e.g., with TableOutputFormat), and specifically where Puts are being emitted - from the Mapper, skip the Reducer step. When a Reducer step is used, all of the output (Puts) from the Mapper will get spooled to disk, then sorted/shuffled to other - Reducers that will most likely be off-node. It's far more efficient to just write directly to HBase. + from the Mapper, skip the Reducer step. When a Reducer step is used, all of the output (Puts) from the Mapper will get spooled to disk, then sorted/shuffled to other + Reducers that will most likely be off-node. It's far more efficient to just write directly to HBase. - For summary jobs where HBase is used as a source and a sink, then writes will be coming from the Reducer step (e.g., summarize values then write out result). - This is a different processing problem than from the the above case. + For summary jobs where HBase is used as a source and a sink, then writes will be coming from the Reducer step (e.g., summarize values then write out result). + This is a different processing problem than from the the above case.
@@ -375,16 +392,16 @@ Deferred log flush can be configured on tables via If all your data is being written to one region at a time, then re-read the section on processing timeseries data. Also, if you are pre-splitting regions and all your data is still winding up in a single region even though - your keys aren't monotonically increasing, confirm that your keyspace actually works with the split strategy. There are a + your keys aren't monotonically increasing, confirm that your keyspace actually works with the split strategy. There are a variety of reasons that regions may appear "well split" but won't work with your data. As - the HBase client communicates directly with the RegionServers, this can be obtained via + the HBase client communicates directly with the RegionServers, this can be obtained via HTable.getRegionLocation. - See , as well as + See , as well as
- +
Reading from HBase @@ -406,7 +423,7 @@ Deferred log flush can be configured on tables via Scan settings in MapReduce jobs deserve special attention. Timeouts can result (e.g., UnknownScannerException) in Map tasks if it takes longer to process a batch of records before the client goes back to the RegionServer for the next set of data. This problem can occur because there is non-trivial processing occuring per row. If you process - rows quickly, set caching higher. If you process rows more slowly (e.g., lots of transformations per row, writes), + rows quickly, set caching higher. If you process rows more slowly (e.g., lots of transformations per row, writes), then set caching lower. Timeouts can also happen in a non-MapReduce use case (i.e., single threaded HBase client doing a Scan), but the @@ -424,6 +441,27 @@ Deferred log flush can be configured on tables via
+
+ Avoid scan seeks + When columns are selected explicitly with scan.addColumn, HBase will schedule seek operations to seek between the + selected columns. When rows have few columns and each column has only a few versions this can be inefficient. A seek operation is generally + slower if does not seek at least past 5-10 columns/versions or 512-1024 bytes. + In order to opportunistically look ahead a few columns/versions to see if the next column/version can be found that + way before a seek operation is scheduled, a new attribute Scan.HINT_LOOKAHEAD can be set the on Scan object. The following code instructs the + RegionServer to attempt two iterations of next before a seek is scheduled: +Scan scan = new Scan(); +scan.addColumn(...); +scan.setAttribute(Scan.HINT_LOOKAHEAD, Bytes.toBytes(2)); +table.getScanner(scan); + +
+
+ MapReduce - Input Splits + For MapReduce jobs that use HBase tables as a source, if there a pattern where the "slow" map tasks seem to + have the same Input Split (i.e., the RegionServer serving the data), see the + Troubleshooting Case Study in . + +
Close ResultScanners @@ -469,13 +507,103 @@ htable.close();
Concurrency: Monitor Data Spread - When performing a high number of concurrent reads, monitor the data spread of the target tables. If the target table(s) have + When performing a high number of concurrent reads, monitor the data spread of the target tables. If the target table(s) have too few regions then the reads could likely be served from too few nodes. - See , as well as + See , as well as
- +
+ Bloom Filters + Enabling Bloom Filters can save your having to go to disk and + can help improve read latencys. + Bloom filters were developed over in HBase-1200 + Add bloomfilters. + For description of the development process -- why static blooms + rather than dynamic -- and for an overview of the unique properties + that pertain to blooms in HBase, as well as possible future + directions, see the Development Process section + of the document BloomFilters + in HBase attached to HBase-1200. + + The bloom filters described here are actually version two of + blooms in HBase. In versions up to 0.19.x, HBase had a dynamic bloom + option based on work done by the European Commission One-Lab + Project 034819. The core of the HBase bloom work was later + pulled up into Hadoop to implement org.apache.hadoop.io.BloomMapFile. + Version 1 of HBase blooms never worked that well. Version 2 is a + rewrite from scratch though again it starts with the one-lab + work. + + See also . + + +
+ Bloom StoreFile footprint + + Bloom filters add an entry to the StoreFile + general FileInfo data structure and then two + extra entries to the StoreFile metadata + section. + +
+ BloomFilter in the <classname>StoreFile</classname> + <classname>FileInfo</classname> data structure + + FileInfo has a + BLOOM_FILTER_TYPE entry which is set to + NONE, ROW or + ROWCOL. +
+ +
+ BloomFilter entries in <classname>StoreFile</classname> + metadata + + BLOOM_FILTER_META holds Bloom Size, Hash + Function used, etc. Its small in size and is cached on + StoreFile.Reader load + BLOOM_FILTER_DATA is the actual bloomfilter + data. Obtained on-demand. Stored in the LRU cache, if it is enabled + (Its enabled by default). +
+
+
+ Bloom Filter Configuration +
+ <varname>io.hfile.bloom.enabled</varname> global kill + switch + + io.hfile.bloom.enabled in + Configuration serves as the kill switch in case + something goes wrong. Default = true. +
+ +
+ <varname>io.hfile.bloom.error.rate</varname> + + io.hfile.bloom.error.rate = average false + positive rate. Default = 1%. Decrease rate by ½ (e.g. to .5%) == +1 + bit per bloom entry. +
+ +
+ <varname>io.hfile.bloom.max.fold</varname> + + io.hfile.bloom.max.fold = guaranteed minimum + fold rate. Most people should leave this alone. Default = 7, or can + collapse to at least 1/128th of original size. See the + Development Process section of the document BloomFilters + in HBase for more on what this option means. +
+
+
+ - +
Deleting from HBase
@@ -503,21 +631,54 @@ htable.close();
Current Issues With Low-Latency Reads The original use-case for HDFS was batch processing. As such, there low-latency reads were historically not a priority. - With the increased adoption of HBase this is changing, and several improvements are already in development. - See the + With the increased adoption of Apache HBase this is changing, and several improvements are already in development. + See the Umbrella Jira Ticket for HDFS Improvements for HBase.
+
+ Leveraging local data +Since Hadoop 1.0.0 (also 0.22.1, 0.23.1, CDH3u3 and HDP 1.0) via +HDFS-2246, +it is possible for the DFSClient to take a "short circuit" and +read directly from disk instead of going through the DataNode when the +data is local. What this means for HBase is that the RegionServers can +read directly off their machine's disks instead of having to open a +socket to talk to the DataNode, the former being generally much +fasterSee JD's Performance Talk. +Also see HBase, mail # dev - read short circuit thread for +more discussion around short circuit reads. + +To enable "short circuit" reads, you must set two configurations. +First, the hdfs-site.xml needs to be amended. Set +the property dfs.block.local-path-access.user +to be the only user that can use the shortcut. +This has to be the user that started HBase. Then in hbase-site.xml, +set dfs.client.read.shortcircuit to be true + + + For optimal performance when short-circuit reads are enabled, it is recommended that HDFS checksums are disabled. + To maintain data integrity with HDFS checksums disabled, HBase can be configured to write its own checksums into + its datablocks and verify against these. See . + + +The DataNodes need to be restarted in order to pick up the new +configuration. Be aware that if a process started under another +username than the one configured here also has the shortcircuit +enabled, it will get an Exception regarding an unauthorized access but +the data will still be read. + +
Performance Comparisons of HBase vs. HDFS - A fairly common question on the dist-list is why HBase isn't as performant as HDFS files in a batch context (e.g., as - a MapReduce source or sink). The short answer is that HBase is doing a lot more than HDFS (e.g., reading the KeyValues, - returning the most current row or specified timestamps, etc.), and as such HBase is 4-5 times slower than HDFS in this + A fairly common question on the dist-list is why HBase isn't as performant as HDFS files in a batch context (e.g., as + a MapReduce source or sink). The short answer is that HBase is doing a lot more than HDFS (e.g., reading the KeyValues, + returning the most current row or specified timestamps, etc.), and as such HBase is 4-5 times slower than HDFS in this processing context. Not that there isn't room for improvement (and this gap will, over time, be reduced), but HDFS will always be faster in this use-case.
- +
Amazon EC2 Performance questions are common on Amazon EC2 environments because it is a shared environment. You will not see the same throughput as a dedicated server. In terms of running tests on EC2, run them several times for the same @@ -527,4 +688,9 @@ htable.close(); because EC2 issues are practically a separate class of performance issues.
+ +
Case Studies + For Performance and Troubleshooting Case Studies, see . + +
diff --git a/src/docbkx/preface.xml b/src/docbkx/preface.xml index 2d9f39d1c678..af54aa29749a 100644 --- a/src/docbkx/preface.xml +++ b/src/docbkx/preface.xml @@ -33,7 +33,7 @@ Herein you will find either the definitive documentation on an HBase topic as of its standing when the referenced HBase version shipped, or it will point to the location in javadoc, + xlink:href="http://hbase.apache.org/apidocs/index.html">javadoc, JIRA or wiki where the pertinent information can be found. diff --git a/src/docbkx/security.xml b/src/docbkx/security.xml new file mode 100644 index 000000000000..ed4a0c2ed638 --- /dev/null +++ b/src/docbkx/security.xml @@ -0,0 +1,532 @@ + + + +Secure Apache HBase (TM) +
+ Secure Client Access to Apache HBase + Newer releases of Apache HBase (TM) (>= 0.92) support optional SASL authentication of clientsSee + also Matteo Bertozzi's article on Understanding User Authentication and Authorization in Apache HBase.. + This describes how to set up Apache HBase and clients for connection to secure HBase resources. + +
Prerequisites + + You need to have a working Kerberos KDC. + + + A HBase configured for secure client access is expected to be running + on top of a secured HDFS cluster. HBase must be able to authenticate + to HDFS services. HBase needs Kerberos credentials to interact with + the Kerberos-enabled HDFS daemons. Authenticating a service should be + done using a keytab file. The procedure for creating keytabs for HBase + service is the same as for creating keytabs for Hadoop. Those steps + are omitted here. Copy the resulting keytab files to wherever HBase + Master and RegionServer processes are deployed and make them readable + only to the user account under which the HBase daemons will run. + + + A Kerberos principal has three parts, with the form + username/fully.qualified.domain.name@YOUR-REALM.COM. We + recommend using hbase as the username portion. + + + The following is an example of the configuration properties for + Kerberos operation that must be added to the + hbase-site.xml file on every server machine in the + cluster. Required for even the most basic interactions with a + secure Hadoop configuration, independent of HBase security. + + + hbase.regionserver.kerberos.principal + hbase/_HOST@YOUR-REALM.COM + + + hbase.regionserver.keytab.file + /etc/hbase/conf/keytab.krb5 + + + hbase.master.kerberos.principal + hbase/_HOST@YOUR-REALM.COM + + + hbase.master.keytab.file + /etc/hbase/conf/keytab.krb5 + + ]]> + + Each HBase client user should also be given a Kerberos principal. This + principal should have a password assigned to it (as opposed to a + keytab file). The client principal's maxrenewlife should + be set so that it can be renewed enough times for the HBase client + process to complete. For example, if a user runs a long-running HBase + client process that takes at most 3 days, we might create this user's + principal within kadmin with: addprinc -maxrenewlife + 3days + + + Long running daemons with indefinite lifetimes that require client + access to HBase can instead be configured to log in from a keytab. For + each host running such daemons, create a keytab with + kadmin or kadmin.local. The procedure for + creating keytabs for HBase service is the same as for creating + keytabs for Hadoop. Those steps are omitted here. Copy the resulting + keytab files to where the client daemon will execute and make them + readable only to the user account under which the daemon will run. + +
+ +
Server-side Configuration for Secure Operation + + Add the following to the hbase-site.xml file on every server machine in the cluster: + + + hbase.security.authentication + kerberos + + + hbase.security.authorization + true + + + hbase.coprocessor.region.classes + org.apache.hadoop.hbase.security.token.TokenProvider + + ]]> + + A full shutdown and restart of HBase service is required when deploying + these configuration changes. + +
+ +
Client-side Configuration for Secure Operation + + Add the following to the hbase-site.xml file on every client: + + + hbase.security.authentication + kerberos + + ]]> + + The client environment must be logged in to Kerberos from KDC or + keytab via the kinit command before communication with + the HBase cluster will be possible. + + + Be advised that if the hbase.security.authentication + in the client- and server-side site files do not match, the client will + not be able to communicate with the cluster. + + + Once HBase is configured for secure RPC it is possible to optionally + configure encrypted communication. To do so, add the following to the + hbase-site.xml file on every client: + + + hbase.rpc.protection + privacy + + ]]> + + This configuration property can also be set on a per connection basis. + Set it in the Configuration supplied to + HTable: + + + Configuration conf = HBaseConfiguration.create(); + conf.set("hbase.rpc.protection", "privacy"); + HTable table = new HTable(conf, tablename); + + + Expect a ~10% performance penalty for encrypted communication. + +
+ +
Client-side Configuration for Secure Operation - Thrift Gateway + + Add the following to the hbase-site.xml file for every Thrift gateway: + + hbase.thrift.keytab.file + /etc/hbase/conf/hbase.keytab + + + hbase.thrift.kerberos.principal + $USER/_HOST@HADOOP.LOCALDOMAIN + + ]]> + + + Substitute the appropriate credential and keytab for $USER and $KEYTAB + respectively. + + + The Thrift gateway will authenticate with HBase using the supplied + credential. No authentication will be performed by the Thrift gateway + itself. All client access via the Thrift gateway will use the Thrift + gateway's credential and have its privilege. + +
+ +
Client-side Configuration for Secure Operation - REST Gateway + + Add the following to the hbase-site.xml file for every REST gateway: + + hbase.rest.keytab.file + $KEYTAB + + + hbase.rest.kerberos.principal + $USER/_HOST@HADOOP.LOCALDOMAIN + + ]]> + + + Substitute the appropriate credential and keytab for $USER and $KEYTAB + respectively. + + + The REST gateway will authenticate with HBase using the supplied + credential. No authentication will be performed by the REST gateway + itself. All client access via the REST gateway will use the REST + gateway's credential and have its privilege. + + + It should be possible for clients to authenticate with the HBase + cluster through the REST gateway in a pass-through manner via SPEGNO + HTTP authentication. This is future work. + +
+ +
+ + +
+ Access Control + + Newer releases of Apache HBase (>= 0.92) support optional access control + list (ACL-) based protection of resources on a column family and/or + table basis. + + + This describes how to set up Secure HBase for access control, with an + example of granting and revoking user permission on table resources + provided. + + +
Prerequisites + + You must configure HBase for secure operation. Refer to the section + "Secure Client Access to HBase" and complete all of the steps described + there. + + + You must also configure ZooKeeper for secure operation. Changes to ACLs + are synchronized throughout the cluster using ZooKeeper. Secure + authentication to ZooKeeper must be enabled or otherwise it will be + possible to subvert HBase access control via direct client access to + ZooKeeper. Refer to the section on secure ZooKeeper configuration and + complete all of the steps described there. + +
+ +
Overview + + With Secure RPC and Access Control enabled, client access to HBase is + authenticated and user data is private unless access has been + explicitly granted. Access to data can be granted at a table or per + column family basis. + + + However, the following items have been left out of the initial + implementation for simplicity: + + + + Row-level or per value (cell): This would require broader changes for storing the ACLs inline with rows. It is a future goal. + + + Push down of file ownership to HDFS: HBase is not designed for the case where files may have different permissions than the HBase system principal. Pushing file ownership down into HDFS would necessitate changes to core code. Also, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it is not clear that it would offer a more secure setup. + + + HBase managed "roles" as collections of permissions: We will not model "roles" internally in HBase to begin with. We instead allow group names to be granted permissions, which allows external modeling of roles via group membership. Groups are created and manipulated externally to HBase, via the Hadoop group mapping service. + + + +Access control mechanisms are mature and fairly standardized in the relational database world. The HBase implementation approximates current convention, but HBase has a simpler feature set than relational databases, especially in terms of client operations. We don't distinguish between an insert (new record) and update (of existing record), for example, as both collapse down into a Put. Accordingly, the important operations condense to four permissions: READ, WRITE, CREATE, and ADMIN. + + + Operation To Permission Mapping + + + Permission + Operation + + + + + + Read + Get + + + + Exists + + + + Scan + + + + Write + Put + + + + Delete + + + + Lock/UnlockRow + + + + IncrementColumnValue + + + + CheckAndDelete/Put + + + + Flush + + + + Compact + + + + Create + Create + + + + Alter + + + + Drop + + + + Admin + Enable/Disable + + + + Split + + + + Major Compact + + + + Grant + + + + Revoke + + + + Shutdown + + +
+ + Permissions can be granted in any of the following scopes, though + CREATE and ADMIN permissions are effective only at table scope. + + + + + Table + + + Read: User can read from any column family in table + Write: User can write to any column family in table + Create: User can alter table attributes; add, alter, or drop column families; and drop the table. + Admin: User can alter table attributes; add, alter, or drop column families; and enable, disable, or drop the table. User can also trigger region (re)assignments or relocation. + + + + + Column Family + + + Read: User can read from the column family + Write: User can write to the column family + + + + + + + There is also an implicit global scope for the superuser. + + + The superuser is a principal, specified in the HBase site configuration + file, that has equivalent access to HBase as the 'root' user would on a + UNIX derived system. Normally this is the principal that the HBase + processes themselves authenticate as. Although future versions of HBase + Access Control may support multiple superusers, the superuser privilege + will always include the principal used to run the HMaster process. Only + the superuser is allowed to create tables, switch the balancer on or + off, or take other actions with global consequence. Furthermore, the + superuser has an implicit grant of all permissions to all resources. + + + Tables have a new metadata attribute: OWNER, the user principal who owns + the table. By default this will be set to the user principal who creates + the table, though it may be changed at table creation time or during an + alter operation by setting or changing the OWNER table attribute. Only a + single user principal can own a table at a given time. A table owner will + have all permissions over a given table. + +
+ +
Server-side Configuration for Access Control + + Enable the AccessController coprocessor in the cluster configuration + and restart HBase. The restart can be a rolling one. Complete the + restart of all Master and RegionServer processes before setting up + ACLs. + + + To enable the AccessController, modify the hbase-site.xml file on every server machine in the cluster to look like: + + + hbase.coprocessor.master.classes + org.apache.hadoop.hbase.security.access.AccessController + + + hbase.coprocessor.region.classes + org.apache.hadoop.hbase.security.token.TokenProvider, + org.apache.hadoop.hbase.security.access.AccessController + + ]]> +
+ +
Shell Enhancements for Access Control + +The HBase shell has been extended to provide simple commands for editing and updating user permissions. The following commands have been added for access control list management: + + Grant + + + grant <user> <permissions> <table> [ <column family> [ <column qualifier> ] ] + + + + <permissions> is zero or more letters from the set "RWCA": READ('R'), WRITE('W'), CREATE('C'), ADMIN('A'). + + + Note: Grants and revocations of individual permissions on a resource are both accomplished using the grant command. A separate revoke command is also provided by the shell, but this is for fast revocation of all of a user's access rights to a given resource only. + + + Revoke + + + + revoke <user> <table> [ <column family> [ <column qualifier> ] ] + + + + Alter + + + The alter command has been extended to allow ownership assignment: + + alter 'tablename', {OWNER => 'username'} + + + + User Permission + + + The user_permission command shows all access permissions for the current user for a given table: + + user_permission <table> + + +
+ +
+ +
+ Secure Bulk Load + + Bulk loading in secure mode is a bit more involved than normal setup, since the client has to transfer the ownership of the files generated from the mapreduce job to HBase. Secure bulk loading is implemented by a coprocessor, named SecureBulkLoadEndpoint. SecureBulkLoadEndpoint uses a staging directory "hbase.bulkload.staging.dir", which defaults to /tmp/hbase-staging/. The algorithm is as follows. + + Create an hbase owned staging directory which is world traversable (-rwx--x--x, 711) /tmp/hbase-staging. + A user writes out data to his secure output directory: /user/foo/data + A call is made to hbase to create a secret staging directory + which is globally readable/writable (-rwxrwxrwx, 777): /tmp/hbase-staging/averylongandrandomdirectoryname + The user makes the data world readable and writable, then moves it + into the random staging directory, then calls bulkLoadHFiles() + + + + Like delegation tokens the strength of the security lies in the length + and randomness of the secret directory. + + + + You have to enable the secure bulk load to work properly. You can modify the hbase-site.xml file on every server machine in the cluster and add the SecureBulkLoadEndpoint class to the list of regionserver coprocessors: + + + hbase.bulkload.staging.dir + /tmp/hbase-staging + + + hbase.coprocessor.region.classes + org.apache.hadoop.hbase.security.token.TokenProvider, + org.apache.hadoop.hbase.security.access.AccessController,org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint + + ]]> +
+ diff --git a/src/docbkx/shell.xml b/src/docbkx/shell.xml index 4fbab08d2236..2a1535336189 100644 --- a/src/docbkx/shell.xml +++ b/src/docbkx/shell.xml @@ -26,13 +26,13 @@ * limitations under the License. */ --> - The HBase Shell + The Apache HBase Shell - The HBase Shell is (J)Ruby's + The Apache HBase (TM) Shell is (J)Ruby's IRB with some HBase particular commands added. Anything you can do in IRB, you should be able to do in the HBase Shell. - To run the HBase shell, + To run the HBase shell, do as follows: $ ./bin/hbase shell @@ -47,7 +47,7 @@ for example basic shell operation.
Scripting - For examples scripting HBase, look in the + For examples scripting Apache HBase, look in the HBase bin directory. Look at the files that end in *.rb. To run one of these files, do as follows: @@ -104,5 +104,16 @@
+
Commands +
count + Count command returns the number of rows in a table. + It's quite fast when configured with the right CACHE + hbase> count '<tablename>', CACHE => 1000 + The above count fetches 1000 rows at a time. Set CACHE lower if your rows are big. + Default is to fetch one row at a time. + +
+
+ diff --git a/src/docbkx/troubleshooting.xml b/src/docbkx/troubleshooting.xml index a92d9794e925..5967b03a3d65 100644 --- a/src/docbkx/troubleshooting.xml +++ b/src/docbkx/troubleshooting.xml @@ -26,7 +26,7 @@ * limitations under the License. */ --> - Troubleshooting and Debugging HBase + Troubleshooting and Debugging Apache HBase (TM)
General Guidelines @@ -37,7 +37,7 @@ should return some hits for those exceptions you’re seeing. - An error rarely comes alone in HBase, usually when something gets screwed up what will + An error rarely comes alone in Apache HBase (TM), usually when something gets screwed up what will follow may be hundreds of exceptions and stack traces coming from all over the place. The best way to approach this type of problem is to walk the log up to where it all began, for example one trick with RegionServers is that they will print some @@ -54,7 +54,7 @@ prolonged garbage collection pauses that last longer than the default ZooKeeper session timeout. For more information on GC pauses, see the 3 part blog post by Todd Lipcon - and above. + and above.
@@ -72,7 +72,7 @@ JobTracker: $HADOOP_HOME/logs/hadoop-<user>-jobtracker-<hostname>.log - TaskTracker: $HADOOP_HOME/logs/hadoop-<user>-jobtracker-<hostname>.log + TaskTracker: $HADOOP_HOME/logs/hadoop-<user>-tasktracker-<hostname>.log HMaster: $HBASE_HOME/logs/hbase-<user>-master-<hostname>.log @@ -91,7 +91,7 @@ NameNode The NameNode log is on the NameNode server. The HBase Master is typically run on the NameNode server, and well as ZooKeeper. For smaller clusters the JobTracker is typically run on the NameNode server as well. -
+
DataNode Each DataNode server will have a DataNode log for HDFS, as well as a RegionServer log for HBase. @@ -105,32 +105,32 @@ insight on timings at the server. Once enabled, the amount of log spewed is voluminous. It is not recommended that you leave this logging on for more than short bursts of time. To enable RPC-level - logging, browse to the RegionServer UI and click on + logging, browse to the RegionServer UI and click on Log Level. Set the log level to DEBUG for the package org.apache.hadoop.ipc (Thats right, for hadoop.ipc, NOT, hbase.ipc). Then tail the RegionServers log. Analyze. To disable, set the logging level back to INFO level. -
- + +
JVM Garbage Collection Logs - HBase is memory intensive, and using the default GC you can see long pauses in all threads including the Juliet Pause aka "GC of Death". - To help debug this or confirm this is happening GC logging can be turned on in the Java virtual machine. + HBase is memory intensive, and using the default GC you can see long pauses in all threads including the Juliet Pause aka "GC of Death". + To help debug this or confirm this is happening GC logging can be turned on in the Java virtual machine. To enable, in hbase-env.sh add: - + export HBASE_OPTS="-XX:+UseConcMarkSweepGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/home/hadoop/hbase/logs/gc-hbase.log" - Adjust the log directory to wherever you log. Note: The GC log does NOT roll automatically, so you'll have to keep an eye on it so it doesn't fill up the disk. + Adjust the log directory to wherever you log. Note: The GC log does NOT roll automatically, so you'll have to keep an eye on it so it doesn't fill up the disk. At this point you should see logs like so: -64898.952: [GC [1 CMS-initial-mark: 2811538K(3055704K)] 2812179K(3061272K), 0.0007360 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] +64898.952: [GC [1 CMS-initial-mark: 2811538K(3055704K)] 2812179K(3061272K), 0.0007360 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 64898.953: [CMS-concurrent-mark-start] -64898.971: [GC 64898.971: [ParNew: 5567K->576K(5568K), 0.0101110 secs] 2817105K->2812715K(3061272K), 0.0102200 secs] [Times: user=0.07 sys=0.00, real=0.01 secs] +64898.971: [GC 64898.971: [ParNew: 5567K->576K(5568K), 0.0101110 secs] 2817105K->2812715K(3061272K), 0.0102200 secs] [Times: user=0.07 sys=0.00, real=0.01 secs] @@ -139,20 +139,20 @@ export HBASE_OPTS="-XX:+UseConcMarkSweepGC -verbose:gc -XX:+PrintGCDetails -XX:+ The third line indicates a "minor GC", which pauses the VM for 0.0101110 seconds - aka 10 milliseconds. It has reduced the "ParNew" from about 5.5m to 576k. Later on in this cycle we see: - -64901.445: [CMS-concurrent-mark: 1.542/2.492 secs] [Times: user=10.49 sys=0.33, real=2.49 secs] + +64901.445: [CMS-concurrent-mark: 1.542/2.492 secs] [Times: user=10.49 sys=0.33, real=2.49 secs] 64901.445: [CMS-concurrent-preclean-start] -64901.453: [GC 64901.453: [ParNew: 5505K->573K(5568K), 0.0062440 secs] 2868746K->2864292K(3061272K), 0.0063360 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] -64901.476: [GC 64901.476: [ParNew: 5563K->575K(5568K), 0.0072510 secs] 2869283K->2864837K(3061272K), 0.0073320 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] -64901.500: [GC 64901.500: [ParNew: 5517K->573K(5568K), 0.0120390 secs] 2869780K->2865267K(3061272K), 0.0121150 secs] [Times: user=0.09 sys=0.00, real=0.01 secs] -64901.529: [GC 64901.529: [ParNew: 5507K->569K(5568K), 0.0086240 secs] 2870200K->2865742K(3061272K), 0.0087180 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] -64901.554: [GC 64901.555: [ParNew: 5516K->575K(5568K), 0.0107130 secs] 2870689K->2866291K(3061272K), 0.0107820 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] -64901.578: [CMS-concurrent-preclean: 0.070/0.133 secs] [Times: user=0.48 sys=0.01, real=0.14 secs] +64901.453: [GC 64901.453: [ParNew: 5505K->573K(5568K), 0.0062440 secs] 2868746K->2864292K(3061272K), 0.0063360 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] +64901.476: [GC 64901.476: [ParNew: 5563K->575K(5568K), 0.0072510 secs] 2869283K->2864837K(3061272K), 0.0073320 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] +64901.500: [GC 64901.500: [ParNew: 5517K->573K(5568K), 0.0120390 secs] 2869780K->2865267K(3061272K), 0.0121150 secs] [Times: user=0.09 sys=0.00, real=0.01 secs] +64901.529: [GC 64901.529: [ParNew: 5507K->569K(5568K), 0.0086240 secs] 2870200K->2865742K(3061272K), 0.0087180 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] +64901.554: [GC 64901.555: [ParNew: 5516K->575K(5568K), 0.0107130 secs] 2870689K->2866291K(3061272K), 0.0107820 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] +64901.578: [CMS-concurrent-preclean: 0.070/0.133 secs] [Times: user=0.48 sys=0.01, real=0.14 secs] 64901.578: [CMS-concurrent-abortable-preclean-start] -64901.584: [GC 64901.584: [ParNew: 5504K->571K(5568K), 0.0087270 secs] 2871220K->2866830K(3061272K), 0.0088220 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] -64901.609: [GC 64901.609: [ParNew: 5512K->569K(5568K), 0.0063370 secs] 2871771K->2867322K(3061272K), 0.0064230 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] -64901.615: [CMS-concurrent-abortable-preclean: 0.007/0.037 secs] [Times: user=0.13 sys=0.00, real=0.03 secs] -64901.616: [GC[YG occupancy: 645 K (5568 K)]64901.616: [Rescan (parallel) , 0.0020210 secs]64901.618: [weak refs processing, 0.0027950 secs] [1 CMS-remark: 2866753K(3055704K)] 2867399K(3061272K), 0.0049380 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] +64901.584: [GC 64901.584: [ParNew: 5504K->571K(5568K), 0.0087270 secs] 2871220K->2866830K(3061272K), 0.0088220 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] +64901.609: [GC 64901.609: [ParNew: 5512K->569K(5568K), 0.0063370 secs] 2871771K->2867322K(3061272K), 0.0064230 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] +64901.615: [CMS-concurrent-abortable-preclean: 0.007/0.037 secs] [Times: user=0.13 sys=0.00, real=0.03 secs] +64901.616: [GC[YG occupancy: 645 K (5568 K)]64901.616: [Rescan (parallel) , 0.0020210 secs]64901.618: [weak refs processing, 0.0027950 secs] [1 CMS-remark: 2866753K(3055704K)] 2867399K(3061272K), 0.0049380 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 64901.621: [CMS-concurrent-sweep-start] @@ -161,20 +161,20 @@ export HBASE_OPTS="-XX:+UseConcMarkSweepGC -verbose:gc -XX:+PrintGCDetails -XX:+ There are a few more minor GCs, then there is a pause at the 2nd last line: - -64901.616: [GC[YG occupancy: 645 K (5568 K)]64901.616: [Rescan (parallel) , 0.0020210 secs]64901.618: [weak refs processing, 0.0027950 secs] [1 CMS-remark: 2866753K(3055704K)] 2867399K(3061272K), 0.0049380 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] + +64901.616: [GC[YG occupancy: 645 K (5568 K)]64901.616: [Rescan (parallel) , 0.0020210 secs]64901.618: [weak refs processing, 0.0027950 secs] [1 CMS-remark: 2866753K(3055704K)] 2867399K(3061272K), 0.0049380 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] - The pause here is 0.0049380 seconds (aka 4.9 milliseconds) to 'remark' the heap. + The pause here is 0.0049380 seconds (aka 4.9 milliseconds) to 'remark' the heap. At this point the sweep starts, and you can watch the heap size go down: -64901.637: [GC 64901.637: [ParNew: 5501K->569K(5568K), 0.0097350 secs] 2871958K->2867441K(3061272K), 0.0098370 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] +64901.637: [GC 64901.637: [ParNew: 5501K->569K(5568K), 0.0097350 secs] 2871958K->2867441K(3061272K), 0.0098370 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] ... lines removed ... -64904.936: [GC 64904.936: [ParNew: 5532K->568K(5568K), 0.0070720 secs] 1365024K->1360689K(3061272K), 0.0071930 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] -64904.953: [CMS-concurrent-sweep: 2.030/3.332 secs] [Times: user=9.57 sys=0.26, real=3.33 secs] +64904.936: [GC 64904.936: [ParNew: 5532K->568K(5568K), 0.0070720 secs] 1365024K->1360689K(3061272K), 0.0071930 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] +64904.953: [CMS-concurrent-sweep: 2.030/3.332 secs] [Times: user=9.57 sys=0.26, real=3.33 secs] At this point, the CMS sweep took 3.332 seconds, and heap went from about ~ 2.8 GB to 1.3 GB (approximate). @@ -186,14 +186,14 @@ export HBASE_OPTS="-XX:+UseConcMarkSweepGC -verbose:gc -XX:+PrintGCDetails -XX:+ Add this to HBASE_OPTS: - + export HBASE_OPTS="-XX:NewSize=64m -XX:MaxNewSize=64m <cms options from above> <gc logging options from above>" For more information on GC pauses, see the 3 part blog post by Todd Lipcon and above. - +
@@ -201,18 +201,18 @@ export HBASE_OPTS="-XX:NewSize=64m -XX:MaxNewSize=64m <cms options from above
search-hadoop.com - search-hadoop.com indexes all the mailing lists and is great for historical searches. + search-hadoop.com indexes all the mailing lists and is great for historical searches. Search here first when you have an issue as its more than likely someone has already had your problem.
Mailing Lists - Ask a question on the HBase mailing lists. - The 'dev' mailing list is aimed at the community of developers actually building HBase and for features currently under development, and 'user' - is generally used for questions on released versions of HBase. Before going to the mailing list, make sure your + Ask a question on the Apache HBase mailing lists. + The 'dev' mailing list is aimed at the community of developers actually building Apache HBase and for features currently under development, and 'user' + is generally used for questions on released versions of Apache HBase. Before going to the mailing list, make sure your question has not already been answered by searching the mailing list archives first. Use . - Take some time crafting your questionSee Getting Answers; a quality question that includes all context and + Take some time crafting your questionSee Getting Answers; a quality question that includes all context and exhibits evidence the author has tried to find answers in the manual and out on lists is more likely to get a prompt response. @@ -236,7 +236,7 @@ export HBASE_OPTS="-XX:NewSize=64m -XX:MaxNewSize=64m <cms options from above Master Web Interface The Master starts a web-interface on port 60010 by default. - The Master web UI lists created tables and their definition (e.g., ColumnFamilies, blocksize, etc.). Additionally, + The Master web UI lists created tables and their definition (e.g., ColumnFamilies, blocksize, etc.). Additionally, the available RegionServers in the cluster are listed along with selected high-level metrics (requests, number of regions, usedHeap, maxHeap). The Master web UI allows navigation to each RegionServer's web UI. @@ -263,13 +263,13 @@ export HBASE_OPTS="-XX:NewSize=64m -XX:MaxNewSize=64m <cms options from above ls path [watch] set path data [version] delquota [-n|-b] path - quit + quit printwatches on|off create [-s] [-e] path data acl stat path [watch] - close + close ls2 path [watch] - history + history listquota path setAcl path acl getAcl path @@ -292,7 +292,7 @@ export HBASE_OPTS="-XX:NewSize=64m -XX:MaxNewSize=64m <cms options from above
top - + top is probably one of the most important tool when first trying to see what’s running on a machine and how the resources are consumed. Here’s an example from production system: top - 14:46:59 up 39 days, 11:55, 1 user, load average: 3.75, 3.57, 3.84 @@ -300,10 +300,10 @@ Tasks: 309 total, 1 running, 308 sleeping, 0 stopped, 0 zombie Cpu(s): 4.5%us, 1.6%sy, 0.0%ni, 91.7%id, 1.4%wa, 0.1%hi, 0.6%si, 0.0%st Mem: 24414432k total, 24296956k used, 117476k free, 7196k buffers Swap: 16008732k total, 14348k used, 15994384k free, 11106908k cached - - PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND -15558 hadoop 18 -2 3292m 2.4g 3556 S 79 10.4 6523:52 java -13268 hadoop 18 -2 8967m 8.2g 4104 S 21 35.1 5170:30 java + + PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND +15558 hadoop 18 -2 3292m 2.4g 3556 S 79 10.4 6523:52 java +13268 hadoop 18 -2 8967m 8.2g 4104 S 21 35.1 5170:30 java 8895 hadoop 18 -2 1581m 497m 3420 S 11 2.1 4002:32 java … @@ -351,7 +351,7 @@ hadoop@sv4borg12:~$ jps hadoop@sv4borg12:~$ ps aux | grep HRegionServer hadoop 17789 155 35.2 9067824 8604364 ? S<l Mar04 9855:48 /usr/java/jdk1.6.0_14/bin/java -Xmx8000m -XX:+DoEscapeAnalysis -XX:+AggressiveOpts -XX:+UseConcMarkSweepGC -XX:NewSize=64m -XX:MaxNewSize=64m -XX:CMSInitiatingOccupancyFraction=88 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/export1/hadoop/logs/gc-hbase.log -Dcom.sun.management.jmxremote.port=10102 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/home/hadoop/hbase/conf/jmxremote.password -Dcom.sun.management.jmxremote -Dhbase.log.dir=/export1/hadoop/logs -Dhbase.log.file=hbase-hadoop-regionserver-sv4borg12.log -Dhbase.home.dir=/home/hadoop/hbase -Dhbase.id.str=hadoop -Dhbase.root.logger=INFO,DRFA -Djava.library.path=/home/hadoop/hbase/lib/native/Linux-amd64-64 -classpath /home/hadoop/hbase/bin/../conf:[many jars]:/home/hadoop/hadoop/conf org.apache.hadoop.hbase.regionserver.HRegionServer start - +
@@ -371,7 +371,7 @@ hadoop 17789 155 35.2 9067824 8604364 ? S<l Mar04 9855:48 /usr/java/j at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:647) at java.lang.Thread.run(Thread.java:619) - + The MemStore flusher thread that is currently flushing to a file: "regionserver60020.cacheFlusher" daemon prio=10 tid=0x0000000040f4e000 nid=0x45eb in Object.wait() [0x00007f16b5b86000..0x00007f16b5b87af0] java.lang.Thread.State: WAITING (on object monitor) @@ -444,7 +444,7 @@ hadoop 17789 155 35.2 9067824 8604364 ? S<l Mar04 9855:48 /usr/java/j A thread that receives data from HDFS: - + "IPC Client (47) connection to sv4borg9/10.4.24.40:9000 from hadoop" daemon prio=10 tid=0x00007f16a02d0000 nid=0x4fa3 runnable [0x00007f16b517d000..0x00007f16b517dbf0] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) @@ -498,63 +498,75 @@ hadoop 17789 155 35.2 9067824 8604364 ? S<l Mar04 9855:48 /usr/java/j
OpenTSDB - OpenTSDB is an excellent alternative to Ganglia as it uses HBase to store all the time series and doesn’t have to downsample. Monitoring your own HBase cluster that hosts OpenTSDB is a good exercise. + OpenTSDB is an excellent alternative to Ganglia as it uses Apache HBase to store all the time series and doesn’t have to downsample. Monitoring your own HBase cluster that hosts OpenTSDB is a good exercise. Here’s an example of a cluster that’s suffering from hundreds of compactions launched almost all around the same time, which severely affects the IO performance: (TODO: insert graph plotting compactionQueueSize) - It’s a good practice to build dashboards with all the important graphs per machine and per cluster so that debugging issues can be done with a single quick look. For example, at StumbleUpon there’s one dashboard per cluster with the most important metrics from both the OS and HBase. You can then go down at the machine level and get even more detailed metrics. + It’s a good practice to build dashboards with all the important graphs per machine and per cluster so that debugging issues can be done with a single quick look. For example, at StumbleUpon there’s one dashboard per cluster with the most important metrics from both the OS and Apache HBase. You can then go down at the machine level and get even more detailed metrics.
clusterssh+top - - clusterssh+top, it’s like a poor man’s monitoring system and it can be quite useful when you have only a few machines as it’s very easy to setup. Starting clusterssh will give you one terminal per machine and another terminal in which whatever you type will be retyped in every window. This means that you can type “top” once and it will start it for all of your machines at the same time giving you full view of the current state of your cluster. You can also tail all the logs at the same time, edit files, etc. + + clusterssh+top, it’s like a poor man’s monitoring system and it can be quite useful when you have only a few machines as it’s very easy to setup. Starting clusterssh will give you one terminal per machine and another terminal in which whatever you type will be retyped in every window. This means that you can type “top” once and it will start it for all of your machines at the same time giving you full view of the current state of your cluster. You can also tail all the logs at the same time, edit files, etc.
- +
Client - For more information on the HBase client, see . + For more information on the HBase client, see .
ScannerTimeoutException or UnknownScannerException - This is thrown if the time between RPC calls from the client to RegionServer exceeds the scan timeout. + This is thrown if the time between RPC calls from the client to RegionServer exceeds the scan timeout. For example, if Scan.setCaching is set to 500, then there will be an RPC call to fetch the next batch of rows every 500 .next() calls on the ResultScanner because data is being transferred in blocks of 500 rows to the client. Reducing the setCaching value may be an option, but setting this value too low makes for inefficient processing on numbers of rows. See . -
+
+
+ <classname>LeaseException</classname> when calling <classname>Scanner.next</classname> + +In some situations clients that fetch data from a RegionServer get a LeaseException instead of the usual +. Usually the source of the exception is +org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230) (line number may vary). +It tends to happen in the context of a slow/freezing RegionServer#next call. +It can be prevented by having hbase.rpc.timeout > hbase.regionserver.lease.period. +Harsh J investigated the issue as part of the mailing list thread +HBase, mail # user - Lease does not exist exceptions + +
Shell or client application throws lots of scary exceptions during normal operation Since 0.20.0 the default log level for org.apache.hadoop.hbase.*is DEBUG. - On your clients, edit $HBASE_HOME/conf/log4j.properties and change this: log4j.logger.org.apache.hadoop.hbase=DEBUG to this: log4j.logger.org.apache.hadoop.hbase=INFO, or even log4j.logger.org.apache.hadoop.hbase=WARN. + On your clients, edit $HBASE_HOME/conf/log4j.properties and change this: log4j.logger.org.apache.hadoop.hbase=DEBUG to this: log4j.logger.org.apache.hadoop.hbase=INFO, or even log4j.logger.org.apache.hadoop.hbase=WARN. -
+
Long Client Pauses With Compression - This is a fairly frequent question on the HBase dist-list. The scenario is that a client is typically inserting a lot of data into a + This is a fairly frequent question on the Apache HBase dist-list. The scenario is that a client is typically inserting a lot of data into a relatively un-optimized HBase cluster. Compression can exacerbate the pauses, although it is not the source of the problem. See on the pattern for pre-creating regions and confirm that the table isn't starting with a single region. - See for cluster configuration, particularly hbase.hstore.blockingStoreFiles, hbase.hregion.memstore.block.multiplier, + See for cluster configuration, particularly hbase.hstore.blockingStoreFiles, hbase.hregion.memstore.block.multiplier, MAX_FILESIZE (region size), and MEMSTORE_FLUSHSIZE. - A slightly longer explanation of why pauses can happen is as follows: Puts are sometimes blocked on the MemStores which are blocked by the flusher thread which is blocked because there are + A slightly longer explanation of why pauses can happen is as follows: Puts are sometimes blocked on the MemStores which are blocked by the flusher thread which is blocked because there are too many files to compact because the compactor is given too many small files to compact and has to compact the same data repeatedly. This situation can occur even with minor compactions. - Compounding this situation, HBase doesn't compress data in memory. Thus, the 64MB that lives in the MemStore could become a 6MB file after compression - which results in a smaller StoreFile. The upside is that + Compounding this situation, Apache HBase doesn't compress data in memory. Thus, the 64MB that lives in the MemStore could become a 6MB file after compression - which results in a smaller StoreFile. The upside is that more data is packed into the same region, but performance is achieved by being able to write larger files - which is why HBase waits until the flushize before writing a new StoreFile. And smaller StoreFiles - become targets for compaction. Without compression the files are much bigger and don't need as much compaction, however this is at the expense of I/O. + become targets for compaction. Without compression the files are much bigger and don't need as much compaction, however this is at the expense of I/O. For additional information, see this thread on Long client pauses with compression. - -
+ +
ZooKeeper Client Connection Errors Errors like this... @@ -576,11 +588,11 @@ hadoop 17789 155 35.2 9067824 8604364 ? S<l Mar04 9855:48 /usr/java/j 11/07/05 11:26:45 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181 - ... are either due to ZooKeeper being down, or unreachable due to network issues. + ... are either due to ZooKeeper being down, or unreachable due to network issues. The utility may help investigate ZooKeeper issues. -
+
Client running out of memory though heap size seems to be stable (but the off-heap/direct heap keeps growing) @@ -595,24 +607,46 @@ it a bit hefty. You want to make this setting client-side only especially if y server-side off-heap cache since this feature depends on being able to use big direct buffers (You may have to keep separate client-side and server-side config dirs). -
+
Client Slowdown When Calling Admin Methods (flush, compact, etc.) This is a client issue fixed by HBASE-5073 in 0.90.6. -There was a ZooKeeper leak in the client and the client was getting pummeled by ZooKeeper events with each additional -invocation of the admin API. +There was a ZooKeeper leak in the client and the client was getting pummeled by ZooKeeper events with each additional +invocation of the admin API. -
+ + +
+ Secure Client Cannot Connect ([Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) + +There can be several causes that produce this symptom. + + +First, check that you have a valid Kerberos ticket. One is required in order to set up communication with a secure Apache HBase cluster. Examine the ticket currently in the credential cache, if any, by running the klist command line utility. If no ticket is listed, you must obtain a ticket by running the kinit command with either a keytab specified, or by interactively entering a password for the desired principal. + + +Then, consult the Java Security Guide troubleshooting section. The most common problem addressed there is resolved by setting javax.security.auth.useSubjectCredsOnly system property value to false. + + +Because of a change in the format in which MIT Kerberos writes its credentials cache, there is a bug in the Oracle JDK 6 Update 26 and earlier that causes Java to be unable to read the Kerberos credentials cache created by versions of MIT Kerberos 1.8.1 or higher. If you have this problematic combination of components in your environment, to work around this problem, first log in with kinit and then immediately refresh the credential cache with kinit -R. The refresh will rewrite the credential cache without the problematic formatting. + + +Finally, depending on your Kerberos configuration, you may need to install the Java Cryptography Extension, or JCE. Insure the JCE jars are on the classpath on both server and client systems. + + +You may also need to download the unlimited strength JCE policy files. Uncompress and extract the downloaded file, and install the policy jars into <java-home>/lib/security. + +
- +
MapReduce
You Think You're On The Cluster, But You're Actually Local This following stacktrace happened using ImportTsv, but things like this - can happen on any job with a mis-configuration. + can happen on any job with a mis-configuration. WARN mapred.LocalJobRunner: job_local_0001 java.lang.IllegalArgumentException: Can't read partitions file @@ -637,17 +671,17 @@ Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. LocalJobRunner means the job is running locally, not on the cluster. - See + See - http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath for more + http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath for more information on HBase MapReduce jobs and classpaths. - +
- +
NameNode - For more information on the NameNode, see . + For more information on the NameNode, see .
HDFS Utilization of Tables and Regions @@ -657,7 +691,7 @@ Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. hadoop fs -du /hbase/myTable ...returns a list of the regions under the HBase table 'myTable' and their disk utilization. For more information on HDFS shell commands, see the HDFS FileSystem Shell documentation. -
+
Browsing HDFS for HBase Objects Somtimes it will be necessary to explore the HBase objects that exist on HDFS. These objects could include the WALs (Write Ahead Logs), tables, regions, StoreFiles, etc. @@ -675,30 +709,30 @@ Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. The HDFS directory structure of HBase WAL is.. /hbase - /.logs + /.logs /<RegionServer> (RegionServers) /<HLog> (WAL HLog files for the RegionServer) - See the HDFS User Guide for other non-shell diagnostic - utilities like fsck. + See the HDFS User Guide for other non-shell diagnostic + utilities like fsck.
Use Cases - Two common use-cases for querying HDFS for HBase objects is research the degree of uncompaction of a table. If there are a large number of StoreFiles for each ColumnFamily it could + Two common use-cases for querying HDFS for HBase objects is research the degree of uncompaction of a table. If there are a large number of StoreFiles for each ColumnFamily it could indicate the need for a major compaction. Additionally, after a major compaction if the resulting StoreFile is "small" it could indicate the need for a reduction of ColumnFamilies for the table.
-
+ - +
Network
Network Spikes - If you are seeing periodic network spikes you might want to check the compactionQueues to see if major + If you are seeing periodic network spikes you might want to check the compactionQueues to see if major compactions are happening. See for more information on managing compactions. @@ -709,11 +743,17 @@ Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. HBase expects the loopback IP Address to be 127.0.0.1. See the Getting Started section on .
+
+ Network Interfaces + Are all the network interfaces functioning correctly? Are you sure? See the Troubleshooting Case Study in . + +
+
- +
RegionServer - For more information on the RegionServers, see . + For more information on the RegionServers, see .
Startup Errors @@ -721,9 +761,9 @@ Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. Master Starts, But RegionServers Do Not The Master believes the RegionServers have the IP of 127.0.0.1 - which is localhost and resolves to the master's own localhost. - The RegionServers are erroneously informing the Master that their IP addresses are 127.0.0.1. + The RegionServers are erroneously informing the Master that their IP addresses are 127.0.0.1. - Modify /etc/hosts on the region servers, from... + Modify /etc/hosts on the region servers, from... # Do not remove the following line, or various programs # that require network functionality will fail. @@ -739,7 +779,7 @@ Caused by: java.io.FileNotFoundException: File _partition.lst does not exist.
- +
Compression Link Errors @@ -753,8 +793,8 @@ java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path .. then there is a path issue with the compression libraries. See the Configuration section on LZO compression configuration. -
-
+ +
Runtime Errors @@ -767,7 +807,7 @@ java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path Adding -XX:+UseMembar to the HBase HBASE_OPTS in conf/hbase-env.sh may fix it. - Also, are you using ? These are discouraged because they can lock up the + Also, are you using ? These are discouraged because they can lock up the RegionServers if not managed properly.
@@ -776,7 +816,7 @@ java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path If you see log messages like this... -2010-09-13 01:24:17,336 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: +2010-09-13 01:24:17,336 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Disk-related IOException in BlockReceiver constructor. Cause is java.io.IOException: Too many open files at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:883) @@ -807,7 +847,7 @@ Disk-related IOException in BlockReceiver constructor. Cause is java.io.IOExcept 2009-02-24 10:01:33,516 WARN org.apache.hadoop.hbase.util.Sleeper: We slept xxx ms, ten times longer than scheduled: 10000 2009-02-24 10:01:33,516 WARN org.apache.hadoop.hbase.util.Sleeper: We slept xxx ms, ten times longer than scheduled: 15000 -2009-02-24 10:01:36,472 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to master for xxx milliseconds - retrying +2009-02-24 10:01:36,472 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to master for xxx milliseconds - retrying ... or see full GC compactions then you may be experiencing full GC's. @@ -838,12 +878,12 @@ java.io.IOException: Session Expired at org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589) at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945) -ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expired +ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expired The JVM is doing a long running garbage collecting which is pausing every threads (aka "stop the world"). Since the RegionServer's local ZooKeeper client cannot send heartbeats, the session times out. - By design, we shut down any node that isn't able to contact the ZooKeeper ensemble after getting a timeout so that it stops serving data that may already be assigned elsewhere. + By design, we shut down any node that isn't able to contact the ZooKeeper ensemble after getting a timeout so that it stops serving data that may already be assigned elsewhere. @@ -852,7 +892,7 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi Make sure you are not CPU starving the RegionServer thread. For example, if you are running a MapReduce job using 6 CPU-intensive tasks on a machine with 4 cores, you are probably starving the RegionServer enough to create longer garbage collection pauses. Increase the ZooKeeper session timeout - If you wish to increase the session timeout, add the following to your hbase-site.xml to increase the timeout from the default of 60 seconds to 120 seconds. + If you wish to increase the session timeout, add the following to your hbase-site.xml to increase the timeout from the default of 60 seconds to 120 seconds. <property> <name>zookeeper.session.timeout</name> @@ -866,8 +906,8 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi Be aware that setting a higher timeout means that the regions served by a failed RegionServer will take at least - that amount of time to be transfered to another RegionServer. For a production system serving live requests, we would instead - recommend setting it lower than 1 minute and over-provision your cluster in order the lower the memory load on each machines (hence having + that amount of time to be transfered to another RegionServer. For a production system serving live requests, we would instead + recommend setting it lower than 1 minute and over-provision your cluster in order the lower the memory load on each machines (hence having less garbage to collect per machine). @@ -884,7 +924,7 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi
Regions listed by domain name, then IP - Fix your DNS. In versions of HBase before 0.92.x, reverse DNS needs to give same answer + Fix your DNS. In versions of Apache HBase before 0.92.x, reverse DNS needs to give same answer as forward lookup. See HBASE 3431 RegionServer is not using the name given it by the master; double entry in master listing of servers for gorey details. @@ -908,35 +948,41 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi
- +
Shutdown Errors -
+ - +
Master - For more information on the Master, see . + For more information on the Master, see .
Startup Errors
Master says that you need to run the hbase migrations script Upon running that, the hbase migrations script says no files in root directory. - HBase expects the root directory to either not exist, or to have already been initialized by hbase running a previous time. If you create a new directory for HBase using Hadoop DFS, this error will occur. - Make sure the HBase root directory does not currently exist or has been initialized by a previous run of HBase. Sure fire solution is to just use Hadoop dfs to delete the HBase root and let HBase create and initialize the directory itself. - + HBase expects the root directory to either not exist, or to have already been initialized by hbase running a previous time. If you create a new directory for HBase using Hadoop DFS, this error will occur. + Make sure the HBase root directory does not currently exist or has been initialized by a previous run of HBase. Sure fire solution is to just use Hadoop dfs to delete the HBase root and let HBase create and initialize the directory itself. + +
+
+ Packet len6080218 is out of range! + If you have many regions on your cluster and you see an error + like that reported above in this sections title in your logs, see + HBASE-4246 Cluster with too many regions cannot withstand some master failover scenarios.
- -
+ +
Shutdown Errors -
+ - +
ZooKeeper @@ -945,28 +991,28 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi
Could not find my address: xyz in list of ZooKeeper quorum servers A ZooKeeper server wasn't able to start, throws that error. xyz is the name of your server. - This is a name lookup problem. HBase tries to start a ZooKeeper server on some machine but that machine isn't able to find itself in the hbase.zookeeper.quorum configuration. - - Use the hostname presented in the error message instead of the value you used. If you have a DNS server, you can set hbase.zookeeper.dns.interface and hbase.zookeeper.dns.nameserver in hbase-site.xml to make sure it resolves to the correct FQDN. - + This is a name lookup problem. HBase tries to start a ZooKeeper server on some machine but that machine isn't able to find itself in the hbase.zookeeper.quorum configuration. + + Use the hostname presented in the error message instead of the value you used. If you have a DNS server, you can set hbase.zookeeper.dns.interface and hbase.zookeeper.dns.nameserver in hbase-site.xml to make sure it resolves to the correct FQDN. +
- -
+ +
ZooKeeper, The Cluster Canary ZooKeeper is the cluster's "canary in the mineshaft". It'll be the first to notice issues if any so making sure its happy is the short-cut to a humming cluster. - + See the ZooKeeper Operating Environment Troubleshooting page. It has suggestions and tools for checking disk and networking performance; i.e. the operating environment your ZooKeeper and HBase are running in. Additionally, the utility may help investigate ZooKeeper issues. -
+ - +
- Amazon EC2 + Amazon EC2
ZooKeeper does not seem to work on Amazon EC2 HBase does not start when deployed as Amazon EC2 instances. Exceptions like the below appear in the Master and/or RegionServer logs: @@ -978,8 +1024,8 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi java.net.ConnectException: Connection refused - Security group policy is blocking the ZooKeeper port on a public address. - Use the internal EC2 host names when configuring the ZooKeeper quorum peer list. + Security group policy is blocking the ZooKeeper port on a public address. + Use the internal EC2 host names when configuring the ZooKeeper quorum peer list.
@@ -993,15 +1039,15 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi See Andrew's answer here, up on the user list: Remote Java client connection into EC2 instance.
- +
- +
- HBase and Hadoop version issues + HBase and Hadoop version issues
<code>NoClassDefFoundError</code> when trying to run 0.90.x on hadoop-0.20.205.x (or hadoop-1.0.x) - HBase 0.90.x does not ship with hadoop-0.20.205.x, etc. To make it run, you need to replace the hadoop - jars that HBase shipped with in its lib directory with those of the Hadoop you want to + Apache HBase 0.90.x does not ship with hadoop-0.20.205.x, etc. To make it run, you need to replace the hadoop + jars that Apache HBase shipped with in its lib directory with those of the Hadoop you want to run HBase on. If even after replacing Hadoop jars you get the below exception: sv4r6s38: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration @@ -1018,7 +1064,27 @@ sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.ensureInitial you need to copy under hbase/lib, the commons-configuration-X.jar you find in your Hadoop's lib directory. That should fix the above complaint. +
+ +
+ ...cannot communicate with client version... +If you see something like the following in your logs +... +2012-09-24 10:20:52,168 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown. +org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4 +... +...are you trying to talk to an Hadoop 2.0.x from an HBase that has an Hadoop 1.0.x client? +Use the HBase built against Hadoop 2.0 or rebuild your HBase passing the -Dhadoop.profile=2.0 +attribute to Maven (See for more). + +
- + +
+ Case Studies + For Performance and Troubleshooting Case Studies, see . + +
+ diff --git a/src/docbkx/upgrading.xml b/src/docbkx/upgrading.xml index 5a1887284cf4..d1dcdd8c0e36 100644 --- a/src/docbkx/upgrading.xml +++ b/src/docbkx/upgrading.xml @@ -27,49 +27,29 @@ */ --> Upgrading + You cannot skip major verisons upgrading. If you are upgrading from + version 0.20.x to 0.92.x, you must first go from 0.20.x to 0.90.x and then go + from 0.90.x to 0.92.x. Review , in particular the section on Hadoop version. -
- Upgrading to HBase 0.90.x from 0.20.x or 0.89.x - This version of 0.90.x HBase can be started on data written by - HBase 0.20.x or HBase 0.89.x. There is no need of a migration step. - HBase 0.89.x and 0.90.x does write out the name of region directories - differently -- it names them with a md5 hash of the region name rather - than a jenkins hash -- so this means that once started, there is no - going back to HBase 0.20.x. - - - Be sure to remove the hbase-default.xml from - your conf - directory on upgrade. A 0.20.x version of this file will have - sub-optimal configurations for 0.90.x HBase. The - hbase-default.xml file is now bundled into the - HBase jar and read from there. If you would like to review - the content of this file, see it in the src tree at - src/main/resources/hbase-default.xml or - see . - - - Finally, if upgrading from 0.20.x, check your - .META. schema in the shell. In the past we would - recommend that users run with a 16kb - MEMSTORE_FLUSHSIZE. - Run hbase> scan '-ROOT-' in the shell. This will output - the current .META. schema. Check - MEMSTORE_FLUSHSIZE size. Is it 16kb (16384)? If so, you will - need to change this (The 'normal'/default value is 64MB (67108864)). - Run the script bin/set_meta_memstore_size.rb. - This will make the necessary edit to your .META. schema. - Failure to run this change will make for a slow cluster - - See HBASE-3499 Users upgrading to 0.90.0 need to have their .META. table updated with the right MEMSTORE_SIZE - - - . - - -
+
+ Upgrading from 0.94.x to 0.96.x + The Singularity + You will have to stop your old 0.94 cluster completely to upgrade. If you are replicating + between clusters, both clusters will have to go down to upgrade. Make sure it is a clean shutdown + so there are no WAL files laying around (TODO: Can 0.96 read 0.94 WAL files?). Make sure + zookeeper is cleared of state. All clients must be upgraded to 0.96 too. + + The API has changed in a few areas; in particular how you use coprocessors (TODO: MapReduce too?) + + TODO: Write about 3.4 zk ensemble and multi support +
+
+ Upgrading from 0.92.x to 0.94.x + 0.92 and 0.94 are interface compatible. You can do a rolling upgrade between these versions. + +
Upgrading from 0.90.x to 0.92.x Upgrade Guide @@ -170,7 +150,7 @@ The block size default size has been changed in 0.92.0 from 0.2 (20 percent of h
Experimental off-heap cache -A new cache was contributed to 0.92.0 to act as a solution between using the “on-heap” cache which is the current LRU cache the region servers have and the operating system cache which is out of our control. +A new cache was contributed to 0.92.0 to act as a solution between using the “on-heap” cache which is the current LRU cache the region servers have and the operating system cache which is out of our control. To enable, set “-XX:MaxDirectMemorySize” in hbase-env.sh to the value for maximum direct memory size and specify hbase.offheapcache.percentage in hbase-site.xml with the percentage that you want to dedicate to off-heap cache. This should only be set for servers and not for clients. Use at your own risk. See this blog post for additional information on this new experimental feature: http://www.cloudera.com/blog/2012/01/caching-in-hbase-slabcache/ @@ -194,8 +174,48 @@ See this blog post for additional information on this new experimental feature: 0.92.0 stores data in a new format, . As HBase runs, it will move all your data from HFile v1 to HFile v2 format. This auto-migration will run in the background as flushes and compactions run. HFile V2 allows HBase run with larger regions/files. In fact, we encourage that all HBasers going forward tend toward Facebook axiom #1, run with larger, fewer regions. -If you have lots of regions now -- more than 100s per host -- you should look into setting your region size up after you move to 0.92.0 (In 0.92.0, default size is not 1G, up from 256M), and then running online merge tool (See “HBASE-1621 merge tool should work on online cluster, but disabled table”). +If you have lots of regions now -- more than 100s per host -- you should look into setting your region size up after you move to 0.92.0 (In 0.92.0, default size is now 1G, up from 256M), and then running online merge tool (See “HBASE-1621 merge tool should work on online cluster, but disabled table”).
+
+ Upgrading to HBase 0.90.x from 0.20.x or 0.89.x + This version of 0.90.x HBase can be started on data written by + HBase 0.20.x or HBase 0.89.x. There is no need of a migration step. + HBase 0.89.x and 0.90.x does write out the name of region directories + differently -- it names them with a md5 hash of the region name rather + than a jenkins hash -- so this means that once started, there is no + going back to HBase 0.20.x. + + + Be sure to remove the hbase-default.xml from + your conf + directory on upgrade. A 0.20.x version of this file will have + sub-optimal configurations for 0.90.x HBase. The + hbase-default.xml file is now bundled into the + HBase jar and read from there. If you would like to review + the content of this file, see it in the src tree at + src/main/resources/hbase-default.xml or + see . + + + Finally, if upgrading from 0.20.x, check your + .META. schema in the shell. In the past we would + recommend that users run with a 16kb + MEMSTORE_FLUSHSIZE. + Run hbase> scan '-ROOT-' in the shell. This will output + the current .META. schema. Check + MEMSTORE_FLUSHSIZE size. Is it 16kb (16384)? If so, you will + need to change this (The 'normal'/default value is 64MB (67108864)). + Run the script bin/set_meta_memstore_size.rb. + This will make the necessary edit to your .META. schema. + Failure to run this change will make for a slow cluster + + See HBASE-3499 Users upgrading to 0.90.0 need to have their .META. table updated with the right MEMSTORE_SIZE + + + . + + +
diff --git a/src/docbkx/zookeeper.xml b/src/docbkx/zookeeper.xml new file mode 100644 index 000000000000..d6301e26fa1c --- /dev/null +++ b/src/docbkx/zookeeper.xml @@ -0,0 +1,595 @@ + + + + + ZooKeeper<indexterm> + <primary>ZooKeeper</primary> + </indexterm> + + A distributed Apache HBase (TM) installation depends on a running ZooKeeper cluster. + All participating nodes and clients need to be able to access the + running ZooKeeper ensemble. Apache HBase by default manages a ZooKeeper + "cluster" for you. It will start and stop the ZooKeeper ensemble + as part of the HBase start/stop process. You can also manage the + ZooKeeper ensemble independent of HBase and just point HBase at + the cluster it should use. To toggle HBase management of + ZooKeeper, use the HBASE_MANAGES_ZK variable in + conf/hbase-env.sh. This variable, which + defaults to true, tells HBase whether to + start/stop the ZooKeeper ensemble servers as part of HBase + start/stop. + + When HBase manages the ZooKeeper ensemble, you can specify + ZooKeeper configuration using its native + zoo.cfg file, or, the easier option is to + just specify ZooKeeper options directly in + conf/hbase-site.xml. A ZooKeeper + configuration option can be set as a property in the HBase + hbase-site.xml XML configuration file by + prefacing the ZooKeeper option name with + hbase.zookeeper.property. For example, the + clientPort setting in ZooKeeper can be changed + by setting the + hbase.zookeeper.property.clientPort property. + For all default values used by HBase, including ZooKeeper + configuration, see . Look for the + hbase.zookeeper.property prefix + For the full list of ZooKeeper configurations, see + ZooKeeper's zoo.cfg. HBase does not ship + with a zoo.cfg so you will need to browse + the conf directory in an appropriate + ZooKeeper download. + + + You must at least list the ensemble servers in + hbase-site.xml using the + hbase.zookeeper.quorum property. This property + defaults to a single ensemble member at + localhost which is not suitable for a fully + distributed HBase. (It binds to the local machine only and remote + clients will not be able to connect). + How many ZooKeepers should I run? + + You can run a ZooKeeper ensemble that comprises 1 node + only but in production it is recommended that you run a + ZooKeeper ensemble of 3, 5 or 7 machines; the more members an + ensemble has, the more tolerant the ensemble is of host + failures. Also, run an odd number of machines. In ZooKeeper, + an even number of peers is supported, but it is normally not used + because an even sized ensemble requires, proportionally, more peers + to form a quorum than an odd sized ensemble requires. For example, an + ensemble with 4 peers requires 3 to form a quorum, while an ensemble with + 5 also requires 3 to form a quorum. Thus, an ensemble of 5 allows 2 peers to + fail, and thus is more fault tolerant than the ensemble of 4, which allows + only 1 down peer. + + Give each ZooKeeper server around 1GB of RAM, and if possible, its own + dedicated disk (A dedicated disk is the best thing you can do + to ensure a performant ZooKeeper ensemble). For very heavily + loaded clusters, run ZooKeeper servers on separate machines + from RegionServers (DataNodes and TaskTrackers). + + + For example, to have HBase manage a ZooKeeper quorum on + nodes rs{1,2,3,4,5}.example.com, bound to + port 2222 (the default is 2181) ensure + HBASE_MANAGE_ZK is commented out or set to + true in conf/hbase-env.sh + and then edit conf/hbase-site.xml and set + hbase.zookeeper.property.clientPort and + hbase.zookeeper.quorum. You should also set + hbase.zookeeper.property.dataDir to other than + the default as the default has ZooKeeper persist data under + /tmp which is often cleared on system + restart. In the example below we have ZooKeeper persist to + /user/local/zookeeper. + <configuration> + ... + <property> + <name>hbase.zookeeper.property.clientPort</name> + <value>2222</value> + <description>Property from ZooKeeper's config zoo.cfg. + The port at which the clients will connect. + </description> + </property> + <property> + <name>hbase.zookeeper.quorum</name> + <value>rs1.example.com,rs2.example.com,rs3.example.com,rs4.example.com,rs5.example.com</value> + <description>Comma separated list of servers in the ZooKeeper Quorum. + For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". + By default this is set to localhost for local and pseudo-distributed modes + of operation. For a fully-distributed setup, this should be set to a full + list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh + this is the list of servers which we will start/stop ZooKeeper on. + </description> + </property> + <property> + <name>hbase.zookeeper.property.dataDir</name> + <value>/usr/local/zookeeper</value> + <description>Property from ZooKeeper's config zoo.cfg. + The directory where the snapshot is stored. + </description> + </property> + ... + </configuration> + + ZooKeeper Maintenance + Be sure to set up the data dir cleaner described under + Zookeeper Maintenance else you could + have 'interesting' problems a couple of months in; i.e. zookeeper could start + dropping sessions if it has to run through a directory of hundreds of thousands of + logs which is wont to do around leader reelection time -- a process rare but run on + occasion whether because a machine is dropped or happens to hiccup. + + +
+ Using existing ZooKeeper ensemble + + To point HBase at an existing ZooKeeper cluster, one that + is not managed by HBase, set HBASE_MANAGES_ZK + in conf/hbase-env.sh to false + + ... + # Tell HBase whether it should manage its own instance of Zookeeper or not. + export HBASE_MANAGES_ZK=false Next set ensemble locations + and client port, if non-standard, in + hbase-site.xml, or add a suitably + configured zoo.cfg to HBase's + CLASSPATH. HBase will prefer the + configuration found in zoo.cfg over any + settings in hbase-site.xml. + + When HBase manages ZooKeeper, it will start/stop the + ZooKeeper servers as a part of the regular start/stop scripts. + If you would like to run ZooKeeper yourself, independent of + HBase start/stop, you would do the following + + +${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper + + + Note that you can use HBase in this manner to spin up a + ZooKeeper cluster, unrelated to HBase. Just make sure to set + HBASE_MANAGES_ZK to false + if you want it to stay up across HBase restarts so that when + HBase shuts down, it doesn't take ZooKeeper down with it. + + For more information about running a distinct ZooKeeper + cluster, see the ZooKeeper Getting + Started Guide. Additionally, see the ZooKeeper Wiki or the + ZooKeeper documentation + for more information on ZooKeeper sizing. + +
+ + +
+ SASL Authentication with ZooKeeper + Newer releases of Apache HBase (>= 0.92) will + support connecting to a ZooKeeper Quorum that supports + SASL authentication (which is available in Zookeeper + versions 3.4.0 or later). + + This describes how to set up HBase to mutually + authenticate with a ZooKeeper Quorum. ZooKeeper/HBase + mutual authentication (HBASE-2418) + is required as part of a complete secure HBase configuration + (HBASE-3025). + + For simplicity of explication, this section ignores + additional configuration required (Secure HDFS and Coprocessor + configuration). It's recommended to begin with an + HBase-managed Zookeeper configuration (as opposed to a + standalone Zookeeper quorum) for ease of learning. + + +
Operating System Prerequisites
+ + + You need to have a working Kerberos KDC setup. For + each $HOST that will run a ZooKeeper + server, you should have a principle + zookeeper/$HOST. For each such host, + add a service key (using the kadmin or + kadmin.local tool's ktadd + command) for zookeeper/$HOST and copy + this file to $HOST, and make it + readable only to the user that will run zookeeper on + $HOST. Note the location of this file, + which we will use below as + $PATH_TO_ZOOKEEPER_KEYTAB. + + + + Similarly, for each $HOST that will run + an HBase server (master or regionserver), you should + have a principle: hbase/$HOST. For each + host, add a keytab file called + hbase.keytab containing a service + key for hbase/$HOST, copy this file to + $HOST, and make it readable only to the + user that will run an HBase service on + $HOST. Note the location of this file, + which we will use below as + $PATH_TO_HBASE_KEYTAB. + + + + Each user who will be an HBase client should also be + given a Kerberos principal. This principal should + usually have a password assigned to it (as opposed to, + as with the HBase servers, a keytab file) which only + this user knows. The client's principal's + maxrenewlife should be set so that it can + be renewed enough so that the user can complete their + HBase client processes. For example, if a user runs a + long-running HBase client process that takes at most 3 + days, we might create this user's principal within + kadmin with: addprinc -maxrenewlife + 3days. The Zookeeper client and server + libraries manage their own ticket refreshment by + running threads that wake up periodically to do the + refreshment. + + + On each host that will run an HBase client + (e.g. hbase shell), add the following + file to the HBase home directory's conf + directory: + + + Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=false + useTicketCache=true; + }; + + + We'll refer to this JAAS configuration file as + $CLIENT_CONF below. + +
+ HBase-managed Zookeeper Configuration + + On each node that will run a zookeeper, a + master, or a regionserver, create a JAAS + configuration file in the conf directory of the node's + HBASE_HOME directory that looks like the + following: + + + Server { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + keyTab="$PATH_TO_ZOOKEEPER_KEYTAB" + storeKey=true + useTicketCache=false + principal="zookeeper/$HOST"; + }; + Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + useTicketCache=false + keyTab="$PATH_TO_HBASE_KEYTAB" + principal="hbase/$HOST"; + }; + + + where the $PATH_TO_HBASE_KEYTAB and + $PATH_TO_ZOOKEEPER_KEYTAB files are what + you created above, and $HOST is the hostname for that + node. + + The Server section will be used by + the Zookeeper quorum server, while the + Client section will be used by the HBase + master and regionservers. The path to this file should + be substituted for the text $HBASE_SERVER_CONF + in the hbase-env.sh + listing below. + + + The path to this file should be substituted for the + text $CLIENT_CONF in the + hbase-env.sh listing below. + + + Modify your hbase-env.sh to include the + following: + + + export HBASE_OPTS="-Djava.security.auth.login.config=$CLIENT_CONF" + export HBASE_MANAGES_ZK=true + export HBASE_ZOOKEEPER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + export HBASE_MASTER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + + + where $HBASE_SERVER_CONF and + $CLIENT_CONF are the full paths to the + JAAS configuration files created above. + + Modify your hbase-site.xml on each node + that will run zookeeper, master or regionserver to contain: + + + + hbase.zookeeper.quorum + $ZK_NODES + + + hbase.cluster.distributed + true + + + hbase.zookeeper.property.authProvider.1 + org.apache.zookeeper.server.auth.SASLAuthenticationProvider + + + hbase.zookeeper.property.kerberos.removeHostFromPrincipal + true + + + hbase.zookeeper.property.kerberos.removeRealmFromPrincipal + true + + + ]]> + + where $ZK_NODES is the + comma-separated list of hostnames of the Zookeeper + Quorum hosts. + + Start your hbase cluster by running one or more + of the following set of commands on the appropriate + hosts: + + + + bin/hbase zookeeper start + bin/hbase master start + bin/hbase regionserver start + + +
+ +
External Zookeeper Configuration + Add a JAAS configuration file that looks like: + + + Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + useTicketCache=false + keyTab="$PATH_TO_HBASE_KEYTAB" + principal="hbase/$HOST"; + }; + + + where the $PATH_TO_HBASE_KEYTAB is the keytab + created above for HBase services to run on this host, and $HOST is the + hostname for that node. Put this in the HBase home's + configuration directory. We'll refer to this file's + full pathname as $HBASE_SERVER_CONF below. + + Modify your hbase-env.sh to include the following: + + + export HBASE_OPTS="-Djava.security.auth.login.config=$CLIENT_CONF" + export HBASE_MANAGES_ZK=false + export HBASE_MASTER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + + + + Modify your hbase-site.xml on each node + that will run a master or regionserver to contain: + + + + hbase.zookeeper.quorum + $ZK_NODES + + + hbase.cluster.distributed + true + + + ]]> + + + where $ZK_NODES is the + comma-separated list of hostnames of the Zookeeper + Quorum hosts. + + + Add a zoo.cfg for each Zookeeper Quorum host containing: + + authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider + kerberos.removeHostFromPrincipal=true + kerberos.removeRealmFromPrincipal=true + + + Also on each of these hosts, create a JAAS configuration file containing: + + + Server { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + keyTab="$PATH_TO_ZOOKEEPER_KEYTAB" + storeKey=true + useTicketCache=false + principal="zookeeper/$HOST"; + }; + + + where $HOST is the hostname of each + Quorum host. We will refer to the full pathname of + this file as $ZK_SERVER_CONF below. + + + + + Start your Zookeepers on each Zookeeper Quorum host with: + + + SERVER_JVMFLAGS="-Djava.security.auth.login.config=$ZK_SERVER_CONF" bin/zkServer start + + + + + + Start your HBase cluster by running one or more of the following set of commands on the appropriate nodes: + + + + bin/hbase master start + bin/hbase regionserver start + + + +
+ +
+ Zookeeper Server Authentication Log Output + If the configuration above is successful, + you should see something similar to the following in + your Zookeeper server logs: + +11/12/05 22:43:39 INFO zookeeper.Login: successfully logged in. +11/12/05 22:43:39 INFO server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 +11/12/05 22:43:39 INFO zookeeper.Login: TGT refresh thread started. +11/12/05 22:43:39 INFO zookeeper.Login: TGT valid starting at: Mon Dec 05 22:43:39 UTC 2011 +11/12/05 22:43:39 INFO zookeeper.Login: TGT expires: Tue Dec 06 22:43:39 UTC 2011 +11/12/05 22:43:39 INFO zookeeper.Login: TGT refresh sleeping until: Tue Dec 06 18:36:42 UTC 2011 +.. +11/12/05 22:43:59 INFO auth.SaslServerCallbackHandler: + Successfully authenticated client: authenticationID=hbase/ip-10-166-175-249.us-west-1.compute.internal@HADOOP.LOCALDOMAIN; + authorizationID=hbase/ip-10-166-175-249.us-west-1.compute.internal@HADOOP.LOCALDOMAIN. +11/12/05 22:43:59 INFO auth.SaslServerCallbackHandler: Setting authorizedID: hbase +11/12/05 22:43:59 INFO server.ZooKeeperServer: adding SASL authorization for authorizationID: hbase + + + + +
+ +
+ Zookeeper Client Authentication Log Output + On the Zookeeper client side (HBase master or regionserver), + you should see something similar to the following: + + +11/12/05 22:43:59 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-166-175-249.us-west-1.compute.internal:2181 sessionTimeout=180000 watcher=master:60000 +11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.166.175.249:2181 +11/12/05 22:43:59 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 14851@ip-10-166-175-249 +11/12/05 22:43:59 INFO zookeeper.Login: successfully logged in. +11/12/05 22:43:59 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. +11/12/05 22:43:59 INFO zookeeper.Login: TGT refresh thread started. +11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-166-175-249.us-west-1.compute.internal/10.166.175.249:2181, initiating session +11/12/05 22:43:59 INFO zookeeper.Login: TGT valid starting at: Mon Dec 05 22:43:59 UTC 2011 +11/12/05 22:43:59 INFO zookeeper.Login: TGT expires: Tue Dec 06 22:43:59 UTC 2011 +11/12/05 22:43:59 INFO zookeeper.Login: TGT refresh sleeping until: Tue Dec 06 18:30:37 UTC 2011 +11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-166-175-249.us-west-1.compute.internal/10.166.175.249:2181, sessionid = 0x134106594320000, negotiated timeout = 180000 + + +
+ +
+ Configuration from Scratch + + This has been tested on the current standard Amazon + Linux AMI. First setup KDC and principals as + described above. Next checkout code and run a sanity + check. + + + git clone git://git.apache.org/hbase.git + cd hbase + mvn clean test -Dtest=TestZooKeeperACL + + + Then configure HBase as described above. + Manually edit target/cached_classpath.txt (see below).. + + + bin/hbase zookeeper & + bin/hbase master & + bin/hbase regionserver & + +
+ + +
+ Future improvements + +
Fix target/cached_classpath.txt + + You must override the standard hadoop-core jar file from the + target/cached_classpath.txt + file with the version containing the HADOOP-7070 fix. You can use the following script to do this: + + + echo `find ~/.m2 -name "*hadoop-core*7070*SNAPSHOT.jar"` ':' `cat target/cached_classpath.txt` | sed 's/ //g' > target/tmp.txt + mv target/tmp.txt target/cached_classpath.txt + + + + +
+ +
+ Set JAAS configuration + programmatically + + + This would avoid the need for a separate Hadoop jar + that fixes HADOOP-7070. +
+ +
+ Elimination of + <code>kerberos.removeHostFromPrincipal</code> and + <code>kerberos.removeRealmFromPrincipal</code> +
+ +
+ + +
+ + + + + diff --git a/src/examples/healthcheck/healthcheck.sh b/src/examples/healthcheck/healthcheck.sh new file mode 100644 index 000000000000..584636054ddc --- /dev/null +++ b/src/examples/healthcheck/healthcheck.sh @@ -0,0 +1,84 @@ +#!/bin/bash + # Licensed to the Apache Software Foundation (ASF) under one + # or more contributor license agreements. See the NOTICE file + # distributed with this work for additional information + # regarding copyright ownership. The ASF licenses this file + # to you under the Apache License, Version 2.0 (the + # "License"); you may not use this file except in compliance + # with the License. You may obtain a copy of the License at + # + # http://www.apache.org/licenses/LICENSE-2.0 + # + # Unless required by applicable law or agreed to in writing, software + # distributed under the License is distributed on an "AS IS" BASIS, + # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + # See the License for the specific language governing permissions and + # limitations under the License. + + # This is an example script for checking health of a node ( master or region server). + # The health chore script should essentially output an message containing "ERROR" at an undesirable + # outcome of the checks in the script. + +err=0; + +function check_disks { + +for m in `awk '$3~/ext3/ {printf" %s ",$2}' /etc/fstab` ; do + fsdev="" + fsdev=`awk -v m=$m '$2==m {print $1}' /proc/mounts`; + if [ -z "$fsdev" ] ; then + msg_="$msg_ $m(u)" + else + msg_="$msg_`awk -v m=$m '$2==m { if ( $4 ~ /^ro,/ ) {printf"%s(ro)",$2 } ; }' /proc/mounts`" + fi + done + + if [ -z "$msg_" ] ; then + echo "disks ok" ; exit 0 + else + echo "$msg_" ; exit 2 + fi + +} + +function check_link { + /usr/bin/snmpwalk -t 5 -Oe -Oq -Os -v 1 -c public localhost if | \ + awk ' { + split($1,a,".") ; + if ( a[1] == "ifIndex" ) { ifIndex[a[2]] = $2 } + if ( a[1] == "ifDescr" ) { ifDescr[a[2]] = $2 } + if ( a[1] == "ifType" ) { ifType[a[2]] = $2 } + if ( a[1] == "ifSpeed" ) { ifSpeed[a[2]] = $2 } + if ( a[1] == "ifAdminStatus" ) { ifAdminStatus[a[2]] = $2 } + if ( a[1] == "ifOperStatus" ) { ifOperStatus[a[2]] = $2 } + } + END { + up=0; + for (i in ifIndex ) { + if ( ifType[i] == 6 && ifAdminStatus[i] == 1 && ifOperStatus[i] == 1 && ifSpeed[i] == 1000000000 ) { + up=i; + } + } + if ( up == 0 ) { print "check link" ; exit 2 } + else { print ifDescr[up],"ok" } + }' + exit $? ; +} + +for check in disks link ; do + msg=`check_${check}` ; + if [ $? -eq 0 ] ; then + ok_msg="$ok_msg$msg," + else + err_msg="$err_msg$msg," + fi +done + +if [ ! -z "$err_msg" ] ; then + echo -n "ERROR $err_msg " +fi +if [ ! -z "$ok_msg" ] ; then + echo -n "OK: $ok_msg" +fi +echo +exit 0 diff --git a/src/examples/mapreduce/org/apache/hadoop/hbase/mapreduce/IndexBuilder.java b/src/examples/mapreduce/org/apache/hadoop/hbase/mapreduce/IndexBuilder.java index 31c1b38dfe47..0d092d090c27 100644 --- a/src/examples/mapreduce/org/apache/hadoop/hbase/mapreduce/IndexBuilder.java +++ b/src/examples/mapreduce/org/apache/hadoop/hbase/mapreduce/IndexBuilder.java @@ -1,6 +1,4 @@ -/** - * Copyright 2009 The Apache Software Foundation - * +/* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information diff --git a/src/examples/mapreduce/org/apache/hadoop/hbase/mapreduce/SampleUploader.java b/src/examples/mapreduce/org/apache/hadoop/hbase/mapreduce/SampleUploader.java index 5629ccac1946..108d65283345 100644 --- a/src/examples/mapreduce/org/apache/hadoop/hbase/mapreduce/SampleUploader.java +++ b/src/examples/mapreduce/org/apache/hadoop/hbase/mapreduce/SampleUploader.java @@ -1,6 +1,4 @@ -/** - * Copyright 2009 The Apache Software Foundation - * +/* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information diff --git a/src/examples/thrift/DemoClient.cpp b/src/examples/thrift/DemoClient.cpp index 06cbc4460d17..e845669cd89d 100644 --- a/src/examples/thrift/DemoClient.cpp +++ b/src/examples/thrift/DemoClient.cpp @@ -1,6 +1,4 @@ /** - * Copyright 2008 The Apache Software Foundation - * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information diff --git a/src/examples/thrift/DemoClient.java b/src/examples/thrift/DemoClient.java index bb03fccd4338..036d7fd022ad 100644 --- a/src/examples/thrift/DemoClient.java +++ b/src/examples/thrift/DemoClient.java @@ -1,6 +1,4 @@ -/** - * Copyright 2008 The Apache Software Foundation - * +/* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information diff --git a/src/examples/thrift/DemoClient.php b/src/examples/thrift/DemoClient.php index 669f2b6fc2fd..93d79d43f509 100644 --- a/src/examples/thrift/DemoClient.php +++ b/src/examples/thrift/DemoClient.php @@ -1,7 +1,5 @@ new ( column => "entry:$key", value => $valid ) ]; # This is another way to use the Mutation class my $mutation = Hbase::Mutation->new (); $mutation->{column} = "entry:$key"; @@ -151,7 +148,7 @@ ($) # non-utf8 is not allowed in row names eval { - $mutations = [ Hbase::Mutation->new ( column => "entry:$key", value => $invalid ) ]; + $mutations = [ Hbase::Mutation->new ( { column => "entry:$key", value => $invalid } ) ]; # this can throw a TApplicationException (HASH) error $client->mutateRow ($demo_table, $key, $mutations); die ("shouldn't get here!"); diff --git a/src/examples/thrift/DemoClient.py b/src/examples/thrift/DemoClient.py index eabbbe84371d..723d7a894c29 100755 --- a/src/examples/thrift/DemoClient.py +++ b/src/examples/thrift/DemoClient.py @@ -1,6 +1,5 @@ #!/usr/bin/python -'''Copyright 2008 The Apache Software Foundation - +''' Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information diff --git a/src/examples/thrift/DemoClient.rb b/src/examples/thrift/DemoClient.rb index 2b7b5e7f934d..ea5acad926b2 100644 --- a/src/examples/thrift/DemoClient.rb +++ b/src/examples/thrift/DemoClient.rb @@ -1,7 +1,5 @@ #!/usr/bin/ruby -# Copyright 2008 The Apache Software Foundation -# # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information diff --git a/src/examples/thrift/Makefile b/src/examples/thrift/Makefile index 691a1e981ae9..da2f3010a302 100644 --- a/src/examples/thrift/Makefile +++ b/src/examples/thrift/Makefile @@ -1,5 +1,3 @@ -# Copyright 2008 The Apache Software Foundation -# # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information diff --git a/src/examples/thrift2/DemoClient.java b/src/examples/thrift2/DemoClient.java index d5b805c83e21..828b4c01d357 100644 --- a/src/examples/thrift2/DemoClient.java +++ b/src/examples/thrift2/DemoClient.java @@ -1,6 +1,4 @@ -/** - * Copyright 2011 The Apache Software Foundation - * +/* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information diff --git a/src/examples/thrift2/DemoClient.py b/src/examples/thrift2/DemoClient.py index 67abc5b5be04..3a3ebbbbbc93 100644 --- a/src/examples/thrift2/DemoClient.py +++ b/src/examples/thrift2/DemoClient.py @@ -1,6 +1,4 @@ """ - Copyright 2011 The Apache Software Foundation - Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information diff --git a/src/main/appended-resources/META-INF/LICENSE b/src/main/appended-resources/META-INF/LICENSE new file mode 100644 index 000000000000..6ec590ec20e6 --- /dev/null +++ b/src/main/appended-resources/META-INF/LICENSE @@ -0,0 +1,37 @@ +---- +This project incorporates portions of the 'Protocol Buffers' project avaialble +under a '3-clause BSD' license. + + Copyright 2008, Google Inc. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following disclaimer + in the documentation and/or other materials provided with the + distribution. + * Neither the name of Google Inc. nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + Code generated by the Protocol Buffer compiler is owned by the owner + of the input file used when generating it. This code is not + standalone and requires a support library to be linked with it. This + support library is itself covered by the above license. diff --git a/src/main/appended-resources/META-INF/NOTICE b/src/main/appended-resources/META-INF/NOTICE new file mode 100644 index 000000000000..d8f61099f299 --- /dev/null +++ b/src/main/appended-resources/META-INF/NOTICE @@ -0,0 +1,6 @@ +-- +This product incorporates portions of the 'Hadoop' project + +Copyright 2007-2009 The Apache Software Foundation + +Licensed under the Apache License v2.0 diff --git a/src/main/jamon/org/apache/hadoop/hbase/tmpl/common/TaskMonitorTmpl.jamon b/src/main/jamon/org/apache/hadoop/hbase/tmpl/common/TaskMonitorTmpl.jamon index 4379ef59b5cd..0304154c00b6 100644 --- a/src/main/jamon/org/apache/hadoop/hbase/tmpl/common/TaskMonitorTmpl.jamon +++ b/src/main/jamon/org/apache/hadoop/hbase/tmpl/common/TaskMonitorTmpl.jamon @@ -1,6 +1,4 @@ <%doc> -Copyright 2011 The Apache Software Foundation - Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information diff --git a/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon b/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon index 0dc0691a894c..e06c5e6124f5 100644 --- a/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon +++ b/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon @@ -1,6 +1,4 @@ <%doc> -Copyright 2011 The Apache Software Foundation - Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information diff --git a/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon b/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon new file mode 100644 index 000000000000..ca8308b5e3fc --- /dev/null +++ b/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon @@ -0,0 +1,77 @@ +<%doc> + +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +<%args> +HMaster master; + +<%import> +java.util.*; +org.apache.hadoop.hbase.util.Bytes; +org.apache.hadoop.hbase.ServerName; +org.apache.hadoop.hbase.ClusterStatus; +org.apache.hadoop.hbase.master.HMaster; +org.apache.hadoop.hbase.master.ServerManager; +org.apache.hadoop.hbase.master.AssignmentManager; +org.apache.hadoop.hbase.master.ActiveMasterManager; + +<%java> +Collection masters; + +if (master.isActiveMaster()) { + ClusterStatus status = master.getClusterStatus(); + masters = status.getBackupMasters(); +} else{ + ServerName sn = master.getActiveMasterManager().getActiveMaster() ; + assert sn != null : "Failed to retreive master's ServerName!"; + + List serverNames = new ArrayList(1); + serverNames.add(sn); + masters = Collections.unmodifiableCollection(serverNames); +} + + +<%java> +ServerName [] serverNames = masters.toArray(new ServerName[masters.size()]); + +<%if (!master.isActiveMaster()) %> +

Master

+ /master-status" target="_blank"><% serverNames[0].getHostname() %> +<%else> +

Backup Masters

+ + + + + + + + <%java> + Arrays.sort(serverNames); + for (ServerName serverName: serverNames) { + + + + + + + <%java> + } + + +
ServerNamePortStart Time
/master-status" target="_blank"><% serverName.getHostname() %><% serverName.getPort() %><% new Date(serverName.getStartcode()) %>
Total:<% (masters != null) ? masters.size() : 0 %>
+ diff --git a/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon b/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon index 69434f7dd867..c1fffedcc124 100644 --- a/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon +++ b/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon @@ -1,6 +1,4 @@ <%doc> -Copyright 2011 The Apache Software Foundation - Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information @@ -28,6 +26,8 @@ Set deadServers = null; boolean showAppendWarning = false; String filter = "general"; String format = "html"; +ServerManager serverManager = null; +AssignmentManager assignmentManager = null; <%import> java.util.*; @@ -36,6 +36,8 @@ org.apache.hadoop.hbase.util.Bytes; org.apache.hadoop.hbase.util.JvmVersion; org.apache.hadoop.hbase.util.FSUtils; org.apache.hadoop.hbase.master.HMaster; +org.apache.hadoop.hbase.master.AssignmentManager; +org.apache.hadoop.hbase.master.ServerManager; org.apache.hadoop.hbase.HConstants; org.apache.hadoop.hbase.HServerLoad; org.apache.hadoop.hbase.ServerName; @@ -43,11 +45,17 @@ org.apache.hadoop.hbase.client.HBaseAdmin; org.apache.hadoop.hbase.client.HConnectionManager; org.apache.hadoop.hbase.HTableDescriptor; org.apache.hadoop.hbase.HBaseConfiguration; +org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; <%if format.equals("json") %> <& ../common/TaskMonitorTmpl; filter = filter; format = "json" &> <%java return; %> +<%java> +ServerManager serverManager = master.getServerManager(); +AssignmentManager assignmentManager = master.getAssignmentManager(); + + + + + +<% if (isActionResultPage) { %> + HBase Master: <%= master.getServerName() %> + +<% } else { %> + Snapshot: <%= snapshotName %> +<% } %> + + + +<% if (isActionResultPage) { %> +

Snapshot action request...

+<% + if (action.equals("restore")) { + hbadmin.restoreSnapshot(snapshotName); + %> Restore Snapshot request accepted. <% + } else if (action.equals("clone")) { + if (cloneName != null && cloneName.length() > 0) { + hbadmin.cloneSnapshot(snapshotName, cloneName); + %> Clone from Snapshot request accepted. <% + } else { + %> Clone from Snapshot request failed, No table name specified. <% + } + } +%> +

Go Back, or wait for the redirect. + +<% } else if (snapshot == null) { %> +

Snapshot "<%= snapshotName %>" does not exists

+ +
+

Go Back, or wait for the redirect. +<% } else { %> +

Snapshot: <%= snapshotName %>

+ +
+

Snapshot Attributes

+ + + + + + + + + + + + + + <% if (stats.isSnapshotCorrupted()) { %> + + <% } else { %> + + <% } %> + +
TableCreation TimeTypeFormat VersionState
<%= snapshot.getTable() %><%= new Date(snapshot.getCreationTime()) %><%= snapshot.getType() %><%= snapshot.getVersion() %>CORRUPTEDok
+

+ <%= stats.getStoreFilesCount() %> HFiles (<%= stats.getArchivedStoreFilesCount() %> in archive), + total size <%= StringUtils.humanReadableInt(stats.getStoreFilesSize()) %> + (<%= stats.getSharedStoreFilePercentage() %>% + <%= StringUtils.humanReadableInt(stats.getSharedStoreFilesSize()) %> shared with the source + table) +

+

+ <%= stats.getLogsCount() %> Logs, total size + <%= StringUtils.humanReadableInt(stats.getLogsSize()) %> +

+ <% if (stats.isSnapshotCorrupted()) { %> +

CORRUPTED Snapshot

+

+ <%= stats.getMissingStoreFilesCount() %> hfile(s) and + <%= stats.getMissingLogsCount() %> log(s) missing. +

+ <% } %> +<% + } // end else + +HConnectionManager.deleteConnection(hbadmin.getConfiguration()); +%> + + +<% if (!readOnly && action == null && snapshot != null) { %> +

+Actions: +

+

+ + + + + + + + + + + + + + + + + + + + + + +
+  New Table Name (clone): + This action will create a new table by cloning the snapshot content. + There are no copies of data involved. + And writing on the newly created table will not influence the snapshot data. +
 
+   Restore a specified snapshot. + The restore will replace the content of the original table, + bringing back the content to the snapshot state. + The table must be disabled.
+
+

+ +<% } %> + + diff --git a/src/main/resources/hbase-webapps/master/table.jsp b/src/main/resources/hbase-webapps/master/table.jsp index 811df46ddc51..3556ed5ec822 100644 --- a/src/main/resources/hbase-webapps/master/table.jsp +++ b/src/main/resources/hbase-webapps/master/table.jsp @@ -18,7 +18,7 @@ */ --%> <%@ page contentType="text/html;charset=UTF-8" - import="java.util.HashMap" + import="java.util.TreeMap" import="org.apache.hadoop.io.Writable" import="org.apache.hadoop.conf.Configuration" import="org.apache.hadoop.hbase.client.HTable" @@ -43,15 +43,13 @@ String tableName = request.getParameter("name"); HTable table = new HTable(conf, tableName); String tableHeader = "

Table Regions

"; - ServerName rl = master.getCatalogTracker().getRootLocation(); + ServerName rl = master.getCatalogTracker().waitForRoot(1); boolean showFragmentation = conf.getBoolean("hbase.master.ui.fragmentation.enabled", false); + boolean readOnly = conf.getBoolean("hbase.master.ui.readonly", false); Map frags = null; if (showFragmentation) { frags = FSUtils.getTableFragmentation(master); } - // HARDCODED FOR NOW TODO: FIX GET FROM ZK - // This port might be wrong if RS actually ended up using something else. - int infoPort = conf.getInt("hbase.regionserver.info.port", 60030); %> @@ -65,11 +63,15 @@ <% String action = request.getParameter("action"); String key = request.getParameter("key"); - if ( action != null ) { + if ( !readOnly && action != null ) { %> - + @@ -113,7 +115,7 @@ %> <%= tableHeader %> <% - String url = "http://" + rl.getHostname() + ":" + infoPort + "/"; + String url = "//" + rl.getHostname() + ":" + master.getRegionServerInfoPort(rl) + "/"; %> @@ -130,13 +132,13 @@ <% // NOTE: Presumes one meta region only. HRegionInfo meta = HRegionInfo.FIRST_META_REGIONINFO; - ServerName metaLocation = master.getCatalogTracker().getMetaLocation(); + ServerName metaLocation = master.getCatalogTracker().waitForMeta(1); for (int i = 0; i < 1; i++) { - String url = "http://" + metaLocation.getHostname() + ":" + infoPort + "/"; + String url = "//" + metaLocation.getHostname() + ":" + master.getRegionServerInfoPort(metaLocation) + "/"; %> - + <% } %> @@ -154,6 +156,11 @@ + + + + + <% if (showFragmentation) { %> @@ -163,7 +170,7 @@ <% } %>
NameRegion ServerStart KeyEnd KeyRequests
<%= tableName %>
<%= meta.getRegionNameAsString() %><%= metaLocation.getHostname().toString() + ":" + infoPort %><%= metaLocation.getHostname().toString() + ":" + metaLocation.getPort() %> -<%= Bytes.toString(meta.getStartKey()) %><%= Bytes.toString(meta.getEndKey()) %>
<%= hbadmin.isTableEnabled(table.getTableName()) %> Is the table enabled
Compaction<%= hbadmin.getCompactionState(table.getTableName()) %>Is the table compacting
Fragmentation
<% - Map regDistribution = new HashMap(); + Map regDistribution = new TreeMap(); Map regions = table.getRegionLocations(); if(regions != null && regions.size() > 0) { %> <%= tableHeader %> @@ -173,7 +180,7 @@ ServerName addr = hriEntry.getValue(); long req = 0; - String urlRegionServer = null; + String regionServer = null; if (addr != null) { HServerLoad sl = master.getServerManager().getLoad(addr); @@ -182,22 +189,19 @@ if (map.containsKey(regionInfo.getRegionName())) { req = map.get(regionInfo.getRegionName()).getRequestsCount(); } - // This port might be wrong if RS actually ended up using something else. - urlRegionServer = - "http://" + addr.getHostname().toString() + ":" + infoPort + "/"; - Integer i = regDistribution.get(urlRegionServer); + Integer i = regDistribution.get(addr); if (null == i) i = new Integer(0); - regDistribution.put(urlRegionServer, i+1); + regDistribution.put(addr, i+1); } } %> <%= Bytes.toStringBinary(regionInfo.getRegionName())%> <% - if (urlRegionServer != null) { + if (addr != null) { %> - <%= addr.getHostname().toString() + ":" + infoPort %> + "><%= addr %> <% } else { @@ -215,10 +219,12 @@

Regions by Region Server

<% - for (Map.Entry rdEntry : regDistribution.entrySet()) { + for (Map.Entry rdEntry : regDistribution.entrySet()) { + ServerName addr = rdEntry.getKey(); + String url = "//" + addr.getHostname() + ":" + master.getRegionServerInfoPort(addr) + "/"; %> - + <% } %> @@ -229,9 +235,11 @@ } } // end else -HConnectionManager.deleteConnection(hbadmin.getConfiguration(), false); +HConnectionManager.deleteConnection(hbadmin.getConfiguration()); %> + +<% if (!readOnly) { %>

Actions:

@@ -268,7 +276,7 @@ Actions:

Region ServerRegion Count
<%= rdEntry.getKey()%>"><%= rdEntry.getKey() %> <%= rdEntry.getValue()%>

- +<% } %> <% } %> diff --git a/src/main/resources/hbase-webapps/rest/index.html b/src/main/resources/hbase-webapps/rest/index.html new file mode 100644 index 000000000000..e4084b7c4887 --- /dev/null +++ b/src/main/resources/hbase-webapps/rest/index.html @@ -0,0 +1,20 @@ + + diff --git a/src/main/resources/hbase-webapps/rest/rest.jsp b/src/main/resources/hbase-webapps/rest/rest.jsp new file mode 100644 index 000000000000..ba9856c13a8f --- /dev/null +++ b/src/main/resources/hbase-webapps/rest/rest.jsp @@ -0,0 +1,74 @@ +<%-- +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +--%> +<%@ page contentType="text/html;charset=UTF-8" + import="org.apache.hadoop.conf.Configuration" + import="org.apache.hadoop.hbase.HBaseConfiguration" + import="org.apache.hadoop.hbase.util.VersionInfo" + import="java.util.Date" +%> + +<% +Configuration conf = (Configuration)getServletContext().getAttribute("hbase.conf"); +long startcode = conf.getLong("startcode", System.currentTimeMillis()); +String listenPort = conf.get("hbase.rest.port", "8080"); +String serverInfo = listenPort + "," + String.valueOf(startcode); +%> + + + + + +HBase REST Server + + + + + +

RESTServer: <%= serverInfo %>

+ +
+ +

Attributes

+ ++++ + + + +
Attribute NameValueDescription
HBase Version<%= VersionInfo.getVersion() %>, r<%= VersionInfo.getRevision() %>HBase version and revision
HBase Compiled<%= VersionInfo.getDate() %>, <%= VersionInfo.getUser() %>When HBase version was compiled and by whom
REST Server Start Time<%= new Date(startcode) %>Date stamp of when this REST server was started
+ +
+Apache HBase Wiki on REST + + + diff --git a/src/main/resources/hbase-webapps/thrift/index.html b/src/main/resources/hbase-webapps/thrift/index.html new file mode 100644 index 000000000000..9925269e8959 --- /dev/null +++ b/src/main/resources/hbase-webapps/thrift/index.html @@ -0,0 +1,20 @@ + + diff --git a/src/main/resources/hbase-webapps/thrift/thrift.jsp b/src/main/resources/hbase-webapps/thrift/thrift.jsp new file mode 100644 index 000000000000..eee99406b7db --- /dev/null +++ b/src/main/resources/hbase-webapps/thrift/thrift.jsp @@ -0,0 +1,80 @@ +<%-- +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +--%> +<%@ page contentType="text/html;charset=UTF-8" + import="org.apache.hadoop.conf.Configuration" + import="org.apache.hadoop.hbase.HBaseConfiguration" + import="org.apache.hadoop.hbase.util.VersionInfo" + import="java.util.Date" +%> + +<% +Configuration conf = (Configuration)getServletContext().getAttribute("hbase.conf"); +long startcode = conf.getLong("startcode", System.currentTimeMillis()); +String listenPort = conf.get("hbase.regionserver.thrift.port", "9090"); +String serverInfo = listenPort + "," + String.valueOf(startcode); +String implType = conf.get("hbase.regionserver.thrift.server.type", "threadpool"); +String compact = conf.get("hbase.regionserver.thrift.compact", "false"); +String framed = conf.get("hbase.regionserver.thrift.framed", "false"); +%> + + + + + +HBase Thrift Server + + + + + +

ThriftServer: <%= serverInfo %>

+ +
+ +

Attributes

+ ++++ + + + + + + +
Attribute NameValueDescription
HBase Version<%= VersionInfo.getVersion() %>, r<%= VersionInfo.getRevision() %>HBase version and revision
HBase Compiled<%= VersionInfo.getDate() %>, <%= VersionInfo.getUser() %>When HBase version was compiled and by whom
Thrift Server Start Time<%= new Date(startcode) %>Date stamp of when this Thrift server was started
Thrift Impl Type<%= implType %>Thrift RPC engine implementation type chosen by this Thrift server
Compact Protocol<%= compact %>Thrift RPC engine uses compact protocol
Framed Transport<%= framed %>Thrift RPC engine uses framed transport
+ +
+Apache HBase Wiki on Thrift + + + diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/XMLSchema.xsd b/src/main/resources/org/apache/hadoop/hbase/rest/XMLSchema.xsd index de4fff1761cd..9577ce23eac5 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/XMLSchema.xsd +++ b/src/main/resources/org/apache/hadoop/hbase/rest/XMLSchema.xsd @@ -98,7 +98,7 @@ - + @@ -111,8 +111,6 @@ - - @@ -166,6 +164,13 @@ + + + + + + + diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellMessage.proto b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellMessage.proto index a7bfe83b79ed..8d4515417f08 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellMessage.proto +++ b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellMessage.proto @@ -1,4 +1,3 @@ -// Copyright 2010 The Apache Software Foundation // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellSetMessage.proto b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellSetMessage.proto index dfdf125ed44d..4774a8d0e8f7 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellSetMessage.proto +++ b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellSetMessage.proto @@ -1,4 +1,3 @@ -// Copyright 2010 The Apache Software Foundation // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ColumnSchemaMessage.proto b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ColumnSchemaMessage.proto index 0a9a9af68657..05e33b648329 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ColumnSchemaMessage.proto +++ b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ColumnSchemaMessage.proto @@ -1,4 +1,3 @@ -// Copyright 2010 The Apache Software Foundation // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ScannerMessage.proto b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ScannerMessage.proto index 6ef3191b0a60..f7aca47131ff 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ScannerMessage.proto +++ b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ScannerMessage.proto @@ -1,4 +1,3 @@ -// Copyright 2010 The Apache Software Foundation // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/StorageClusterStatusMessage.proto b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/StorageClusterStatusMessage.proto index 2b032f7f4ac2..a0291b4e9e8a 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/StorageClusterStatusMessage.proto +++ b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/StorageClusterStatusMessage.proto @@ -1,4 +1,3 @@ -// Copyright 2010 The Apache Software Foundation // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file @@ -26,6 +25,13 @@ message StorageClusterStatus { optional int32 storefileSizeMB = 4; optional int32 memstoreSizeMB = 5; optional int32 storefileIndexSizeMB = 6; + optional int64 readRequestsCount = 7; + optional int64 writeRequestsCount = 8; + optional int32 rootIndexSizeKB = 9; + optional int32 totalStaticIndexSizeKB = 10; + optional int32 totalStaticBloomSizeKB = 11; + optional int64 totalCompactingKVs = 12; + optional int64 currentCompactedKVs = 13; } message Node { required string name = 1; // name:port diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableInfoMessage.proto b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableInfoMessage.proto index 5dd91204247a..674499c26874 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableInfoMessage.proto +++ b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableInfoMessage.proto @@ -1,4 +1,3 @@ -// Copyright 2010 The Apache Software Foundation // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableListMessage.proto b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableListMessage.proto index 2ce4d25ee8aa..fbd76ea029af 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableListMessage.proto +++ b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableListMessage.proto @@ -1,4 +1,3 @@ -// Copyright 2010 The Apache Software Foundation // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableSchemaMessage.proto b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableSchemaMessage.proto index d8177229d45b..47a4da589b95 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableSchemaMessage.proto +++ b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableSchemaMessage.proto @@ -1,4 +1,3 @@ -// Copyright 2010 The Apache Software Foundation // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file diff --git a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/VersionMessage.proto b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/VersionMessage.proto index 2404a2ebd35d..cc107b3d9a61 100644 --- a/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/VersionMessage.proto +++ b/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/VersionMessage.proto @@ -1,4 +1,3 @@ -// Copyright 2010 The Apache Software Foundation // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file diff --git a/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift b/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift index f698a6c8f67f..890339d5d9b9 100644 --- a/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift +++ b/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift @@ -110,13 +110,32 @@ struct BatchMutation { 2:list mutations } +/** + * For increments that are not incrementColumnValue + * equivalents. + */ +struct TIncrement { + 1:Text table, + 2:Text row, + 3:Text column, + 4:i64 ammount +} + +/** + * Holds column name and the cell. + */ +struct TColumn { + 1:Text columnName, + 2:TCell cell + } /** * Holds row name and then a map of columns to cells. */ struct TRowResult { 1:Text row, - 2:map columns + 2:optional map columns, + 3:optional list sortedColumns } /** @@ -128,7 +147,8 @@ struct TScan { 3:optional i64 timestamp, 4:optional list columns, 5:optional i32 caching, - 6:optional Text filterString + 6:optional Text filterString, + 7:optional bool sortColumns } // @@ -627,6 +647,23 @@ service Hbase { 3:map attributes ) throws (1:IOError io) + /** + * Increment a cell by the ammount. + * Increments can be applied async if hbase.regionserver.thrift.coalesceIncrement is set to true. + * False is the default. Turn to true if you need the extra performance and can accept some + * data loss if a thrift server dies with increments still in the queue. + */ + void increment( + /** The single increment to apply */ + 1:TIncrement increment + ) throws (1:IOError io) + + + void incrementRows( + /** The list of increments */ + 1:list increments + ) throws (1:IOError io) + /** * Completely delete the row's cells marked with a timestamp * equal-to or older than the passed timestamp. diff --git a/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift b/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift index 5bb0f51cbd3c..502cea925aa5 100644 --- a/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift +++ b/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift @@ -77,6 +77,21 @@ enum TDeleteType { DELETE_COLUMNS = 1 } +/** + * Specify Durability: + * - SKIP_WAL means do not write the Mutation to the WAL. + * - ASYNC_WAL means write the Mutation to the WAL asynchronously, + * - SYNC_WAL means write the Mutation to the WAL synchronously, + * - FSYNC_WAL means Write the Mutation to the WAL synchronously and force the entries to disk. + */ + +enum TDurability { + SKIP_WAL = 1, + ASYNC_WAL = 2, + SYNC_WAL = 3, + FSYNC_WAL = 4 +} + /** * Used to perform Get operations on a single row. * @@ -89,8 +104,6 @@ enum TDeleteType { * * If you specify a time range and a timestamp the range is ignored. * Timestamps on TColumns are ignored. - * - * TODO: Filter, Locks */ struct TGet { 1: required binary row, @@ -100,6 +113,8 @@ struct TGet { 4: optional TTimeRange timeRange, 5: optional i32 maxVersions, + 6: optional binary filterString, + 7: optional map attributes } /** @@ -117,7 +132,9 @@ struct TPut { 1: required binary row, 2: required list columnValues 3: optional i64 timestamp, - 4: optional bool writeToWal = 1 + 4: optional bool writeToWal, + 5: optional map attributes, + 6: optional TDurability durability } /** @@ -148,7 +165,9 @@ struct TDelete { 2: optional list columns, 3: optional i64 timestamp, 4: optional TDeleteType deleteType = 1, - 5: optional bool writeToWal = 1 + 5: optional bool writeToWal, + 6: optional map attributes, + 7: optional TDurability durability } /** @@ -174,6 +193,25 @@ struct TScan { 4: optional i32 caching, 5: optional i32 maxVersions=1, 6: optional TTimeRange timeRange, + 7: optional binary filterString, + 8: optional i32 batchSize, + 9: optional map attributes +} + +/** + * Atomic mutation for the specified row. It can be either Put or Delete. + */ +union TMutation { + 1: optional TPut put, + 2: optional TDelete deleteSingle, +} + +/** + * A TRowMutations object is used to apply a number of Mutations to a single row. + */ +struct TRowMutations { + 1: required binary row + 2: required list mutations } // @@ -394,10 +432,9 @@ service THBaseService { ) /** - * Closes the scanner. Should be called if you need to close - * the Scanner before all results are read. - * - * Exhausted scanners are closed automatically. + * Closes the scanner. Should be called to free server side resources timely. + * Typically close once the scanner is not needed anymore, i.e. after looping + * over it to get all the required rows. */ void closeScanner( /** the Id of the Scanner to close **/ @@ -409,4 +446,34 @@ service THBaseService { 2: TIllegalArgument ia ) + /** + * mutateRow performs multiple mutations atomically on a single row. + */ + void mutateRow( + /** table to apply the mutations */ + 1: required binary table, + + /** mutations to apply */ + 2: required TRowMutations rowMutations + ) throws (1: TIOError io) + + /** + * Get results for the provided TScan object. + * This helper function opens a scanner, get the results and close the scanner. + * + * @return between zero and numRows TResults + */ + list getScannerResults( + /** the table to get the Scanner for */ + 1: required binary table, + + /** the scan object to get a Scanner for */ + 2: required TScan scan, + + /** number of rows to return */ + 3: i32 numRows = 1 + ) throws ( + 1: TIOError io + ) + } diff --git a/src/main/ruby/hbase.rb b/src/main/ruby/hbase.rb index 2a4abcb2ed7a..a90976a726fe 100644 --- a/src/main/ruby/hbase.rb +++ b/src/main/ruby/hbase.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -40,6 +39,7 @@ module HBaseConstants NAME = org.apache.hadoop.hbase.HConstants::NAME VERSIONS = org.apache.hadoop.hbase.HConstants::VERSIONS IN_MEMORY = org.apache.hadoop.hbase.HConstants::IN_MEMORY + CONFIG = org.apache.hadoop.hbase.HConstants::CONFIG STOPROW = "STOPROW" STARTROW = "STARTROW" ENDROW = STOPROW @@ -56,6 +56,7 @@ module HBaseConstants SPLITS_FILE = 'SPLITS_FILE' SPLITALGO = 'SPLITALGO' NUMREGIONS = 'NUMREGIONS' + SKIP_FLUSH = 'SKIP_FLUSH' # Load constants from hbase java API def self.promote_constants(constants) diff --git a/src/main/ruby/hbase/admin.rb b/src/main/ruby/hbase/admin.rb index 601e7e12764f..29599e071932 100644 --- a/src/main/ruby/hbase/admin.rb +++ b/src/main/ruby/hbase/admin.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -20,6 +19,9 @@ include Java java_import org.apache.hadoop.hbase.util.Pair +java_import org.apache.hadoop.hbase.util.RegionSplitter +java_import org.apache.hadoop.hbase.util.Bytes +java_import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos::SnapshotDescription # Wrapper for org.apache.hadoop.hbase.client.HBaseAdmin @@ -39,8 +41,20 @@ def initialize(configuration, formatter) #---------------------------------------------------------------------------------------------- # Returns a list of tables in hbase - def list - @admin.listTables.map { |t| t.getNameAsString } + def list(regex = ".*") + begin + # Use the old listTables API first for compatibility with older servers + @admin.listTables(regex).map { |t| t.getNameAsString } + rescue => e + # listTables failed, try the new unprivileged getTableNames API if the cause was + # an AccessDeniedException + if e.cause.kind_of? org.apache.hadoop.ipc.RemoteException and e.cause.unwrapRemoteException().kind_of? org.apache.hadoop.hbase.security.AccessDeniedException + @admin.getTableNames(regex) + else + # Not an access control failure, re-raise + raise e + end + end end #---------------------------------------------------------------------------------------------- @@ -50,15 +64,25 @@ def flush(table_or_region_name) end #---------------------------------------------------------------------------------------------- - # Requests a table or region compaction - def compact(table_or_region_name) - @admin.compact(table_or_region_name) + # Requests a table or region or column family compaction + def compact(table_or_region_name, family = nil) + if family == nil + @admin.compact(table_or_region_name) + else + # We are compacting a column family within a region. + @admin.compact(table_or_region_name, family) + end end #---------------------------------------------------------------------------------------------- - # Requests a table or region major compaction - def major_compact(table_or_region_name) - @admin.majorCompact(table_or_region_name) + # Requests a table or region or column family major compaction + def major_compact(table_or_region_name, family = nil) + if family == nil + @admin.majorCompact(table_or_region_name) + else + # We are major compacting a column family within a region or table. + @admin.majorCompact(table_or_region_name, family) + end end #---------------------------------------------------------------------------------------------- @@ -88,7 +112,8 @@ def balancer() # Enable/disable balancer # Returns previous balancer switch setting. def balance_switch(enableDisable) - @admin.balanceSwitch(java.lang.Boolean::valueOf(enableDisable)) + @admin.setBalancerRunning( + java.lang.Boolean::valueOf(enableDisable), java.lang.Boolean::valueOf(false)) end #---------------------------------------------------------------------------------------------- @@ -140,8 +165,6 @@ def drop(table_name) raise ArgumentError, "Table #{table_name} is enabled. Disable it first.'" if enabled?(table_name) @admin.deleteTable(table_name) - flush(org.apache.hadoop.hbase.HConstants::META_TABLE_NAME) - major_compact(org.apache.hadoop.hbase.HConstants::META_TABLE_NAME) end #---------------------------------------------------------------------------------------------- @@ -149,8 +172,6 @@ def drop(table_name) def drop_all(regex) regex = regex.to_s failed = @admin.deleteTables(regex).map { |t| t.getNameAsString } - flush(org.apache.hadoop.hbase.HConstants::META_TABLE_NAME) - major_compact(org.apache.hadoop.hbase.HConstants::META_TABLE_NAME) return failed end @@ -182,37 +203,65 @@ def create(table_name, *args) raise(ArgumentError, "#{arg.class} of #{arg.inspect} is not of Hash or String type") end - if arg.kind_of?(Hash) and (arg.has_key?(SPLITS) or arg.has_key?(SPLITS_FILE)) - if arg.has_key?(SPLITS_FILE) - unless File.exist?(arg[SPLITS_FILE]) - raise(ArgumentError, "Splits file #{arg[SPLITS_FILE]} doesn't exist") - end - arg[SPLITS] = [] - File.foreach(arg[SPLITS_FILE]) do |line| - arg[SPLITS].push(line.strip()) + if arg.kind_of?(String) + # the arg is a string, default action is to add a column to the table + htd.addFamily(hcd(arg, htd)) + else + # arg is a hash. 4 possibilities: + if (arg.has_key?(SPLITS) or arg.has_key?(SPLITS_FILE)) + if arg.has_key?(SPLITS_FILE) + unless File.exist?(arg[SPLITS_FILE]) + raise(ArgumentError, "Splits file #{arg[SPLITS_FILE]} doesn't exist") + end + arg[SPLITS] = [] + File.foreach(arg[SPLITS_FILE]) do |line| + arg[SPLITS].push(line.strip()) + end end - end - splits = Java::byte[][arg[SPLITS].size].new - idx = 0 - arg[SPLITS].each do |split| - splits[idx] = split.to_java_bytes - idx = idx + 1 - end - elsif arg.kind_of?(Hash) and (arg.has_key?(NUMREGIONS) or arg.has_key?(SPLITALGO)) - raise(ArgumentError, "Number of regions must be specified") unless arg.has_key?(NUMREGIONS) - raise(ArgumentError, "Split algorithm must be specified") unless arg.has_key?(SPLITALGO) - raise(ArgumentError, "Number of regions must be geter than 1") unless arg[NUMREGIONS] > 1 - num_regions = arg[NUMREGIONS] - split_algo = org.apache.hadoop.hbase.util.RegionSplitter.newSplitAlgoInstance(@conf, arg[SPLITALGO]) - splits = split_algo.split(JInteger.valueOf(num_regions)) - else - # Add column to the table - descriptor = hcd(arg, htd) - if arg[COMPRESSION_COMPACT] - descriptor.setValue(COMPRESSION_COMPACT, arg[COMPRESSION_COMPACT]) + splits = Java::byte[][arg[SPLITS].size].new + idx = 0 + arg[SPLITS].each do |split| + splits[idx] = split.to_java_bytes + idx = idx + 1 + end + elsif (arg.has_key?(NUMREGIONS) or arg.has_key?(SPLITALGO)) + # (1) deprecated region pre-split API + raise(ArgumentError, "Column family configuration should be specified in a separate clause") if arg.has_key?(NAME) + raise(ArgumentError, "Number of regions must be specified") unless arg.has_key?(NUMREGIONS) + raise(ArgumentError, "Split algorithm must be specified") unless arg.has_key?(SPLITALGO) + raise(ArgumentError, "Number of regions must be greater than 1") unless arg[NUMREGIONS] > 1 + num_regions = arg[NUMREGIONS] + split_algo = RegionSplitter.newSplitAlgoInstance(@conf, arg[SPLITALGO]) + splits = split_algo.split(JInteger.valueOf(num_regions)) + elsif (method = arg.delete(METHOD)) + # (2) table_att modification + raise(ArgumentError, "table_att is currently the only supported method") unless method == 'table_att' + raise(ArgumentError, "NUMREGIONS & SPLITALGO must both be specified") unless arg.has_key?(NUMREGIONS) == arg.has_key?(split_algo) + htd.setMaxFileSize(JLong.valueOf(arg[MAX_FILESIZE])) if arg[MAX_FILESIZE] + htd.setReadOnly(JBoolean.valueOf(arg[READONLY])) if arg[READONLY] + htd.setMemStoreFlushSize(JLong.valueOf(arg[MEMSTORE_FLUSHSIZE])) if arg[MEMSTORE_FLUSHSIZE] + htd.setDeferredLogFlush(JBoolean.valueOf(arg[DEFERRED_LOG_FLUSH])) if arg[DEFERRED_LOG_FLUSH] + htd.setValue(COMPRESSION_COMPACT, arg[COMPRESSION_COMPACT]) if arg[COMPRESSION_COMPACT] + if arg[NUMREGIONS] + raise(ArgumentError, "Number of regions must be greater than 1") unless arg[NUMREGIONS] > 1 + num_regions = arg[NUMREGIONS] + split_algo = RegionSplitter.newSplitAlgoInstance(@conf, arg[SPLITALGO]) + splits = split_algo.split(JInteger.valueOf(num_regions)) + end + if arg[CONFIG] + raise(ArgumentError, "#{CONFIG} must be a Hash type") unless arg.kind_of?(Hash) + for k,v in arg[CONFIG] + v = v.to_s unless v.nil? + htd.setValue(k, v) + end + end + else + # (3) column family spec + descriptor = hcd(arg, htd) + htd.setValue(COMPRESSION_COMPACT, arg[COMPRESSION_COMPACT]) if arg[COMPRESSION_COMPACT] + htd.addFamily(hcd(arg, htd)) end - htd.addFamily(descriptor) end end @@ -257,16 +306,7 @@ def move(encoded_region_name, server = nil) #---------------------------------------------------------------------------------------------- # Returns table's structure description def describe(table_name) - tables = @admin.listTables.to_a - tables << org.apache.hadoop.hbase.HTableDescriptor::META_TABLEDESC - tables << org.apache.hadoop.hbase.HTableDescriptor::ROOT_TABLEDESC - - tables.each do |t| - # Found the table - return t.to_s if t.getNameAsString == table_name - end - - raise(ArgumentError, "Failed to find table named #{table_name}") + @admin.getTableDescriptor(table_name.to_java_bytes).to_s end #---------------------------------------------------------------------------------------------- @@ -274,16 +314,35 @@ def describe(table_name) def truncate(table_name, conf = @conf) h_table = org.apache.hadoop.hbase.client.HTable.new(conf, table_name) table_description = h_table.getTableDescriptor() + raise ArgumentError, "Table #{table_name} is not enabled. Enable it first.'" unless enabled?(table_name) yield 'Disabling table...' if block_given? - disable(table_name) + @admin.disableTable(table_name) yield 'Dropping table...' if block_given? - drop(table_name) + @admin.deleteTable(table_name) yield 'Creating table...' if block_given? @admin.createTable(table_description) end + #---------------------------------------------------------------------------------------------- + # Truncates table while maintaing region boundaries (deletes all records by recreating the table) + def truncate_preserve(table_name, conf = @conf) + h_table = org.apache.hadoop.hbase.client.HTable.new(conf, table_name) + splits = h_table.getRegionLocations().keys().map{|i| Bytes.toStringBinary(i.getStartKey)}.delete_if{|k| k == ""}.to_java :String + splits = org.apache.hadoop.hbase.util.Bytes.toBinaryByteArrays(splits) + table_description = h_table.getTableDescriptor() + yield 'Disabling table...' if block_given? + disable(table_name) + + yield 'Dropping table...' if block_given? + drop(table_name) + + yield 'Creating table with region boundaries...' if block_given? + @admin.createTable(table_description, splits) + end + + #---------------------------------------------------------------------------------------------- # Check the status of alter command (number of regions reopened) def alter_status(table_name) # Table name should be a string @@ -359,6 +418,7 @@ def alter(table_name, wait = true, *args) puts "Updating all regions with the new schema..." alter_status(table_name) end + htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) end next end @@ -413,6 +473,13 @@ def alter(table_name, wait = true, *args) end end + if arg[CONFIG] + raise(ArgumentError, "#{CONFIG} must be a Hash type") unless arg.kind_of?(Hash) + for k,v in arg[CONFIG] + v = v.to_s unless v.nil? + htd.setValue(k, v) + end + end @admin.modifyTable(table_name.to_java_bytes, htd) if wait == true puts "Updating all regions with the new schema..." @@ -531,7 +598,6 @@ def hcd(arg, htd) family.setScope(JInteger.valueOf(arg[org.apache.hadoop.hbase.HColumnDescriptor::REPLICATION_SCOPE])) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::REPLICATION_SCOPE) family.setInMemory(JBoolean.valueOf(arg[org.apache.hadoop.hbase.HColumnDescriptor::IN_MEMORY])) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::IN_MEMORY) family.setTimeToLive(JInteger.valueOf(arg[org.apache.hadoop.hbase.HColumnDescriptor::TTL])) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::TTL) - family.setCompressionType(org.apache.hadoop.hbase.io.hfile.Compression::Algorithm.valueOf(arg[org.apache.hadoop.hbase.HColumnDescriptor::COMPRESSION])) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::COMPRESSION) family.setDataBlockEncoding(org.apache.hadoop.hbase.io.encoding.DataBlockEncoding.valueOf(arg[org.apache.hadoop.hbase.HColumnDescriptor::DATA_BLOCK_ENCODING])) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::DATA_BLOCK_ENCODING) family.setEncodeOnDisk(JBoolean.valueOf(arg[org.apache.hadoop.hbase.HColumnDescriptor::ENCODE_ON_DISK])) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::ENCODE_ON_DISK) family.setBlocksize(JInteger.valueOf(arg[org.apache.hadoop.hbase.HColumnDescriptor::BLOCKSIZE])) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::BLOCKSIZE) @@ -554,6 +620,14 @@ def hcd(arg, htd) family.setCompressionType(org.apache.hadoop.hbase.io.hfile.Compression::Algorithm.valueOf(compression)) end end + + if arg[CONFIG] + raise(ArgumentError, "#{CONFIG} must be a Hash type") unless arg.kind_of?(Hash) + for k,v in arg[CONFIG] + v = v.to_s unless v.nil? + family.setValue(k, v) + end + end return family end @@ -579,5 +653,45 @@ def online(region_name, on_off) put.add(org.apache.hadoop.hbase.HConstants::CATALOG_FAMILY, org.apache.hadoop.hbase.HConstants::REGIONINFO_QUALIFIER, org.apache.hadoop.hbase.util.Writables.getBytes(hri)) meta.put(put) end + + #---------------------------------------------------------------------------------------------- + # Take a snapshot of specified table + def snapshot(table, snapshot_name, *args) + if args.empty? + @admin.snapshot(snapshot_name.to_java_bytes, table.to_java_bytes) + else + args.each do |arg| + if arg[SKIP_FLUSH] == true + @admin.snapshot(snapshot_name.to_java_bytes, table.to_java_bytes, SnapshotDescription::Type::SKIPFLUSH) + else + @admin.snapshot(snapshot_name.to_java_bytes, table.to_java_bytes) + end + end + end + end + + #---------------------------------------------------------------------------------------------- + # Restore specified snapshot + def restore_snapshot(snapshot_name) + @admin.restoreSnapshot(snapshot_name.to_java_bytes) + end + + #---------------------------------------------------------------------------------------------- + # Create a new table by cloning the snapshot content + def clone_snapshot(snapshot_name, table) + @admin.cloneSnapshot(snapshot_name.to_java_bytes, table.to_java_bytes) + end + + #---------------------------------------------------------------------------------------------- + # Delete specified snapshot + def delete_snapshot(snapshot_name) + @admin.deleteSnapshot(snapshot_name.to_java_bytes) + end + + #---------------------------------------------------------------------------------------------- + # Returns a list of snapshots + def list_snapshot + @admin.listSnapshots + end end end diff --git a/src/main/ruby/hbase/hbase.rb b/src/main/ruby/hbase/hbase.rb index 2c37840dd206..f97d0f95fdfb 100644 --- a/src/main/ruby/hbase/hbase.rb +++ b/src/main/ruby/hbase/hbase.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/hbase/replication_admin.rb b/src/main/ruby/hbase/replication_admin.rb index c4be93c299f2..2ade317315f5 100644 --- a/src/main/ruby/hbase/replication_admin.rb +++ b/src/main/ruby/hbase/replication_admin.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -43,12 +42,24 @@ def remove_peer(id) @replication_admin.removePeer(id) end + #--------------------------------------------------------------------------------------------- + # Show replcated tables/column families, and their ReplicationType + def list_replicated_tables + @replication_admin.listReplicated() + end + #---------------------------------------------------------------------------------------------- # List all peer clusters def list_peers @replication_admin.listPeers end + #---------------------------------------------------------------------------------------------- + # Get peer cluster state + def get_peer_state(id) + @replication_admin.getPeerState(id) + end + #---------------------------------------------------------------------------------------------- # Restart the replication stream to the specified peer def enable_peer(id) diff --git a/src/main/ruby/hbase/security.rb b/src/main/ruby/hbase/security.rb index 54965ae41cb8..ce452d186a87 100644 --- a/src/main/ruby/hbase/security.rb +++ b/src/main/ruby/hbase/security.rb @@ -31,62 +31,122 @@ def initialize(configuration, formatter) end #---------------------------------------------------------------------------------------------- - def grant(user, permissions, table_name, family=nil, qualifier=nil) + def grant(user, permissions, table_name=nil, family=nil, qualifier=nil) security_available? - # Table should exist - raise(ArgumentError, "Can't find a table: #{table_name}") unless exists?(table_name) + # TODO: need to validate user name - htd = @admin.getTableDescriptor(table_name.to_java_bytes) - - if (family != nil) - raise(ArgumentError, "Can't find a family: #{family}") unless htd.hasFamily(family.to_java_bytes) + # Verify that the specified permission is valid + if (permissions == nil || permissions.length == 0) + raise(ArgumentError, "Invalid permission: no actions associated with user") end - #TODO: need to validate user name + if (table_name != nil) + # Table should exist + raise(ArgumentError, "Can't find a table: #{table_name}") unless exists?(table_name) + + htd = @admin.getTableDescriptor(table_name.to_java_bytes) + + if (family != nil) + raise(ArgumentError, "Can't find a family: #{family}") unless htd.hasFamily(family.to_java_bytes) + end + + # invoke cp endpoint to perform access controlse + fambytes = family.to_java_bytes if (family != nil) + qualbytes = qualifier.to_java_bytes if (qualifier != nil) + user_permission = org.apache.hadoop.hbase.security.access.UserPermission.new( + user.to_java_bytes, table_name.to_java_bytes, + fambytes, qualbytes, permissions.to_java_bytes) + else + user_permission = org.apache.hadoop.hbase.security.access.UserPermission.new( + user.to_java_bytes, permissions.to_java_bytes) + end - # invoke cp endpoint to perform access control - fambytes = family.to_java_bytes if (family != nil) - qualbytes = qualifier.to_java_bytes if (qualifier != nil) - tp = org.apache.hadoop.hbase.security.access.TablePermission.new(table_name.to_java_bytes, fambytes, qualbytes, permissions.to_java_bytes) - meta_table = org.apache.hadoop.hbase.client.HTable.new(@config, org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME) - protocol = meta_table.coprocessorProxy(org.apache.hadoop.hbase.security.access.AccessControllerProtocol.java_class, + meta_table = org.apache.hadoop.hbase.client.HTable.new(@config, + org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME) + protocol = meta_table.coprocessorProxy( + org.apache.hadoop.hbase.security.access.AccessControllerProtocol.java_class, org.apache.hadoop.hbase.HConstants::EMPTY_START_ROW) - protocol.grant(user.to_java_bytes, tp) + begin + protocol.grant(user_permission) + rescue java.io.IOException => e + if !(e.message.include? "java.lang.NoSuchMethodException") + raise e + end + + # Server has not the new API, try the old one + if (table_name == nil) + raise "Global permissions not supported by HBase Server" + end + + tp = org.apache.hadoop.hbase.security.access.TablePermission.new(table_name.to_java_bytes, fambytes, qualbytes, permissions.to_java_bytes) + protocol.grant(user.to_java_bytes, tp) + end end #---------------------------------------------------------------------------------------------- - def revoke(user, table_name, family=nil, qualifier=nil) + def revoke(user, table_name=nil, family=nil, qualifier=nil) security_available? - # Table should exist - raise(ArgumentError, "Can't find table: #{table_name}") unless exists?(table_name) + # TODO: need to validate user name - htd = @admin.getTableDescriptor(table_name.to_java_bytes) + if (table_name != nil) + # Table should exist + raise(ArgumentError, "Can't find a table: #{table_name}") unless exists?(table_name) - if (family != nil) - raise(ArgumentError, "Can't find a family: #{family}") unless htd.hasFamily(family.to_java_bytes) + htd = @admin.getTableDescriptor(table_name.to_java_bytes) + + if (family != nil) + raise(ArgumentError, "Can't find family: #{family}") unless htd.hasFamily(family.to_java_bytes) + end + + # invoke cp endpoint to perform access control + fambytes = family.to_java_bytes if (family != nil) + qualbytes = qualifier.to_java_bytes if (qualifier != nil) + user_permission = org.apache.hadoop.hbase.security.access.UserPermission.new( + user.to_java_bytes, table_name.to_java_bytes, + fambytes, qualbytes, "".to_java_bytes) + else + user_permission = org.apache.hadoop.hbase.security.access.UserPermission.new( + user.to_java_bytes, "".to_java_bytes) end - fambytes = family.to_java_bytes if (family != nil) - qualbytes = qualifier.to_java_bytes if (qualifier != nil) - tp = org.apache.hadoop.hbase.security.access.TablePermission.new(table_name.to_java_bytes, fambytes, qualbytes, "".to_java_bytes) - meta_table = org.apache.hadoop.hbase.client.HTable.new(@config, org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME) - protocol = meta_table.coprocessorProxy(org.apache.hadoop.hbase.security.access.AccessControllerProtocol.java_class, + meta_table = org.apache.hadoop.hbase.client.HTable.new(@config, + org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME) + protocol = meta_table.coprocessorProxy( + org.apache.hadoop.hbase.security.access.AccessControllerProtocol.java_class, org.apache.hadoop.hbase.HConstants::EMPTY_START_ROW) - protocol.revoke(user.to_java_bytes, tp) + begin + protocol.revoke(user_permission) + rescue java.io.IOException => e + if !(e.message.include? "java.lang.NoSuchMethodException") + raise e + end + + # Server has not the new API, try the old one + if (table_name == nil) + raise "Global permissions not supported by HBase Server" + end + + tp = org.apache.hadoop.hbase.security.access.TablePermission.new(table_name.to_java_bytes, fambytes, qualbytes, "".to_java_bytes) + protocol.revoke(user.to_java_bytes, tp) + end end #---------------------------------------------------------------------------------------------- - def user_permission(table_name) + def user_permission(table_name=nil) security_available? - raise(ArgumentError, "Can't find table: #{table_name}") unless exists?(table_name) + if (table_name != nil) + raise(ArgumentError, "Can't find table: #{table_name}") unless exists?(table_name) + end - meta_table = org.apache.hadoop.hbase.client.HTable.new(@config, org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME) - protocol = meta_table.coprocessorProxy(org.apache.hadoop.hbase.security.access.AccessControllerProtocol.java_class, - org.apache.hadoop.hbase.HConstants::EMPTY_START_ROW) - perms = protocol.getUserPermissions(table_name.to_java_bytes) + meta_table = org.apache.hadoop.hbase.client.HTable.new(@config, + org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME) + protocol = meta_table.coprocessorProxy( + org.apache.hadoop.hbase.security.access.AccessControllerProtocol.java_class, + org.apache.hadoop.hbase.HConstants::EMPTY_START_ROW) + perms = protocol.getUserPermissions(table_name != nil ? table_name.to_java_bytes : nil) res = {} count = 0 @@ -122,6 +182,8 @@ def security_available?() rescue NameError raise(ArgumentError, "DISABLED: Security features are not available in this build of HBase") end + raise(ArgumentError, "Command not supported as authorization is turned off ") \ + unless exists?(org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME) end end diff --git a/src/main/ruby/hbase/table.rb b/src/main/ruby/hbase/table.rb index 6dcf47e697a8..125abd4f0726 100644 --- a/src/main/ruby/hbase/table.rb +++ b/src/main/ruby/hbase/table.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -25,9 +24,16 @@ module Hbase class Table include HBaseConstants + attr_reader :table + @@thread_pool = nil def initialize(configuration, table_name, formatter) - @table = org.apache.hadoop.hbase.client.HTable.new(configuration, table_name) + if @@thread_pool then + @table = org.apache.hadoop.hbase.client.HTable.new(configuration, table_name.to_java_bytes, @@thread_pool) + else + @table = org.apache.hadoop.hbase.client.HTable.new(configuration, table_name) + @@thread_pool = @table.getPool() + end end #---------------------------------------------------------------------------------------------- @@ -88,7 +94,8 @@ def count(interval = 1000, caching_rows = 10) count += 1 next unless (block_given? && count % interval == 0) # Allow command modules to visualize counting process - yield(count, String.from_java_bytes(row.getRow)) + yield(count, + org.apache.hadoop.hbase.util.Bytes::toStringBinary(row.getRow)) end # Return the counter @@ -117,6 +124,7 @@ def get(row, *args) # Get maxlength parameter if passed maxlength = args.delete(MAXLENGTH) if args[MAXLENGTH] + filter = args.delete(FILTER) if args[FILTER] unless args.empty? columns = args[COLUMN] || args[COLUMNS] @@ -161,6 +169,12 @@ def get(row, *args) end end + unless filter.class == String + get.setFilter(filter) + else + get.setFilter(org.apache.hadoop.hbase.filter.ParseFilter.new.parseFilterString(filter)) + end + # Call hbase for the results result = @table.get(get) return nil if result.isEmpty @@ -219,7 +233,8 @@ def scan(args = {}) stoprow = args["STOPROW"] timestamp = args["TIMESTAMP"] columns = args["COLUMNS"] || args["COLUMN"] || [] - cache = args["CACHE_BLOCKS"] || true + cache_blocks = args["CACHE_BLOCKS"] || true + cache = args["CACHE"] || 0 versions = args["VERSIONS"] || 1 timerange = args[TIMERANGE] raw = args["RAW"] || false @@ -252,7 +267,8 @@ def scan(args = {}) end scan.setTimeStamp(timestamp) if timestamp - scan.setCacheBlocks(cache) + scan.setCacheBlocks(cache_blocks) + scan.setCaching(cache) if cache > 0 scan.setMaxVersions(versions) if versions > 1 scan.setTimeRange(timerange[0], timerange[1]) if timerange scan.setRaw(raw) diff --git a/src/main/ruby/irb/hirb.rb b/src/main/ruby/irb/hirb.rb index 584f70087f44..b32e691cf8c6 100644 --- a/src/main/ruby/irb/hirb.rb +++ b/src/main/ruby/irb/hirb.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell.rb b/src/main/ruby/shell.rb index 53f3de878524..e8778f4e56e8 100644 --- a/src/main/ruby/shell.rb +++ b/src/main/ruby/shell.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -211,6 +210,7 @@ def help_footer :commands => %w[ status version + whoami ] ) @@ -250,6 +250,7 @@ def help_footer put scan truncate + truncate_preserve ] ) @@ -285,6 +286,19 @@ def help_footer disable_peer start_replication stop_replication + list_replicated_tables + ] +) + +Shell.load_command_group( + 'snapshot', + :full_name => 'CLUSTER SNAPSHOT TOOLS', + :commands => %w[ + snapshot + clone_snapshot + restore_snapshot + delete_snapshot + list_snapshots ] ) diff --git a/src/main/ruby/shell/commands.rb b/src/main/ruby/shell/commands.rb index af6df3344008..57b8f1c49462 100644 --- a/src/main/ruby/shell/commands.rb +++ b/src/main/ruby/shell/commands.rb @@ -1,5 +1,4 @@ # -# Copyright 2009 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -72,13 +71,37 @@ def format_simple_command def translate_hbase_exceptions(*args) yield - rescue org.apache.hadoop.hbase.TableNotFoundException - raise "Unknown table #{args.first}!" - rescue org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException - valid_cols = table(args.first).get_all_columns.map { |c| c + '*' } - raise "Unknown column family! Valid column names: #{valid_cols.join(", ")}" - rescue org.apache.hadoop.hbase.TableExistsException - raise "Table already exists: #{args.first}!" + rescue => e + raise e unless e.respond_to?(:cause) && e.cause != nil + + # Get the special java exception which will be handled + cause = e.cause + if cause.kind_of?(org.apache.hadoop.hbase.TableNotFoundException) then + str = java.lang.String.new("#{cause}") + raise "Unknown table #{str}!" + end + if cause.kind_of?(org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException) then + exceptions = cause.getCauses + exceptions.each do |exception| + if exception.kind_of?(org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException) then + valid_cols = table(args.first).get_all_columns.map { |c| c + '*' } + raise "Unknown column family! Valid column names: #{valid_cols.join(", ")}" + end + end + end + if cause.kind_of?(org.apache.hadoop.hbase.TableExistsException) then + str = java.lang.String.new("#{cause}") + strs = str.split("\n") + if strs.size > 0 then + s = strs[0].split(' '); + if(s.size > 1) + raise "Table already exists: #{s[1]}!" + end + raise "Table already exists: #{strs[0]}!" + end + end + # Throw the other exception which hasn't been handled above + raise e end end end diff --git a/src/main/ruby/shell/commands/add_peer.rb b/src/main/ruby/shell/commands/add_peer.rb index 7669fb7c5266..deb5c5def51f 100644 --- a/src/main/ruby/shell/commands/add_peer.rb +++ b/src/main/ruby/shell/commands/add_peer.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/alter.rb b/src/main/ruby/shell/commands/alter.rb index 33fbcb3cdd6b..dc9e1e18ae0c 100644 --- a/src/main/ruby/shell/commands/alter.rb +++ b/src/main/ruby/shell/commands/alter.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -40,6 +39,9 @@ def help or a shorter version: hbase> alter 't1', 'delete' => 'f1' + +You can also change the column family config by set attribute CONFIG like this: + hbase> alter 'test', NAME=>'f', CONFIG => {'hbase.hstore.compaction.min' => '5'} You can also change table-scope attributes like MAX_FILESIZE MEMSTORE_FLUSHSIZE, READONLY, and DEFERRED_LOG_FLUSH. @@ -47,6 +49,9 @@ def help For example, to change the max size of a family to 128MB, do: hbase> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '134217728' + +You can also change the table-scope by set attribute CONFIG like this: + hbase> alter 'test', METHOD=>'table_att', CONFIG => {'hbase.hstore.compaction.min' => '5'} You can add a table coprocessor by setting a table coprocessor attribute: diff --git a/src/main/ruby/shell/commands/alter_async.rb b/src/main/ruby/shell/commands/alter_async.rb index 01dfd994656a..164f33914915 100644 --- a/src/main/ruby/shell/commands/alter_async.rb +++ b/src/main/ruby/shell/commands/alter_async.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/alter_status.rb b/src/main/ruby/shell/commands/alter_status.rb index cc79e1108b6d..d62edacc9c67 100644 --- a/src/main/ruby/shell/commands/alter_status.rb +++ b/src/main/ruby/shell/commands/alter_status.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/assign.rb b/src/main/ruby/shell/commands/assign.rb index 4c83d3cf2006..de0071a68d87 100644 --- a/src/main/ruby/shell/commands/assign.rb +++ b/src/main/ruby/shell/commands/assign.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/balance_switch.rb b/src/main/ruby/shell/commands/balance_switch.rb index 0eac7654a812..4d7778daa41b 100644 --- a/src/main/ruby/shell/commands/balance_switch.rb +++ b/src/main/ruby/shell/commands/balance_switch.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/balancer.rb b/src/main/ruby/shell/commands/balancer.rb index 05ecd3a2bc71..c329eced4b9d 100644 --- a/src/main/ruby/shell/commands/balancer.rb +++ b/src/main/ruby/shell/commands/balancer.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/clone_snapshot.rb b/src/main/ruby/shell/commands/clone_snapshot.rb new file mode 100644 index 000000000000..43d240f12f34 --- /dev/null +++ b/src/main/ruby/shell/commands/clone_snapshot.rb @@ -0,0 +1,40 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class CloneSnapshot < Command + def help + return <<-EOF +Create a new table by cloning the snapshot content. +There're no copies of data involved. +And writing on the newly created table will not influence the snapshot data. + +Examples: + hbase> clone_snapshot 'snapshotName', 'tableName' +EOF + end + + def command(snapshot_name, table) + format_simple_command do + admin.clone_snapshot(snapshot_name, table) + end + end + end + end +end diff --git a/src/main/ruby/shell/commands/close_region.rb b/src/main/ruby/shell/commands/close_region.rb index f1010ac6ca01..1cd149725a7a 100644 --- a/src/main/ruby/shell/commands/close_region.rb +++ b/src/main/ruby/shell/commands/close_region.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/compact.rb b/src/main/ruby/shell/commands/compact.rb index d8f71de4904f..47cf0380ac7c 100644 --- a/src/main/ruby/shell/commands/compact.rb +++ b/src/main/ruby/shell/commands/compact.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -23,14 +22,24 @@ module Commands class Compact < Command def help return <<-EOF -Compact all regions in passed table or pass a region row -to compact an individual region -EOF + Compact all regions in passed table or pass a region row + to compact an individual region. You can also compact a single column + family within a region. + Examples: + Compact all regions in a table: + hbase> compact 't1' + Compact an entire region: + hbase> compact 'r1' + Compact only a column family within a region: + hbase> compact 'r1', 'c1' + Compact a column family within a table: + hbase> compact 't1', 'c1' + EOF end - def command(table_or_region_name) + def command(table_or_region_name, family = nil) format_simple_command do - admin.compact(table_or_region_name) + admin.compact(table_or_region_name, family) end end end diff --git a/src/main/ruby/shell/commands/count.rb b/src/main/ruby/shell/commands/count.rb index 6596441c1c32..fdc49a2b0e6d 100644 --- a/src/main/ruby/shell/commands/count.rb +++ b/src/main/ruby/shell/commands/count.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/create.rb b/src/main/ruby/shell/commands/create.rb index 14c1b0ff4f65..219b26338578 100644 --- a/src/main/ruby/shell/commands/create.rb +++ b/src/main/ruby/shell/commands/create.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/delete.rb b/src/main/ruby/shell/commands/delete.rb index 12bc40593359..2235d27da29c 100644 --- a/src/main/ruby/shell/commands/delete.rb +++ b/src/main/ruby/shell/commands/delete.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/delete_snapshot.rb b/src/main/ruby/shell/commands/delete_snapshot.rb new file mode 100644 index 000000000000..b8c3791a5401 --- /dev/null +++ b/src/main/ruby/shell/commands/delete_snapshot.rb @@ -0,0 +1,37 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class DeleteSnapshot < Command + def help + return <<-EOF +Delete a specified snapshot. Examples: + + hbase> delete_snapshot 'snapshotName', +EOF + end + + def command(snapshot_name) + format_simple_command do + admin.delete_snapshot(snapshot_name) + end + end + end + end +end diff --git a/src/main/ruby/shell/commands/deleteall.rb b/src/main/ruby/shell/commands/deleteall.rb index 5731b606b41e..a8c51c2b4a50 100644 --- a/src/main/ruby/shell/commands/deleteall.rb +++ b/src/main/ruby/shell/commands/deleteall.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/describe.rb b/src/main/ruby/shell/commands/describe.rb index 0f355074cda6..cce82c47705e 100644 --- a/src/main/ruby/shell/commands/describe.rb +++ b/src/main/ruby/shell/commands/describe.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/disable.rb b/src/main/ruby/shell/commands/disable.rb index 34c5f9ca7d08..b10aa9bf0357 100644 --- a/src/main/ruby/shell/commands/disable.rb +++ b/src/main/ruby/shell/commands/disable.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/disable_all.rb b/src/main/ruby/shell/commands/disable_all.rb index b7c9cf5fdd95..0e7c30ee9978 100644 --- a/src/main/ruby/shell/commands/disable_all.rb +++ b/src/main/ruby/shell/commands/disable_all.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/disable_peer.rb b/src/main/ruby/shell/commands/disable_peer.rb index ad1ebbd4c7d4..416545b21fb8 100644 --- a/src/main/ruby/shell/commands/disable_peer.rb +++ b/src/main/ruby/shell/commands/disable_peer.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -26,8 +25,6 @@ def help Stops the replication stream to the specified cluster, but still keeps track of new edits to replicate. -CURRENTLY UNSUPPORTED - Examples: hbase> disable_peer '1' diff --git a/src/main/ruby/shell/commands/drop.rb b/src/main/ruby/shell/commands/drop.rb index 181b83500f64..6238fe65b8f0 100644 --- a/src/main/ruby/shell/commands/drop.rb +++ b/src/main/ruby/shell/commands/drop.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -23,10 +22,7 @@ module Commands class Drop < Command def help return <<-EOF -Drop the named table. Table must first be disabled. If table has -more than one region, run a major compaction on .META.: - - hbase> major_compact ".META." +Drop the named table. Table must first be disabled: e.g. "hbase> drop 't1'" EOF end diff --git a/src/main/ruby/shell/commands/drop_all.rb b/src/main/ruby/shell/commands/drop_all.rb index dcbefa32346b..e482d6b0c79c 100644 --- a/src/main/ruby/shell/commands/drop_all.rb +++ b/src/main/ruby/shell/commands/drop_all.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/enable.rb b/src/main/ruby/shell/commands/enable.rb index a0dc340f5ddd..a4d495a55e9f 100644 --- a/src/main/ruby/shell/commands/enable.rb +++ b/src/main/ruby/shell/commands/enable.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/enable_all.rb b/src/main/ruby/shell/commands/enable_all.rb index ee016373f47f..fc0f081f1509 100644 --- a/src/main/ruby/shell/commands/enable_all.rb +++ b/src/main/ruby/shell/commands/enable_all.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/enable_peer.rb b/src/main/ruby/shell/commands/enable_peer.rb index 099f3fd4549a..55136ffb7fb5 100644 --- a/src/main/ruby/shell/commands/enable_peer.rb +++ b/src/main/ruby/shell/commands/enable_peer.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -26,8 +25,6 @@ def help Restarts the replication to the specified peer cluster, continuing from where it was disabled. -CURRENTLY UNSUPPORTED - Examples: hbase> enable_peer '1' diff --git a/src/main/ruby/shell/commands/exists.rb b/src/main/ruby/shell/commands/exists.rb index f35f197238fb..b7932ced528f 100644 --- a/src/main/ruby/shell/commands/exists.rb +++ b/src/main/ruby/shell/commands/exists.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/flush.rb b/src/main/ruby/shell/commands/flush.rb index ba597662c52a..185b4731706c 100644 --- a/src/main/ruby/shell/commands/flush.rb +++ b/src/main/ruby/shell/commands/flush.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/get.rb b/src/main/ruby/shell/commands/get.rb index 754c3d6b08f3..8c3161e100c5 100644 --- a/src/main/ruby/shell/commands/get.rb +++ b/src/main/ruby/shell/commands/get.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -33,6 +32,7 @@ def help hbase> get 't1', 'r1', {COLUMN => 'c1', TIMESTAMP => ts1} hbase> get 't1', 'r1', {COLUMN => 'c1', TIMERANGE => [ts1, ts2], VERSIONS => 4} hbase> get 't1', 'r1', {COLUMN => 'c1', TIMESTAMP => ts1, VERSIONS => 4} + hbase> get 't1', 'r1', {FILTER => "ValueFilter(=, 'binary:abc')"} hbase> get 't1', 'r1', 'c1' hbase> get 't1', 'r1', 'c1', 'c2' hbase> get 't1', 'r1', ['c1', 'c2'] diff --git a/src/main/ruby/shell/commands/get_counter.rb b/src/main/ruby/shell/commands/get_counter.rb index 3cbe2268d9ad..5c25f05c91de 100644 --- a/src/main/ruby/shell/commands/get_counter.rb +++ b/src/main/ruby/shell/commands/get_counter.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/grant.rb b/src/main/ruby/shell/commands/grant.rb index 7504f4349bd8..7bdc95943b93 100644 --- a/src/main/ruby/shell/commands/grant.rb +++ b/src/main/ruby/shell/commands/grant.rb @@ -21,19 +21,20 @@ module Commands class Grant < Command def help return <<-EOF -Grant users specific rights to tables. -Syntax : grant +Grant users specific rights. +Syntax : grant [
[ []] permissions is either zero or more letters from the set "RWXCA". READ('R'), WRITE('W'), EXEC('X'), CREATE('C'), ADMIN('A') For example: + hbase> grant 'bobsmith', 'RWXCA' hbase> grant 'bobsmith', 'RW', 't1', 'f1', 'col1' EOF end - def command(user, rights, table_name, family=nil, qualifier=nil) + def command(user, rights, table_name=nil, family=nil, qualifier=nil) format_simple_command do security_admin.grant(user, rights, table_name, family, qualifier) end diff --git a/src/main/ruby/shell/commands/hlog_roll.rb b/src/main/ruby/shell/commands/hlog_roll.rb index 02c7c592822d..b292791541fd 100644 --- a/src/main/ruby/shell/commands/hlog_roll.rb +++ b/src/main/ruby/shell/commands/hlog_roll.rb @@ -1,5 +1,4 @@ # -# Copyright 2011 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/incr.rb b/src/main/ruby/shell/commands/incr.rb index 38a2fc517934..c0c60a5fd328 100644 --- a/src/main/ruby/shell/commands/incr.rb +++ b/src/main/ruby/shell/commands/incr.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/is_disabled.rb b/src/main/ruby/shell/commands/is_disabled.rb index 9d3c7ee7471e..28918ee42980 100644 --- a/src/main/ruby/shell/commands/is_disabled.rb +++ b/src/main/ruby/shell/commands/is_disabled.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/is_enabled.rb b/src/main/ruby/shell/commands/is_enabled.rb index 96b2b152c2e1..6a98a36b71d0 100644 --- a/src/main/ruby/shell/commands/is_enabled.rb +++ b/src/main/ruby/shell/commands/is_enabled.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/list.rb b/src/main/ruby/shell/commands/list.rb index 592fb5eec253..16af27b9c097 100644 --- a/src/main/ruby/shell/commands/list.rb +++ b/src/main/ruby/shell/commands/list.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/list_peers.rb b/src/main/ruby/shell/commands/list_peers.rb index 93a430c87104..2f57592e5876 100644 --- a/src/main/ruby/shell/commands/list_peers.rb +++ b/src/main/ruby/shell/commands/list_peers.rb @@ -33,10 +33,11 @@ def command() now = Time.now peers = replication_admin.list_peers - formatter.header(["PEER ID", "CLUSTER KEY"]) + formatter.header(["PEER_ID", "CLUSTER_KEY", "STATE"]) peers.entrySet().each do |e| - formatter.row([ e.key, e.value ]) + state = replication_admin.get_peer_state(e.key) + formatter.row([ e.key, e.value, state ]) end formatter.footer(now) diff --git a/src/main/ruby/shell/commands/list_replicated_tables.rb b/src/main/ruby/shell/commands/list_replicated_tables.rb new file mode 100644 index 000000000000..b1494b8d3a50 --- /dev/null +++ b/src/main/ruby/shell/commands/list_replicated_tables.rb @@ -0,0 +1,52 @@ +# +# Copyright The Apache Software Foundation +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ListReplicatedTables< Command + def help + return <<-EOF +List all the tables and column families replicated from this cluster + + hbase> list_replicated_tables + hbase> list_replicated_tables 'abc.*' +EOF + end + + def command(regex = ".*") + now = Time.now + + formatter.header([ "TABLE:COLUMNFAMILY", "ReplicationType" ], [ 32 ]) + list = replication_admin.list_replicated_tables + regex = /#{regex}/ unless regex.is_a?(Regexp) + list = list.select {|s| regex.match(s.get(org.apache.hadoop.hbase.client.replication.ReplicationAdmin::TNAME))} + list.each do |e| + if e.get(org.apache.hadoop.hbase.client.replication.ReplicationAdmin::REPLICATIONTYPE) == org.apache.hadoop.hbase.client.replication.ReplicationAdmin::REPLICATIONGLOBAL + replicateType = "GLOBAL" + else + replicateType = "unknown" + end + formatter.row([e.get(org.apache.hadoop.hbase.client.replication.ReplicationAdmin::TNAME) + ":" + e.get(org.apache.hadoop.hbase.client.replication.ReplicationAdmin::CFNAME), replicateType], true, [32]) + end + formatter.footer(now) + end + end + end +end diff --git a/src/main/ruby/shell/commands/list_snapshots.rb b/src/main/ruby/shell/commands/list_snapshots.rb new file mode 100644 index 000000000000..95135185c2ee --- /dev/null +++ b/src/main/ruby/shell/commands/list_snapshots.rb @@ -0,0 +1,52 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +require 'time' + +module Shell + module Commands + class ListSnapshots < Command + def help + return <<-EOF +List all snapshots taken (by printing the names and relative information). +Optional regular expression parameter could be used to filter the output +by snapshot name. + +Examples: + hbase> list_snapshots + hbase> list_snapshots 'abc.*' +EOF + end + + def command(regex = ".*") + now = Time.now + formatter.header([ "SNAPSHOT", "TABLE + CREATION TIME"]) + + regex = /#{regex}/ unless regex.is_a?(Regexp) + list = admin.list_snapshot.select {|s| regex.match(s.getName)} + list.each do |snapshot| + creation_time = Time.at(snapshot.getCreationTime() / 1000).to_s + formatter.row([ snapshot.getName, snapshot.getTable + " (" + creation_time + ")" ]) + end + + formatter.footer(now, list.size) + return list.map { |s| s.getName() } + end + end + end +end diff --git a/src/main/ruby/shell/commands/major_compact.rb b/src/main/ruby/shell/commands/major_compact.rb index 56b081e3f968..f05cc1d78dc9 100644 --- a/src/main/ruby/shell/commands/major_compact.rb +++ b/src/main/ruby/shell/commands/major_compact.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -23,14 +22,25 @@ module Commands class MajorCompact < Command def help return <<-EOF -Run major compaction on passed table or pass a region row -to major compact an individual region -EOF + Run major compaction on passed table or pass a region row + to major compact an individual region. To compact a single + column family within a region specify the region name + followed by the column family name. + Examples: + Compact all regions in a table: + hbase> major_compact 't1' + Compact an entire region: + hbase> major_compact 'r1' + Compact a single column family within a region: + hbase> major_compact 'r1', 'c1' + Compact a single column family within a table: + hbase> major_compact 't1', 'c1' + EOF end - def command(table_or_region_name) + def command(table_or_region_name, family = nil) format_simple_command do - admin.major_compact(table_or_region_name) + admin.major_compact(table_or_region_name, family) end end end diff --git a/src/main/ruby/shell/commands/move.rb b/src/main/ruby/shell/commands/move.rb index 0e3db8fe4729..e6b28288e3fa 100644 --- a/src/main/ruby/shell/commands/move.rb +++ b/src/main/ruby/shell/commands/move.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/put.rb b/src/main/ruby/shell/commands/put.rb index dde043370838..6bbe7cddf37b 100644 --- a/src/main/ruby/shell/commands/put.rb +++ b/src/main/ruby/shell/commands/put.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/remove_peer.rb b/src/main/ruby/shell/commands/remove_peer.rb index 034434aa5dab..5ae5786fc4a4 100644 --- a/src/main/ruby/shell/commands/remove_peer.rb +++ b/src/main/ruby/shell/commands/remove_peer.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/restore_snapshot.rb b/src/main/ruby/shell/commands/restore_snapshot.rb new file mode 100644 index 000000000000..4d531711bca6 --- /dev/null +++ b/src/main/ruby/shell/commands/restore_snapshot.rb @@ -0,0 +1,41 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class RestoreSnapshot < Command + def help + return <<-EOF +Restore a specified snapshot. +The restore will replace the content of the original table, +bringing back the content to the snapshot state. +The table must be disabled. + +Examples: + hbase> restore_snapshot 'snapshotName' +EOF + end + + def command(snapshot_name) + format_simple_command do + admin.restore_snapshot(snapshot_name) + end + end + end + end +end diff --git a/src/main/ruby/shell/commands/revoke.rb b/src/main/ruby/shell/commands/revoke.rb index e94c3644f772..721519e7d6ca 100644 --- a/src/main/ruby/shell/commands/revoke.rb +++ b/src/main/ruby/shell/commands/revoke.rb @@ -21,15 +21,16 @@ module Commands class Revoke < Command def help return <<-EOF -Revoke a user's access rights to tables. -Syntax : revoke
+Revoke a user's access rights. +Syntax : revoke [
[ []] For example: - hbase> revoke 'bobsmith', 't1', 'f1' + hbase> revoke 'bobsmith' + hbase> revoke 'bobsmith', 't1', 'f1', 'col1' EOF end - def command(user, table_name, family=nil, qualifier=nil) + def command(user, table_name=nil, family=nil, qualifier=nil) format_simple_command do security_admin.revoke(user, table_name, family, qualifier) end diff --git a/src/main/ruby/shell/commands/scan.rb b/src/main/ruby/shell/commands/scan.rb index e58aaaceb8ee..f758fb21f7a2 100644 --- a/src/main/ruby/shell/commands/scan.rb +++ b/src/main/ruby/shell/commands/scan.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -26,7 +25,7 @@ def help Scan a table; pass table name and optionally a dictionary of scanner specifications. Scanner specifications may include one or more of: TIMERANGE, FILTER, LIMIT, STARTROW, STOPROW, TIMESTAMP, MAXLENGTH, -or COLUMNS. +or COLUMNS, CACHE If no columns are specified, all columns will be scanned. To scan all members of a column family, leave the qualifier empty as in diff --git a/src/main/ruby/shell/commands/show_filters.rb b/src/main/ruby/shell/commands/show_filters.rb index 1581db5b37aa..716859fc0618 100644 --- a/src/main/ruby/shell/commands/show_filters.rb +++ b/src/main/ruby/shell/commands/show_filters.rb @@ -1,5 +1,4 @@ # -# Copyright 2011 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/snapshot.rb b/src/main/ruby/shell/commands/snapshot.rb new file mode 100644 index 000000000000..026e5622b187 --- /dev/null +++ b/src/main/ruby/shell/commands/snapshot.rb @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Snapshot < Command + def help + return <<-EOF +Take a snapshot of specified table. Examples: + + hbase> snapshot 'sourceTable', 'snapshotName' + hbase> snapshot 'sourceTable', 'snapshotName', {SKIP_FLUSH => true} +EOF + end + + def command(table, snapshot_name, *args) + format_simple_command do + admin.snapshot(table, snapshot_name, *args) + end + end + end + end +end diff --git a/src/main/ruby/shell/commands/split.rb b/src/main/ruby/shell/commands/split.rb index 47e21f86c218..9488ed9ef156 100644 --- a/src/main/ruby/shell/commands/split.rb +++ b/src/main/ruby/shell/commands/split.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/start_replication.rb b/src/main/ruby/shell/commands/start_replication.rb index 5d1cd1b5ee1c..3ea97a90c993 100644 --- a/src/main/ruby/shell/commands/start_replication.rb +++ b/src/main/ruby/shell/commands/start_replication.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/status.rb b/src/main/ruby/shell/commands/status.rb index 4b22acb4abf1..f72c13caef60 100644 --- a/src/main/ruby/shell/commands/status.rb +++ b/src/main/ruby/shell/commands/status.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/stop_replication.rb b/src/main/ruby/shell/commands/stop_replication.rb index f5074d73a870..2e24fa5cfe69 100644 --- a/src/main/ruby/shell/commands/stop_replication.rb +++ b/src/main/ruby/shell/commands/stop_replication.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/truncate.rb b/src/main/ruby/shell/commands/truncate.rb index a24e167b9829..b7812fbe3dd5 100644 --- a/src/main/ruby/shell/commands/truncate.rb +++ b/src/main/ruby/shell/commands/truncate.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/truncate_preserve.rb b/src/main/ruby/shell/commands/truncate_preserve.rb new file mode 100644 index 000000000000..918b23289a9f --- /dev/null +++ b/src/main/ruby/shell/commands/truncate_preserve.rb @@ -0,0 +1,38 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class TruncatePreserve < Command + def help + return <<-EOF + Disables, drops and recreates the specified table while still maintaing the previous region boundaries. +EOF + end + + def command(table) + format_simple_command do + puts "Truncating '#{table}' table (it may take a while):" + admin.truncate_preserve(table) { |log| puts " - #{log}" } + end + end + + end + end +end diff --git a/src/main/ruby/shell/commands/unassign.rb b/src/main/ruby/shell/commands/unassign.rb index 75242baf4bc7..60cc6ca3fc80 100644 --- a/src/main/ruby/shell/commands/unassign.rb +++ b/src/main/ruby/shell/commands/unassign.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/user_permission.rb b/src/main/ruby/shell/commands/user_permission.rb index a4da0824fabb..ad4a7b029e77 100644 --- a/src/main/ruby/shell/commands/user_permission.rb +++ b/src/main/ruby/shell/commands/user_permission.rb @@ -21,15 +21,16 @@ module Commands class UserPermission < Command def help return <<-EOF -Show all table access permissions for the particular user. +Show all permissions for the particular user. Syntax : user_permission
For example: + hbase> user_permission hbase> user_permission 'table1' EOF end - def command(table) + def command(table=nil) #format_simple_command do #admin.user_permission(table) now = Time.now diff --git a/src/main/ruby/shell/commands/version.rb b/src/main/ruby/shell/commands/version.rb index 372b0dc341fe..63e9712a52d6 100644 --- a/src/main/ruby/shell/commands/version.rb +++ b/src/main/ruby/shell/commands/version.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/commands/whoami.rb b/src/main/ruby/shell/commands/whoami.rb new file mode 100644 index 000000000000..040ad7e7ef7e --- /dev/null +++ b/src/main/ruby/shell/commands/whoami.rb @@ -0,0 +1,37 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Whoami < Command + def help + return <<-EOF +Show the current hbase user. +Syntax : whoami +For example: + + hbase> whoami +EOF + end + + def command() + puts "#{org.apache.hadoop.hbase.security.User.getCurrent().toString()}" + end + end + end +end diff --git a/src/main/ruby/shell/commands/zk_dump.rb b/src/main/ruby/shell/commands/zk_dump.rb index bb23962df860..c0b509a3c6e8 100644 --- a/src/main/ruby/shell/commands/zk_dump.rb +++ b/src/main/ruby/shell/commands/zk_dump.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/ruby/shell/formatter.rb b/src/main/ruby/shell/formatter.rb index ea17882a6ecb..36aaf76aea7e 100644 --- a/src/main/ruby/shell/formatter.rb +++ b/src/main/ruby/shell/formatter.rb @@ -1,5 +1,4 @@ # -# Copyright 2010 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file diff --git a/src/main/xslt/configuration_to_docbook_section.xsl b/src/main/xslt/configuration_to_docbook_section.xsl index 95f7fd5d8c33..5f412df9b647 100644 --- a/src/main/xslt/configuration_to_docbook_section.xsl +++ b/src/main/xslt/configuration_to_docbook_section.xsl @@ -4,7 +4,6 @@ + + 2012-04-14 + + Apache HBase + + + Apache HBase software is the Hadoop database. Think of it as a distributed, scalable, big data store. + Use Apache HBase software when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware. HBase is an open-source, distributed, versioned, column-oriented store modeled after Google's Bigtable: A Distributed Storage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Hadoop and HDFS. + + + + Java + + + + Apache hbase 0.92.1 + 2012-03-19 + 0.92.1 + + + + + + + + + + + Apache HBase PMC + + + + + diff --git a/src/site/resources/images/big_h_logo.png b/src/site/resources/images/big_h_logo.png new file mode 100644 index 000000000000..5256094e46b6 Binary files /dev/null and b/src/site/resources/images/big_h_logo.png differ diff --git a/src/site/resources/images/big_h_logo.svg b/src/site/resources/images/big_h_logo.svg new file mode 100644 index 000000000000..ab241986fc37 --- /dev/null +++ b/src/site/resources/images/big_h_logo.svg @@ -0,0 +1,139 @@ + + + +image/svg+xml + + + + + + + + +APACHE + \ No newline at end of file diff --git a/src/site/resources/images/hbase_logo.png b/src/site/resources/images/hbase_logo.png index 615b0a80c891..e962ce04975b 100644 Binary files a/src/site/resources/images/hbase_logo.png and b/src/site/resources/images/hbase_logo.png differ diff --git a/src/site/resources/images/hbase_logo.svg b/src/site/resources/images/hbase_logo.svg index c4b3343ecc2c..2cc26d934100 100644 --- a/src/site/resources/images/hbase_logo.svg +++ b/src/site/resources/images/hbase_logo.svg @@ -1,41 +1,78 @@ - - - - - - - - - - - + + + +image/svg+xml + + + + + + + \ No newline at end of file diff --git a/src/site/site.vm b/src/site/site.vm index 03c2e758ebd6..f651010401cb 100644 --- a/src/site/site.vm +++ b/src/site/site.vm @@ -89,11 +89,9 @@ #if ( $banner.alt ) #set ( $alt = $banner.alt ) #else - #set ( $alt = $banner.name ) #end $alt #else - $banner.name #end ## #if( $banner.href ) @@ -481,6 +479,20 @@ #end #end ## $headContent + +
+ + + + + + + + + + + + + + + + + + + + + + + + +
ParameterDefault Value
zookeeper.znode.parent/hbase
zookeeper.znode.replicationreplication
zookeeper.znode.replication.peerspeers
zookeeper.znode.replication.peers.statepeer-state
zookeeper.znode.replication.rsrs
+

+ The default replication znode structure looks like the following: +

+ 
+                /hbase/replication/state
+                /hbase/replication/peers/{peerId}/peer-state
+                /hbase/replication/rs
+
+ +
+
    +
  • hbase.replication (Default: false) - Controls whether replication is enabled + or disabled for the cluster.
    • +
  • replication.sleep.before.failover (Default: 2000) - The amount of time a failover + worker waits before attempting to replicate a dead region server’s HLog queues.
    • +
  • replication.executor.workers (Default: 1) - The number of dead region servers + one region server should attempt to failover simultaneously.
    • +
+
+

When a master cluster RS initiates a replication source to a slave cluster, diff --git a/src/site/xdoc/resources.xml b/src/site/xdoc/resources.xml new file mode 100644 index 000000000000..d067c1e1e1e3 --- /dev/null +++ b/src/site/xdoc/resources.xml @@ -0,0 +1,42 @@ + + + + + Other Apache HBase (TM) Resources + + + +

+
+
+

HBase: The Definitive Guide Random Access to Your Planet-Size Data by Lars George. Publisher: O'Reilly Media, Released: August 2011, Pages: 556.

+
+
+

HBase In Action By Nick Dimiduk and Amandeep Khurana. Publisher: Manning, MEAP Began: January 2012, Softbound print: Fall 2012, Pages: 350.

+
+
+

HBase Administration Cookbook by Yifeng Jiang. Publisher: PACKT Publishing, Release: Expected August 2012, Pages: 335.

+
+
+
+ + diff --git a/src/site/xdoc/sponsors.xml b/src/site/xdoc/sponsors.xml index e39730bbbd2e..2d39344a1432 100644 --- a/src/site/xdoc/sponsors.xml +++ b/src/site/xdoc/sponsors.xml @@ -1,35 +1,49 @@ - Installing HBase on Windows using Cygwin + Apache HBase™ Sponsors
-

The below companies have been gracious enough to provide their commerical tool offerings free of charge to the Apache HBase project. +

First off, thanks to all who sponsor + our parent, the Apache Software Foundation. +

+

The below companies have been gracious enough to provide their commerical tool offerings free of charge to the Apache HBase™ project.

+
+

To contribute to the Apache Software Foundation, a good idea in our opinion, see the ASF Sponsorship page. +

+
 diff --git a/src/test/java/org/apache/hadoop/hbase/ClassFinder.java b/src/test/java/org/apache/hadoop/hbase/ClassFinder.java new file mode 100644 index 000000000000..afc4aa6f7da4 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/ClassFinder.java @@ -0,0 +1,232 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import java.io.File; +import java.io.FileFilter; +import java.io.FileInputStream; +import java.io.IOException; +import java.lang.reflect.Method; +import java.lang.reflect.Modifier; +import java.net.URL; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.jar.*; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; + +/** + * A class that finds a set of classes that are locally accessible + * (from .class or .jar files), and satisfy the conditions that are + * imposed by name and class filters provided by the user. + */ +public class ClassFinder { + private static final Log LOG = LogFactory.getLog(ClassFinder.class); + private static String CLASS_EXT = ".class"; + + private FileNameFilter fileNameFilter; + private ClassFilter classFilter; + private FileFilter fileFilter; + + public static interface FileNameFilter { + public boolean isCandidateFile(String fileName, String absFilePath); + }; + + public static interface ClassFilter { + public boolean isCandidateClass(Class c); + }; + + public ClassFinder(FileNameFilter fileNameFilter, ClassFilter classFilter) { + this.classFilter = classFilter; + this.fileNameFilter = fileNameFilter; + this.fileFilter = new FileFilterWithName(fileNameFilter); + } + + /** + * Finds the classes in current package (of ClassFinder) and nested packages. + * @param proceedOnExceptions whether to ignore exceptions encountered for + * individual jars/files/classes, and proceed looking for others. + */ + public Set> findClasses(boolean proceedOnExceptions) + throws ClassNotFoundException, IOException, LinkageError { + return findClasses(this.getClass().getPackage().getName(), proceedOnExceptions); + } + + /** + * Finds the classes in a package and nested packages. + * @param packageName package names + * @param proceedOnExceptions whether to ignore exceptions encountered for + * individual jars/files/classes, and proceed looking for others. + */ + public Set> findClasses(String packageName, boolean proceedOnExceptions) + throws ClassNotFoundException, IOException, LinkageError { + final String path = packageName.replace('.', '/'); + final Pattern jarResourceRe = Pattern.compile("^file:(.+\\.jar)!/" + path + "$"); + + Enumeration resources = ClassLoader.getSystemClassLoader().getResources(path); + List dirs = new ArrayList(); + List jars = new ArrayList(); + + while (resources.hasMoreElements()) { + URL resource = resources.nextElement(); + String resourcePath = resource.getFile(); + Matcher matcher = jarResourceRe.matcher(resourcePath); + if (matcher.find()) { + jars.add(matcher.group(1)); + } else { + dirs.add(new File(resource.getFile())); + } + } + + Set> classes = new HashSet>(); + for (File directory : dirs) { + classes.addAll(findClassesFromFiles(directory, packageName, proceedOnExceptions)); + } + for (String jarFileName : jars) { + classes.addAll(findClassesFromJar(jarFileName, packageName, proceedOnExceptions)); + } + return classes; + } + + private Set> findClassesFromJar(String jarFileName, + String packageName, boolean proceedOnExceptions) + throws IOException, ClassNotFoundException, LinkageError { + JarInputStream jarFile = null; + try { + jarFile = new JarInputStream(new FileInputStream(jarFileName)); + } catch (IOException ioEx) { + if (!proceedOnExceptions) { + throw ioEx; + } + LOG.warn("Failed to look for classes in " + jarFileName + ": " + ioEx); + } + + Set> classes = new HashSet>(); + JarEntry entry = null; + while (true) { + try { + entry = jarFile.getNextJarEntry(); + } catch (IOException ioEx) { + if (!proceedOnExceptions) { + throw ioEx; + } + LOG.warn("Failed to get next entry from " + jarFileName + ": " + ioEx); + break; + } + if (entry == null) { + break; // loop termination condition + } + + String className = entry.getName(); + if (!className.endsWith(CLASS_EXT)) { + continue; + } + int ix = className.lastIndexOf('/'); + String fileName = (ix >= 0) ? className.substring(ix + 1) : className; + if (!this.fileNameFilter.isCandidateFile(fileName, className)) { + continue; + } + className = className + .substring(0, className.length() - CLASS_EXT.length()).replace('/', '.'); + if (!className.startsWith(packageName)) { + continue; + } + Class c = makeClass(className, proceedOnExceptions); + if (c != null) { + if (!classes.add(c)) { + LOG.warn("Ignoring duplicate class " + className); + } + } + } + return classes; + } + + private Set> findClassesFromFiles(File baseDirectory, String packageName, + boolean proceedOnExceptions) throws ClassNotFoundException, LinkageError { + Set> classes = new HashSet>(); + if (!baseDirectory.exists()) { + LOG.warn("Failed to find " + baseDirectory.getAbsolutePath()); + return classes; + } + + File[] files = baseDirectory.listFiles(this.fileFilter); + if (files == null) { + LOG.warn("Failed to get files from " + baseDirectory.getAbsolutePath()); + return classes; + } + + for (File file : files) { + final String fileName = file.getName(); + if (file.isDirectory()) { + classes.addAll(findClassesFromFiles(file, packageName + "." + fileName, + proceedOnExceptions)); + } else { + String className = packageName + '.' + + fileName.substring(0, fileName.length() - CLASS_EXT.length()); + Class c = makeClass(className, proceedOnExceptions); + if (c != null) { + if (!classes.add(c)) { + LOG.warn("Ignoring duplicate class " + className); + } + } + } + } + return classes; + } + + private Class makeClass(String className, boolean proceedOnExceptions) + throws ClassNotFoundException, LinkageError { + try { + Class c = Class.forName(className, false, this.getClass().getClassLoader()); + return classFilter.isCandidateClass(c) ? c : null; + } catch (ClassNotFoundException classNotFoundEx) { + if (!proceedOnExceptions) { + throw classNotFoundEx; + } + LOG.info("Failed to instantiate or check " + className + ": " + classNotFoundEx); + } catch (LinkageError linkageEx) { + if (!proceedOnExceptions) { + throw linkageEx; + } + LOG.info("Failed to instantiate or check " + className + ": " + linkageEx); + } + return null; + } + + private class FileFilterWithName implements FileFilter { + private FileNameFilter nameFilter; + + public FileFilterWithName(FileNameFilter nameFilter) { + this.nameFilter = nameFilter; + } + + @Override + public boolean accept(File file) { + return file.isDirectory() + || (file.getName().endsWith(CLASS_EXT) + && nameFilter.isCandidateFile(file.getName(), file.getAbsolutePath())); + } + }; +}; diff --git a/src/test/java/org/apache/hadoop/hbase/ClassTestFinder.java b/src/test/java/org/apache/hadoop/hbase/ClassTestFinder.java new file mode 100644 index 000000000000..f40aa7984b5b --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/ClassTestFinder.java @@ -0,0 +1,116 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import java.lang.reflect.Method; +import java.lang.reflect.Modifier; +import java.util.regex.Pattern; + +import org.apache.hadoop.hbase.ClassFinder.ClassFilter; +import org.apache.hadoop.hbase.ClassFinder.FileNameFilter; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.junit.runners.Suite; + +/** + * ClassFinder that is pre-configured with filters that will only allow test classes. + * The name is strange because a logical name would start with "Test" and be confusing. + */ +public class ClassTestFinder extends ClassFinder { + + public ClassTestFinder() { + super(new TestFileNameFilter(), new TestClassFilter()); + } + + public ClassTestFinder(Class category) { + super(new TestFileNameFilter(), new TestClassFilter(category)); + } + + public static Class[] getCategoryAnnotations(Class c) { + Category category = c.getAnnotation(Category.class); + if (category != null) { + return category.value(); + } + return new Class[0]; + } + + public static class TestFileNameFilter implements FileNameFilter { + private static final Pattern hadoopCompactRe = + Pattern.compile("hbase-hadoop\\d?-compat"); + + @Override + public boolean isCandidateFile(String fileName, String absFilePath) { + boolean isTestFile = fileName.startsWith("Test") + || fileName.startsWith("IntegrationTest"); + return isTestFile && !hadoopCompactRe.matcher(absFilePath).find(); + } + }; + + /* + * A class is considered as a test class if: + * - it's not Abstract AND + * - one or more of its methods is annotated with org.junit.Test OR + * - the class is annotated with Suite.SuiteClasses + * */ + public static class TestClassFilter implements ClassFilter { + private Class categoryAnnotation = null; + public TestClassFilter(Class categoryAnnotation) { + this.categoryAnnotation = categoryAnnotation; + } + + public TestClassFilter() { + this(null); + } + + @Override + public boolean isCandidateClass(Class c) { + return isTestClass(c) && isCategorizedClass(c); + } + + private boolean isTestClass(Class c) { + if (Modifier.isAbstract(c.getModifiers())) { + return false; + } + + if (c.getAnnotation(Suite.SuiteClasses.class) != null) { + return true; + } + + for (Method met : c.getMethods()) { + if (met.getAnnotation(Test.class) != null) { + return true; + } + } + + return false; + } + + private boolean isCategorizedClass(Class c) { + if (this.categoryAnnotation == null) { + return true; + } + for (Class cc : getCategoryAnnotations(c)) { + if (cc.equals(this.categoryAnnotation)) { + return true; + } + } + return false; + } + }; +}; diff --git a/src/test/java/org/apache/hadoop/hbase/ClusterManager.java b/src/test/java/org/apache/hadoop/hbase/ClusterManager.java new file mode 100644 index 000000000000..5a5b010959b2 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/ClusterManager.java @@ -0,0 +1,132 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configured; + + +/** + * ClusterManager is an api to manage servers in a distributed environment. It provides services + * for starting / stopping / killing Hadoop/HBase daemons. Concrete implementations provide actual + * functionality for carrying out deployment-specific tasks. + */ +@InterfaceAudience.Private +public abstract class ClusterManager extends Configured { + protected static final Log LOG = LogFactory.getLog(ClusterManager.class); + + private static final String SIGKILL = "SIGKILL"; + private static final String SIGSTOP = "SIGSTOP"; + private static final String SIGCONT = "SIGCONT"; + + public ClusterManager() { + } + + /** + * Type of the service daemon + */ + public static enum ServiceType { + HADOOP_NAMENODE("namenode"), + HADOOP_DATANODE("datanode"), + HADOOP_JOBTRACKER("jobtracker"), + HADOOP_TASKTRACKER("tasktracker"), + HBASE_MASTER("master"), + HBASE_REGIONSERVER("regionserver"); + + private String name; + + ServiceType(String name) { + this.name = name; + } + + public String getName() { + return name; + } + + @Override + public String toString() { + return getName(); + } + } + + /** + * Start the service on the given host + */ + public abstract void start(ServiceType service, String hostname) throws IOException; + + /** + * Stop the service on the given host + */ + public abstract void stop(ServiceType service, String hostname) throws IOException; + + /** + * Restart the service on the given host + */ + public abstract void restart(ServiceType service, String hostname) throws IOException; + + /** + * Send the given posix signal to the service + */ + public abstract void signal(ServiceType service, String signal, + String hostname) throws IOException; + + /** + * Kill the service running on given host + */ + public void kill(ServiceType service, String hostname) throws IOException { + signal(service, SIGKILL, hostname); + } + + /** + * Suspend the service running on given host + */ + public void suspend(ServiceType service, String hostname) throws IOException { + signal(service, SIGSTOP, hostname); + } + + /** + * Resume the services running on given hosts + */ + public void resume(ServiceType service, String hostname) throws IOException { + signal(service, SIGCONT, hostname); + } + + /** + * Returns whether the service is running on the remote host. This only checks whether the + * service still has a pid. + */ + public abstract boolean isRunning(ServiceType service, String hostname) throws IOException; + + /* TODO: further API ideas: + * + * //return services running on host: + * ServiceType[] getRunningServicesOnHost(String hostname); + * + * //return which services can be run on host (for example, to query whether hmaster can run on this host) + * ServiceType[] getRunnableServicesOnHost(String hostname); + * + * //return which hosts can run this service + * String[] getRunnableHostsForService(ServiceType service); + */ + +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/DistributedHBaseCluster.java b/src/test/java/org/apache/hadoop/hbase/DistributedHBaseCluster.java new file mode 100644 index 000000000000..7aa53803a936 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/DistributedHBaseCluster.java @@ -0,0 +1,291 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import java.io.IOException; +import java.util.HashMap; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.ClusterManager.ServiceType; +import org.apache.hadoop.hbase.ipc.HRegionInterface; +import org.apache.hadoop.hbase.ipc.HMasterInterface; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HConnection; +import org.apache.hadoop.hbase.client.HConnectionManager; +import org.apache.hadoop.hbase.util.Threads; + +import com.google.common.collect.Sets; + +/** + * Manages the interactions with an already deployed distributed cluster (as opposed to + * a pseudo-distributed, or mini/local cluster). This is used by integration and system tests. + */ +@InterfaceAudience.Private +public class DistributedHBaseCluster extends HBaseCluster { + + private HBaseAdmin admin; + + private ClusterManager clusterManager; + + public DistributedHBaseCluster(Configuration conf, ClusterManager clusterManager) + throws IOException { + super(conf); + this.clusterManager = clusterManager; + this.admin = new HBaseAdmin(conf); + this.initialClusterStatus = getClusterStatus(); + } + + public void setClusterManager(ClusterManager clusterManager) { + this.clusterManager = clusterManager; + } + + public ClusterManager getClusterManager() { + return clusterManager; + } + + /** + * Returns a ClusterStatus for this HBase cluster + * @throws IOException + */ + @Override + public ClusterStatus getClusterStatus() throws IOException { + return admin.getClusterStatus(); + } + + @Override + public ClusterStatus getInitialClusterStatus() throws IOException { + return initialClusterStatus; + } + + @Override + public void close() throws IOException { + if (this.admin != null) { + admin.close(); + } + } + + @Override + public void startRegionServer(String hostname) throws IOException { + LOG.info("Starting RS on: " + hostname); + clusterManager.start(ServiceType.HBASE_REGIONSERVER, hostname); + } + + @Override + public void killRegionServer(ServerName serverName) throws IOException { + LOG.info("Aborting RS: " + serverName.getServerName()); + clusterManager.kill(ServiceType.HBASE_REGIONSERVER, serverName.getHostname()); + } + + @Override + public void stopRegionServer(ServerName serverName) throws IOException { + LOG.info("Stopping RS: " + serverName.getServerName()); + clusterManager.stop(ServiceType.HBASE_REGIONSERVER, serverName.getHostname()); + } + + @Override + public void waitForRegionServerToStop(ServerName serverName, long timeout) throws IOException { + waitForServiceToStop(ServiceType.HBASE_REGIONSERVER, serverName, timeout); + } + + private void waitForServiceToStop(ServiceType service, ServerName serverName, long timeout) + throws IOException { + LOG.info("Waiting service:" + service + " to stop: " + serverName.getServerName()); + long start = System.currentTimeMillis(); + + while ((System.currentTimeMillis() - start) < timeout) { + if (!clusterManager.isRunning(service, serverName.getHostname())) { + return; + } + Threads.sleep(1000); + } + throw new IOException("did timeout waiting for service to stop:" + serverName); + } + + @Override + public HMasterInterface getMasterAdmin() throws IOException { + HConnection conn = HConnectionManager.getConnection(conf); + return conn.getMaster(); + } + + @Override + public void startMaster(String hostname) throws IOException { + LOG.info("Starting Master on: " + hostname); + clusterManager.start(ServiceType.HBASE_MASTER, hostname); + } + + @Override + public void killMaster(ServerName serverName) throws IOException { + LOG.info("Aborting Master: " + serverName.getServerName()); + clusterManager.kill(ServiceType.HBASE_MASTER, serverName.getHostname()); + } + + @Override + public void stopMaster(ServerName serverName) throws IOException { + LOG.info("Stopping Master: " + serverName.getServerName()); + clusterManager.stop(ServiceType.HBASE_MASTER, serverName.getHostname()); + } + + @Override + public void waitForMasterToStop(ServerName serverName, long timeout) throws IOException { + waitForServiceToStop(ServiceType.HBASE_MASTER, serverName, timeout); + } + + @Override + public boolean waitForActiveAndReadyMaster(long timeout) throws IOException { + long start = System.currentTimeMillis(); + while (System.currentTimeMillis() - start < timeout) { + try { + getMasterAdmin(); + return true; + } catch (MasterNotRunningException m) { + LOG.warn("Master not started yet " + m); + } catch (ZooKeeperConnectionException e) { + LOG.warn("Failed to connect to ZK " + e); + } + Threads.sleep(1000); + } + return false; + } + + @Override + public ServerName getServerHoldingRegion(byte[] regionName) throws IOException { + HConnection connection = admin.getConnection(); + HRegionLocation regionLoc = connection.locateRegion(regionName); + if (regionLoc == null) { + return null; + } + + org.apache.hadoop.hbase.HServerInfo sn + = connection.getHRegionConnection(regionLoc.getHostname(), regionLoc.getPort()).getHServerInfo(); + + return new ServerName(sn.getServerAddress().getHostname(), sn.getServerAddress().getPort(), sn.getStartCode()); + } + + @Override + public void waitUntilShutDown() { + //Simply wait for a few seconds for now (after issuing serverManager.kill + throw new RuntimeException("Not implemented yet"); + } + + @Override + public void shutdown() throws IOException { + //not sure we want this + throw new RuntimeException("Not implemented yet"); + } + + @Override + public boolean isDistributedCluster() { + return true; + } + + @Override + public void restoreClusterStatus(ClusterStatus initial) throws IOException { + //TODO: caution: not tested throughly + ClusterStatus current = getClusterStatus(); + + //restore masters + + //check whether current master has changed + if (!ServerName.isSameHostnameAndPort(initial.getMaster(), current.getMaster())) { + LOG.info("Initial active master : " + initial.getMaster().getHostname() + + " has changed to : " + current.getMaster().getHostname()); + // If initial master is stopped, start it, before restoring the state. + // It will come up as a backup master, if there is already an active master. + if (!clusterManager.isRunning(ServiceType.HBASE_MASTER, initial.getMaster().getHostname())) { + startMaster(initial.getMaster().getHostname()); + } + + //master has changed, we would like to undo this. + //1. Kill the current backups + //2. Stop current master + //3. Start backup masters + for (ServerName currentBackup : current.getBackupMasters()) { + if (!ServerName.isSameHostnameAndPort(currentBackup, initial.getMaster())) { + stopMaster(currentBackup); + } + } + stopMaster(current.getMaster()); + waitForActiveAndReadyMaster(); //wait so that active master takes over + //start backup masters + for (ServerName backup : initial.getBackupMasters()) { + //these are not started in backup mode, but we should already have an active master + if(!clusterManager.isRunning(ServiceType.HBASE_MASTER, backup.getHostname())) { + startMaster(backup.getHostname()); + } + } + } else { + //current master has not changed, match up backup masters + HashMap initialBackups = new HashMap(); + HashMap currentBackups = new HashMap(); + + for (ServerName server : initial.getBackupMasters()) { + initialBackups.put(server.getHostname(), server); + } + for (ServerName server : current.getBackupMasters()) { + currentBackups.put(server.getHostname(), server); + } + + for (String hostname : Sets.difference(initialBackups.keySet(), currentBackups.keySet())) { + if(!clusterManager.isRunning(ServiceType.HBASE_MASTER, hostname)) { + startMaster(hostname); + } + } + + for (String hostname : Sets.difference(currentBackups.keySet(), initialBackups.keySet())) { + if(clusterManager.isRunning(ServiceType.HBASE_MASTER, hostname)) { + stopMaster(currentBackups.get(hostname)); + } + } + } + + //restore region servers + HashMap initialServers = new HashMap(); + HashMap currentServers = new HashMap(); + + for (ServerName server : initial.getServers()) { + initialServers.put(server.getHostname(), server); + } + for (ServerName server : current.getServers()) { + currentServers.put(server.getHostname(), server); + } + + for (String hostname : Sets.difference(initialServers.keySet(), currentServers.keySet())) { + if(!clusterManager.isRunning(ServiceType.HBASE_REGIONSERVER, hostname)) { + startRegionServer(hostname); + } + } + + for (String hostname : Sets.difference(currentServers.keySet(), initialServers.keySet())) { + if(clusterManager.isRunning(ServiceType.HBASE_REGIONSERVER, hostname)) { + stopRegionServer(currentServers.get(hostname)); + } + } + // While restoring above, if the HBase Master which was initially the Active one, was down + // and the restore put the cluster back to Initial configuration, HAdmin instance will need + // to refresh its connections (otherwise it will return incorrect information) or we can + // point it to new instance. + try { + admin.close(); + } catch (IOException ioe) { + LOG.info("While closing the old connection", ioe); + } + this.admin = new HBaseAdmin(conf); + LOG.info("Added new HBaseAdmin"); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/HBaseCluster.java b/src/test/java/org/apache/hadoop/hbase/HBaseCluster.java new file mode 100644 index 000000000000..4a5e39420b05 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/HBaseCluster.java @@ -0,0 +1,264 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import java.io.Closeable; +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configurable; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.util.Threads; +import org.apache.hadoop.hbase.ipc.HMasterInterface; +import org.apache.hadoop.hbase.ipc.HRegionInterface; + +/** + * This class defines methods that can help with managing HBase clusters + * from unit tests and system tests. There are 3 types of cluster deployments: + *
    + *
  • MiniHBaseCluster: each server is run in the same JVM in separate threads, + * used by unit tests
    • + *
  • DistributedHBaseCluster: the cluster is pre-deployed, system and integration tests can + * interact with the cluster.
    • + *
  • ProcessBasedLocalHBaseCluster: each server is deployed locally but in separate + * JVMs.
    • + *
+ *

+ * HBaseCluster unifies the way tests interact with the cluster, so that the same test can + * be run against a mini-cluster during unit test execution, or a distributed cluster having + * tens/hundreds of nodes during execution of integration tests. + * + *

+ * HBaseCluster exposes client-side public interfaces to tests, so that tests does not assume + * running in a particular mode. Not all the tests are suitable to be run on an actual cluster, + * and some tests will still need to mock stuff and introspect internal state. For those use + * cases from unit tests, or if more control is needed, you can use the subclasses directly. + * In that sense, this class does not abstract away every interface that + * MiniHBaseCluster or DistributedHBaseCluster provide. + */ +@InterfaceAudience.Private +public abstract class HBaseCluster implements Closeable, Configurable { + static final Log LOG = LogFactory.getLog(HBaseCluster.class.getName()); + protected Configuration conf; + + /** the status of the cluster before we begin */ + protected ClusterStatus initialClusterStatus; + + /** + * Construct an HBaseCluster + * @param conf Configuration to be used for cluster + */ + public HBaseCluster(Configuration conf) { + setConf(conf); + } + + @Override + public void setConf(Configuration conf) { + this.conf = conf; + } + + @Override + public Configuration getConf() { + return conf; + } + + /** + * Returns a ClusterStatus for this HBase cluster. + * @see #getInitialClusterStatus() + */ + public abstract ClusterStatus getClusterStatus() throws IOException; + + /** + * Returns a ClusterStatus for this HBase cluster as observed at the + * starting of the HBaseCluster + */ + public ClusterStatus getInitialClusterStatus() throws IOException { + return initialClusterStatus; + } + + /** + * Returns an {@link HmasterInterface} to the active master + */ + public abstract HMasterInterface getMasterAdmin() + throws IOException; + + /** + * Starts a new region server on the given hostname or if this is a mini/local cluster, + * starts a region server locally. + * @param hostname the hostname to start the regionserver on + * @throws IOException if something goes wrong + */ + public abstract void startRegionServer(String hostname) throws IOException; + + /** + * Kills the region server process if this is a distributed cluster, otherwise + * this causes the region server to exit doing basic clean up only. + * @throws IOException if something goes wrong + */ + public abstract void killRegionServer(ServerName serverName) throws IOException; + + /** + * Stops the given region server, by attempting a gradual stop. + * @return whether the operation finished with success + * @throws IOException if something goes wrong + */ + public abstract void stopRegionServer(ServerName serverName) throws IOException; + + /** + * Wait for the specified region server to join the cluster + * @return whether the operation finished with success + * @throws IOException if something goes wrong or timeout occurs + */ + public void waitForRegionServerToStart(String hostname, long timeout) + throws IOException { + long start = System.currentTimeMillis(); + while ((System.currentTimeMillis() - start) < timeout) { + for (ServerName server : getClusterStatus().getServers()) { + if (server.getHostname().equals(hostname)) { + return; + } + } + Threads.sleep(100); + } + throw new IOException("did timeout waiting for region server to start:" + hostname); + } + + /** + * Wait for the specified region server to stop the thread / process. + * @return whether the operation finished with success + * @throws IOException if something goes wrong or timeout occurs + */ + public abstract void waitForRegionServerToStop(ServerName serverName, long timeout) + throws IOException; + + /** + * Starts a new master on the given hostname or if this is a mini/local cluster, + * starts a master locally. + * @param hostname the hostname to start the master on + * @return whether the operation finished with success + * @throws IOException if something goes wrong + */ + public abstract void startMaster(String hostname) throws IOException; + + /** + * Kills the master process if this is a distributed cluster, otherwise, + * this causes master to exit doing basic clean up only. + * @throws IOException if something goes wrong + */ + public abstract void killMaster(ServerName serverName) throws IOException; + + /** + * Stops the given master, by attempting a gradual stop. + * @throws IOException if something goes wrong + */ + public abstract void stopMaster(ServerName serverName) throws IOException; + + /** + * Wait for the specified master to stop the thread / process. + * @throws IOException if something goes wrong or timeout occurs + */ + public abstract void waitForMasterToStop(ServerName serverName, long timeout) + throws IOException; + + /** + * Blocks until there is an active master and that master has completed + * initialization. + * + * @return true if an active master becomes available. false if there are no + * masters left. + * @throws IOException if something goes wrong or timeout occurs + */ + public boolean waitForActiveAndReadyMaster() + throws IOException { + return waitForActiveAndReadyMaster(Long.MAX_VALUE); + } + + /** + * Blocks until there is an active master and that master has completed + * initialization. + * @param timeout the timeout limit in ms + * @return true if an active master becomes available. false if there are no + * masters left. + */ + public abstract boolean waitForActiveAndReadyMaster(long timeout) + throws IOException; + + /** + * Wait for HBase Cluster to shut down. + */ + public abstract void waitUntilShutDown() throws IOException; + + /** + * Shut down the HBase cluster + */ + public abstract void shutdown() throws IOException; + + /** + * Restores the cluster to it's initial state if this is a real cluster, + * otherwise does nothing. + */ + public void restoreInitialStatus() throws IOException { + restoreClusterStatus(getInitialClusterStatus()); + } + + /** + * Restores the cluster to given state if this is a real cluster, + * otherwise does nothing. + */ + public void restoreClusterStatus(ClusterStatus desiredStatus) throws IOException { + } + + /** + * Get the ServerName of region server serving ROOT region + */ + public ServerName getServerHoldingRoot() throws IOException { + return getServerHoldingRegion(HRegionInfo.ROOT_REGIONINFO.getRegionName()); + } + + /** + * Get the ServerName of region server serving the first META region + */ + public ServerName getServerHoldingMeta() throws IOException { + return getServerHoldingRegion(HRegionInfo.FIRST_META_REGIONINFO.getRegionName()); + } + + /** + * Get the ServerName of region server serving the specified region + * @param regionName Name of the region in bytes + * @return ServerName that hosts the region or null + */ + public abstract ServerName getServerHoldingRegion(byte[] regionName) throws IOException; + + /** + * @return whether we are interacting with a distributed cluster as opposed to an + * in-process mini/local cluster. + */ + public boolean isDistributedCluster() { + return false; + } + + /** + * Closes all the resources held open for this cluster. Note that this call does not shutdown + * the cluster. + * @see #shutdown() + */ + @Override + public abstract void close() throws IOException; +} diff --git a/src/test/java/org/apache/hadoop/hbase/HBaseClusterManager.java b/src/test/java/org/apache/hadoop/hbase/HBaseClusterManager.java new file mode 100644 index 000000000000..bc3b3fd7f496 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/HBaseClusterManager.java @@ -0,0 +1,222 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import java.io.File; +import java.io.IOException; +import java.util.Map; + +import org.apache.commons.lang.StringUtils; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseClusterManager.CommandProvider.Operation; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.hadoop.util.Shell; + +/** + * A default cluster manager for HBase. Uses SSH, and hbase shell scripts + * to manage the cluster. Assumes Unix-like commands are available like 'ps', + * 'kill', etc. Also assumes the user running the test has enough "power" to start & stop + * servers on the remote machines (for example, the test user could be the same user as the + * user the daemon isrunning as) + */ +@InterfaceAudience.Private +public class HBaseClusterManager extends ClusterManager { + private String sshUserName; + private String sshOptions; + + /** + * The command format that is used to execute the remote command. Arguments: + * 1 SSH options, 2 user name , 3 "@" if username is set, 4 host, 5 original command. + */ + private static final String DEFAULT_TUNNEL_CMD = "/usr/bin/ssh %1$s %2$s%3$s%4$s \"%5$s\""; + private String tunnelCmd; + + @Override + public void setConf(Configuration conf) { + super.setConf(conf); + if (conf == null) { + // Configured gets passed null before real conf. Why? I don't know. + return; + } + sshUserName = conf.get("hbase.it.clustermanager.ssh.user", ""); + String extraSshOptions = conf.get("hbase.it.clustermanager.ssh.opts", ""); + sshOptions = System.getenv("HBASE_SSH_OPTS"); + if (!extraSshOptions.isEmpty()) { + sshOptions = StringUtils.join(new Object[] { sshOptions, extraSshOptions }, " "); + } + sshOptions = (sshOptions == null) ? "" : sshOptions; + tunnelCmd = conf.get("hbase.it.clustermanager.ssh.cmd", DEFAULT_TUNNEL_CMD); + LOG.info("Running with SSH user [" + sshUserName + "] and options [" + sshOptions + "]"); + } + + /** + * Executes commands over SSH + */ + protected class RemoteShell extends Shell.ShellCommandExecutor { + private String hostname; + + public RemoteShell(String hostname, String[] execString, File dir, Map env, + long timeout) { + super(execString, dir, env, timeout); + this.hostname = hostname; + } + + public RemoteShell(String hostname, String[] execString, File dir, Map env) { + super(execString, dir, env); + this.hostname = hostname; + } + + public RemoteShell(String hostname, String[] execString, File dir) { + super(execString, dir); + this.hostname = hostname; + } + + public RemoteShell(String hostname, String[] execString) { + super(execString); + this.hostname = hostname; + } + + public String[] getExecString() { + String at = sshUserName.isEmpty() ? "" : "@"; + String remoteCmd = StringUtils.join(super.getExecString(), " "); + String cmd = String.format(tunnelCmd, sshOptions, sshUserName, at, hostname, remoteCmd); + LOG.info("Executing full command [" + cmd + "]"); + return new String[] { "/usr/bin/env", "bash", "-c", cmd }; + } + + @Override + public void execute() throws IOException { + super.execute(); + } + } + + /** + * Provides command strings for services to be executed by Shell. CommandProviders are + * pluggable, and different deployments(windows, bigtop, etc) can be managed by + * plugging-in custom CommandProvider's or ClusterManager's. + */ + static abstract class CommandProvider { + + enum Operation { + START, STOP, RESTART + } + + public abstract String getCommand(ServiceType service, Operation op); + + public String isRunningCommand(ServiceType service) { + return findPidCommand(service); + } + + protected String findPidCommand(ServiceType service) { + String servicePathFilter = ""; + if (service == ServiceType.HBASE_MASTER || service == ServiceType.HBASE_REGIONSERVER) { + servicePathFilter = " | grep hbase"; + } + return String.format("ps ux | grep %s %s | grep -v grep | tr -s ' ' | cut -d ' ' -f2", + service, servicePathFilter); + } + + public String signalCommand(ServiceType service, String signal) { + return String.format("%s | xargs kill -s %s", findPidCommand(service), signal); + } + } + + /** + * CommandProvider to manage the service using bin/hbase-* scripts + */ + static class HBaseShellCommandProvider extends CommandProvider { + private String getHBaseHome() { + return System.getenv("HBASE_HOME"); + } + + private String getConfig() { + String confDir = System.getenv("HBASE_CONF_DIR"); + if (confDir != null) { + return String.format("--config %s", confDir); + } + return ""; + } + + @Override + public String getCommand(ServiceType service, Operation op) { + return String.format("%s/bin/hbase-daemon.sh %s %s %s", getHBaseHome(), getConfig(), + op.toString().toLowerCase(), service); + } + } + + public HBaseClusterManager() { + super(); + } + + protected CommandProvider getCommandProvider(ServiceType service) { + //TODO: make it pluggable, or auto-detect the best command provider, should work with + //hadoop daemons as well + return new HBaseShellCommandProvider(); + } + + /** + * Execute the given command on the host using SSH + * @return pair of exit code and command output + * @throws IOException if something goes wrong. + */ + private Pair exec(String hostname, String... cmd) throws IOException { + LOG.info("Executing remote command: " + StringUtils.join(cmd, " ") + " , hostname:" + hostname); + + RemoteShell shell = new RemoteShell(hostname, cmd); + shell.execute(); + + LOG.info("Executed remote command, exit code:" + shell.getExitCode() + + " , output:" + shell.getOutput()); + + return new Pair(shell.getExitCode(), shell.getOutput()); + } + + private void exec(String hostname, ServiceType service, Operation op) throws IOException { + exec(hostname, getCommandProvider(service).getCommand(service, op)); + } + + @Override + public void start(ServiceType service, String hostname) throws IOException { + exec(hostname, service, Operation.START); + } + + @Override + public void stop(ServiceType service, String hostname) throws IOException { + exec(hostname, service, Operation.STOP); + } + + @Override + public void restart(ServiceType service, String hostname) throws IOException { + exec(hostname, service, Operation.RESTART); + } + + @Override + public void signal(ServiceType service, String signal, String hostname) throws IOException { + exec(hostname, getCommandProvider(service).signalCommand(service, signal)); + } + + @Override + public boolean isRunning(ServiceType service, String hostname) throws IOException { + String ret = exec(hostname, getCommandProvider(service).isRunningCommand(service)) + .getSecond(); + return ret.length() > 0; + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java b/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java index 5fb9e2c81bce..0abfeeed50fe 100644 --- a/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java +++ b/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -71,7 +70,7 @@ public abstract class HBaseTestCase extends TestCase { protected static final byte [][] COLUMNS = {fam1, fam2, fam3}; private boolean localfs = false; - protected Path testDir = null; + protected static Path testDir = null; protected FileSystem fs = null; protected HRegion root = null; protected HRegion meta = null; @@ -170,8 +169,24 @@ protected Path getUnitTestdir(String testName) { ); } - protected HRegion createNewHRegion(HTableDescriptor desc, byte [] startKey, + /** + * You must call close on the returned region and then close on the log file + * it created. Do {@link HRegion#close()} followed by {@link HRegion#getLog()} + * and on it call close. + * @param desc + * @param startKey + * @param endKey + * @return An {@link HRegion} + * @throws IOException + */ + public HRegion createNewHRegion(HTableDescriptor desc, byte [] startKey, byte [] endKey) + throws IOException { + return createNewHRegion(desc, startKey, endKey, this.conf); + } + + public HRegion createNewHRegion(HTableDescriptor desc, byte [] startKey, + byte [] endKey, Configuration conf) throws IOException { FileSystem filesystem = FileSystem.get(conf); HRegionInfo hri = new HRegionInfo(desc.getName(), startKey, endKey); @@ -180,9 +195,7 @@ protected HRegion createNewHRegion(HTableDescriptor desc, byte [] startKey, protected HRegion openClosedRegion(final HRegion closedRegion) throws IOException { - HRegion r = new HRegion(closedRegion.getTableDir(), closedRegion.getLog(), - closedRegion.getFilesystem(), closedRegion.getConf(), - closedRegion.getRegionInfo(), closedRegion.getTableDesc(), null); + HRegion r = new HRegion(closedRegion); r.initialize(); return r; } @@ -238,10 +251,11 @@ protected HTableDescriptor createTableDescriptor(final String name, * Adds data of the from 'aaa', 'aab', etc where key and value are the same. * @param r * @param columnFamily + * @param column * @throws IOException * @return count of what we added. */ - protected static long addContent(final HRegion r, final byte [] columnFamily) + public static long addContent(final HRegion r, final byte [] columnFamily, final byte[] column) throws IOException { byte [] startKey = r.getRegionInfo().getStartKey(); byte [] endKey = r.getRegionInfo().getEndKey(); @@ -249,10 +263,24 @@ protected static long addContent(final HRegion r, final byte [] columnFamily) if (startKeyBytes == null || startKeyBytes.length == 0) { startKeyBytes = START_KEY_BYTES; } - return addContent(new HRegionIncommon(r), Bytes.toString(columnFamily), null, + return addContent(new HRegionIncommon(r), Bytes.toString(columnFamily), Bytes.toString(column), startKeyBytes, endKey, -1); } + /** + * Add content to region r on the passed column + * column. + * Adds data of the from 'aaa', 'aab', etc where key and value are the same. + * @param r + * @param columnFamily + * @throws IOException + * @return count of what we added. + */ + protected static long addContent(final HRegion r, final byte [] columnFamily) + throws IOException { + return addContent(r, columnFamily, null); + } + /** * Add content to region r on the passed column * column. @@ -263,12 +291,12 @@ protected static long addContent(final HRegion r, final byte [] columnFamily) * @return count of what we added. */ protected static long addContent(final Incommon updater, - final String columnFamily) throws IOException { + final String columnFamily) throws IOException { return addContent(updater, columnFamily, START_KEY_BYTES, null); } protected static long addContent(final Incommon updater, final String family, - final String column) throws IOException { + final String column) throws IOException { return addContent(updater, family, column, START_KEY_BYTES, null); } @@ -673,6 +701,11 @@ public static void shutdownDfs(MiniDFSCluster cluster) { } } + /** + * You must call {@link #closeRootAndMeta()} when done after calling this + * method. It does cleanup. + * @throws IOException + */ protected void createRootAndMetaRegions() throws IOException { root = HRegion.createHRegion(HRegionInfo.ROOT_REGIONINFO, testDir, conf, HTableDescriptor.ROOT_TABLEDESC); diff --git a/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java b/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java index 498075d1bb60..1feea38bdde2 100644 --- a/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java +++ b/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -25,7 +24,9 @@ import java.io.IOException; import java.io.OutputStream; import java.lang.reflect.Field; +import java.lang.reflect.Method; import java.net.InetAddress; +import java.net.InetSocketAddress; import java.net.ServerSocket; import java.net.Socket; import java.net.UnknownHostException; @@ -56,10 +57,15 @@ import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.fs.HFileSystem; import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; +import org.apache.hadoop.hbase.io.hfile.ChecksumUtil; import org.apache.hadoop.hbase.io.hfile.Compression; import org.apache.hadoop.hbase.io.hfile.Compression.Algorithm; +import org.apache.hadoop.hbase.io.hfile.HFile; +import org.apache.hadoop.hbase.mapreduce.MapreduceTestingShim; import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.master.ServerManager; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.HRegionServer; import org.apache.hadoop.hbase.regionserver.InternalScanner; @@ -69,6 +75,8 @@ import org.apache.hadoop.hbase.security.User; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.JVMClusterUtil; +import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread; import org.apache.hadoop.hbase.util.RegionSplitter; import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.hbase.util.Writables; @@ -79,9 +87,12 @@ import org.apache.hadoop.hdfs.DFSClient; import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapred.MiniMRCluster; +import org.apache.hadoop.security.UserGroupInformation; import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.KeeperException.NodeExistsException; +import org.apache.zookeeper.WatchedEvent; import org.apache.zookeeper.ZooKeeper; /** @@ -89,7 +100,9 @@ * old HBaseTestCase and HBaseClusterTestCase functionality. * Create an instance and keep it around testing HBase. This class is * meant to be your one-stop shop for anything you might need testing. Manages - * one cluster at a time only. + * one cluster at a time only. Managed cluster can be an in-process + * {@link MiniHBaseCluster}, or a deployed cluster of type {@link DistributedHBaseCluster}. + * Not all methods work with the real cluster. * Depends on log4j being on classpath and * hbase-site.xml for logging and test-run configuration. It does not set * logging levels nor make changes to configuration parameters. @@ -112,7 +125,7 @@ public class HBaseTestingUtility { private boolean passedZkCluster = false; private MiniDFSCluster dfsCluster = null; - private MiniHBaseCluster hbaseCluster = null; + private HBaseCluster hbaseCluster = null; private MiniMRCluster mrCluster = null; // Directory where we put the data for this instance of HBaseTestingUtility @@ -185,6 +198,21 @@ public HBaseTestingUtility() { public HBaseTestingUtility(Configuration conf) { this.conf = conf; + + // a hbase checksum verification failure will cause unit tests to fail + ChecksumUtil.generateExceptionForChecksumFailureForTest(true); + setHDFSClientRetry(1); + } + + /** + * Controls how many attempts we will make in the face of failures in HDFS. + */ + public void setHDFSClientRetry(final int retries) { + this.conf.setInt("hdfs.client.retries.number", retries); + HBaseFileSystem.setRetryCounts(conf); + if (0 == retries) { + makeDFSClientNonRetrying(); + } } /** @@ -202,6 +230,10 @@ public Configuration getConfiguration() { return this.conf; } + public void setHBaseCluster(HBaseCluster hbaseCluster) { + this.hbaseCluster = hbaseCluster; + } + /** * @return Where to write test data on local filesystem; usually * {@link #DEFAULT_BASE_TEST_DIRECTORY} @@ -299,6 +331,10 @@ private void setupDataTestDir() { createSubDirAndSystemProperty( "mapred.working.dir", testPath, "mapred-working-dir"); + + createSubDir( + "hbase.local.dir", + testPath, "hbase-local-dir"); } private void createSubDir(String propertyName, Path parent, String subDirName){ @@ -618,9 +654,13 @@ public MiniHBaseCluster startMiniHBaseCluster(final int numMasters, createRootDir(); // These settings will make the server waits until this exact number of - // regions servers are connected. - conf.setInt("hbase.master.wait.on.regionservers.mintostart", numSlaves); - conf.setInt("hbase.master.wait.on.regionservers.maxtostart", numSlaves); + // regions servers are connected. + if (conf.getInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, -1) == -1) { + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, numSlaves); + } + if (conf.getInt(ServerManager.WAIT_ON_REGIONSERVERS_MAXTOSTART, -1) == -1) { + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MAXTOSTART, numSlaves); + } Configuration c = new Configuration(this.conf); this.hbaseCluster = new MiniHBaseCluster(c, numMasters, numSlaves); @@ -635,7 +675,7 @@ public MiniHBaseCluster startMiniHBaseCluster(final int numMasters, getHBaseAdmin(); // create immediately the hbaseAdmin LOG.info("Minicluster is up"); - return this.hbaseCluster; + return (MiniHBaseCluster)this.hbaseCluster; } /** @@ -663,7 +703,11 @@ public void restartHBaseCluster(int servers) throws IOException, InterruptedExce * @see #startMiniCluster() */ public MiniHBaseCluster getMiniHBaseCluster() { - return this.hbaseCluster; + if (this.hbaseCluster == null || this.hbaseCluster instanceof MiniHBaseCluster) { + return (MiniHBaseCluster)this.hbaseCluster; + } + throw new RuntimeException(hbaseCluster + " not an instance of " + + MiniHBaseCluster.class.getName()); } /** @@ -700,10 +744,13 @@ public void shutdownMiniHBaseCluster() throws IOException { hbaseAdmin.close(); hbaseAdmin = null; } + // unset the configuration for MIN and MAX RS to start + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, -1); + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MAXTOSTART, -1); if (this.hbaseCluster != null) { this.hbaseCluster.shutdown(); // Wait till hbase is down before going on to shutdown zk. - this.hbaseCluster.join(); + this.hbaseCluster.waitUntilShutDown(); this.hbaseCluster = null; } } @@ -741,7 +788,7 @@ public Path createRootDir() throws IOException { * @throws IOException */ public void flush() throws IOException { - this.hbaseCluster.flushcache(); + getMiniHBaseCluster().flushcache(); } /** @@ -749,7 +796,23 @@ public void flush() throws IOException { * @throws IOException */ public void flush(byte [] tableName) throws IOException { - this.hbaseCluster.flushcache(tableName); + getMiniHBaseCluster().flushcache(tableName); + } + + /** + * Compact all regions in the mini hbase cluster + * @throws IOException + */ + public void compact(boolean major) throws IOException { + getMiniHBaseCluster().compact(major); + } + + /** + * Compact all of a table's reagion in the mini hbase cluster + * @throws IOException + */ + public void compact(byte [] tableName, boolean major) throws IOException { + getMiniHBaseCluster().compact(tableName, major); } @@ -788,6 +851,8 @@ public HTable createTable(byte[] tableName, byte[][] families, desc.addFamily(hcd); } getHBaseAdmin().createTable(desc, startKey, endKey, numRegions); + // HBaseAdmin only waits for regions to appear in META we should wait until they are assigned + waitUntilAllRegionsAssigned(tableName); return new HTable(getConfiguration(), tableName); } @@ -807,6 +872,8 @@ public HTable createTable(byte[] tableName, byte[][] families, desc.addFamily(new HColumnDescriptor(family)); } getHBaseAdmin().createTable(desc); + // HBaseAdmin only waits for regions to appear in META we should wait until they are assigned + waitUntilAllRegionsAssigned(tableName); return new HTable(c, tableName); } @@ -829,6 +896,8 @@ public HTable createTable(byte[] tableName, byte[][] families, desc.addFamily(hcd); } getHBaseAdmin().createTable(desc); + // HBaseAdmin only waits for regions to appear in META we should wait until they are assigned + waitUntilAllRegionsAssigned(tableName); return new HTable(c, tableName); } @@ -863,6 +932,8 @@ public HTable createTable(byte[] tableName, byte[][] families, desc.addFamily(hcd); } getHBaseAdmin().createTable(desc); + // HBaseAdmin only waits for regions to appear in META we should wait until they are assigned + waitUntilAllRegionsAssigned(tableName); return new HTable(new Configuration(getConfiguration()), tableName); } @@ -884,6 +955,8 @@ public HTable createTable(byte[] tableName, byte[][] families, desc.addFamily(hcd); } getHBaseAdmin().createTable(desc); + // HBaseAdmin only waits for regions to appear in META we should wait until they are assigned + waitUntilAllRegionsAssigned(tableName); return new HTable(new Configuration(getConfiguration()), tableName); } @@ -907,6 +980,8 @@ public HTable createTable(byte[] tableName, byte[][] families, i++; } getHBaseAdmin().createTable(desc); + // HBaseAdmin only waits for regions to appear in META we should wait until they are assigned + waitUntilAllRegionsAssigned(tableName); return new HTable(new Configuration(getConfiguration()), tableName); } @@ -915,7 +990,11 @@ public HTable createTable(byte[] tableName, byte[][] families, * @param tableName existing table */ public void deleteTable(byte[] tableName) throws IOException { - getHBaseAdmin().disableTable(tableName); + try { + getHBaseAdmin().disableTable(tableName); + } catch (TableNotEnabledException e) { + LOG.debug("Table: " + Bytes.toString(tableName) + " already disabled, so just deleting it."); + } getHBaseAdmin().deleteTable(tableName); } @@ -965,6 +1044,94 @@ public int loadTable(final HTable t, final byte[] f) throws IOException { t.flushCommits(); return rowCount; } + + /** + * Load table of multiple column families with rows from 'aaa' to 'zzz'. + * @param t Table + * @param f Array of Families to load + * @return Count of rows loaded. + * @throws IOException + */ + public int loadTable(final HTable t, final byte[][] f) throws IOException { + return loadTable(t, f, null); + } + + /** + * Load table of multiple column families with rows from 'aaa' to 'zzz'. + * @param t Table + * @param f Array of Families to load + * @param value the values of the KVs. If null is passed, the row key is used as value + * @return Count of rows loaded. + * @throws IOException + */ + public int loadTable(final HTable t, final byte[][] f, byte[] value) throws IOException { + t.setAutoFlush(false); + byte[] k = new byte[3]; + int rowCount = 0; + for (byte b1 = 'a'; b1 <= 'z'; b1++) { + for (byte b2 = 'a'; b2 <= 'z'; b2++) { + for (byte b3 = 'a'; b3 <= 'z'; b3++) { + k[0] = b1; + k[1] = b2; + k[2] = b3; + Put put = new Put(k); + for (int i = 0; i < f.length; i++) { + put.add(f[i], null, value != null ? value : k); + } + t.put(put); + rowCount++; + } + } + } + t.flushCommits(); + return rowCount; + } + + /** A tracker for tracking and validating table rows + * generated with {@link HBaseTestingUtility#loadTable(HTable, byte[])} + */ + public static class SeenRowTracker { + int dim = 'z' - 'a' + 1; + int[][][] seenRows = new int[dim][dim][dim]; //count of how many times the row is seen + byte[] startRow; + byte[] stopRow; + + public SeenRowTracker(byte[] startRow, byte[] stopRow) { + this.startRow = startRow; + this.stopRow = stopRow; + } + + int i(byte b) { + return b - 'a'; + } + + public void addRow(byte[] row) { + seenRows[i(row[0])][i(row[1])][i(row[2])]++; + } + + /** Validate that all the rows between startRow and stopRow are seen exactly once, and + * all other rows none + */ + public void validate() { + for (byte b1 = 'a'; b1 <= 'z'; b1++) { + for (byte b2 = 'a'; b2 <= 'z'; b2++) { + for (byte b3 = 'a'; b3 <= 'z'; b3++) { + int count = seenRows[i(b1)][i(b2)][i(b3)]; + int expectedCount = 0; + if (Bytes.compareTo(new byte[] {b1,b2,b3}, startRow) >= 0 + && Bytes.compareTo(new byte[] {b1,b2,b3}, stopRow) < 0) { + expectedCount = 1; + } + if (count != expectedCount) { + String row = new String(new byte[] {b1,b2,b3}); + throw new RuntimeException("Row:" + row + " has a seen count of " + count + " instead of " + expectedCount); + } + } + } + } + } + } + /** * Load region with rows from 'aaa' to 'zzz'. * @param r Region @@ -974,6 +1141,19 @@ public int loadTable(final HTable t, final byte[] f) throws IOException { */ public int loadRegion(final HRegion r, final byte[] f) throws IOException { + return loadRegion(r, f, false); + } + + /** + * Load region with rows from 'aaa' to 'zzz'. + * @param r Region + * @param f Family + * @param flush flush the cache if true + * @return Count of rows loaded. + * @throws IOException + */ + public int loadRegion(final HRegion r, final byte[] f, final boolean flush) + throws IOException { byte[] k = new byte[3]; int rowCount = 0; for (byte b1 = 'a'; b1 <= 'z'; b1++) { @@ -989,6 +1169,9 @@ public int loadRegion(final HRegion r, final byte[] f) rowCount++; } } + if (flush) { + r.flushcache(); + } } return rowCount; } @@ -1007,6 +1190,20 @@ public int countRows(final HTable table) throws IOException { return count; } + public int countRows(final HTable table, final byte[]... families) throws IOException { + Scan scan = new Scan(); + for (byte[] family: families) { + scan.addFamily(family); + } + ResultScanner results = table.getScanner(scan); + int count = 0; + for (@SuppressWarnings("unused") Result res : results) { + count++; + } + results.close(); + return count; + } + /** * Return an md5 digest of the entire contents of a table. */ @@ -1031,7 +1228,7 @@ public String checksumRows(final HTable table) throws Exception { */ public int createMultiRegions(HTable table, byte[] columnFamily) throws IOException { - return createMultiRegions(getConfiguration(), table, columnFamily); + return createMultiRegions(table, columnFamily, true); } public static final byte[][] KEYS = { @@ -1046,18 +1243,31 @@ public int createMultiRegions(HTable table, byte[] columnFamily) Bytes.toBytes("xxx"), Bytes.toBytes("yyy") }; + public static final byte[][] KEYS_FOR_HBA_CREATE_TABLE = { + Bytes.toBytes("bbb"), + Bytes.toBytes("ccc"), Bytes.toBytes("ddd"), Bytes.toBytes("eee"), + Bytes.toBytes("fff"), Bytes.toBytes("ggg"), Bytes.toBytes("hhh"), + Bytes.toBytes("iii"), Bytes.toBytes("jjj"), Bytes.toBytes("kkk"), + Bytes.toBytes("lll"), Bytes.toBytes("mmm"), Bytes.toBytes("nnn"), + Bytes.toBytes("ooo"), Bytes.toBytes("ppp"), Bytes.toBytes("qqq"), + Bytes.toBytes("rrr"), Bytes.toBytes("sss"), Bytes.toBytes("ttt"), + Bytes.toBytes("uuu"), Bytes.toBytes("vvv"), Bytes.toBytes("www"), + Bytes.toBytes("xxx"), Bytes.toBytes("yyy"), Bytes.toBytes("zzz") + }; + + /** * Creates many regions names "aaa" to "zzz". - * @param c Configuration to use. + * * @param table The table to use for the data. * @param columnFamily The family to insert the data into. + * @param cleanupFS True if a previous region should be remove from the FS * @return count of regions created. * @throws IOException When creating the regions fails. */ - public int createMultiRegions(final Configuration c, final HTable table, - final byte[] columnFamily) + public int createMultiRegions(HTable table, byte[] columnFamily, boolean cleanupFS) throws IOException { - return createMultiRegions(c, table, columnFamily, KEYS); + return createMultiRegions(getConfiguration(), table, columnFamily, KEYS, cleanupFS); } /** @@ -1085,7 +1295,12 @@ public int createMultiRegions(final Configuration c, final HTable table, } public int createMultiRegions(final Configuration c, final HTable table, - final byte[] columnFamily, byte [][] startKeys) + final byte[] columnFamily, byte [][] startKeys) throws IOException { + return createMultiRegions(c, table, columnFamily, startKeys, true); + } + + public int createMultiRegions(final Configuration c, final HTable table, + final byte[] columnFamily, byte [][] startKeys, boolean cleanupFS) throws IOException { Arrays.sort(startKeys, Bytes.BYTES_COMPARATOR); HTable meta = new HTable(c, HConstants.META_TABLE_NAME); @@ -1099,6 +1314,9 @@ public int createMultiRegions(final Configuration c, final HTable table, // and end key. Adding the custom regions below adds those blindly, // including the new start region from empty to "bbb". lg List rows = getMetaTableRows(htd.getName()); + String regionToDeleteInFS = table + .getRegionsInRange(Bytes.toBytes(""), Bytes.toBytes("")).get(0) + .getRegionInfo().getEncodedName(); List newRegions = new ArrayList(startKeys.length); // add custom ones int count = 0; @@ -1120,13 +1338,22 @@ public int createMultiRegions(final Configuration c, final HTable table, Bytes.toStringBinary(row)); meta.delete(new Delete(row)); } + if (cleanupFS) { + // see HBASE-7417 - this confused TestReplication + // remove the "old" region from FS + Path tableDir = new Path(getDefaultRootDirPath().toString() + + System.getProperty("file.separator") + htd.getNameAsString() + + System.getProperty("file.separator") + regionToDeleteInFS); + FileSystem.get(c).delete(tableDir); + } // flush cache of regions HConnection conn = table.getConnection(); conn.clearRegionCache(); // assign all the new regions IF table is enabled. - if (getHBaseAdmin().isTableEnabled(table.getTableName())) { + HBaseAdmin admin = getHBaseAdmin(); + if (admin.isTableEnabled(table.getTableName())) { for(HRegionInfo hri : newRegions) { - hbaseCluster.getMaster().assignRegion(hri); + admin.assign(hri.getRegionName()); } } @@ -1199,11 +1426,16 @@ public List getMetaTableRows(byte[] tableName) throws IOException { List rows = new ArrayList(); ResultScanner s = t.getScanner(new Scan()); for (Result result : s) { - HRegionInfo info = Writables.getHRegionInfo( - result.getValue(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER)); + byte[] val = result.getValue(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER); + if (val == null) { + LOG.error("No region info for row " + Bytes.toString(result.getRow())); + // TODO figure out what to do for this new hosed case. + continue; + } + HRegionInfo info = Writables.getHRegionInfo(val); if (Bytes.compareTo(info.getTableName(), tableName) == 0) { LOG.info("getMetaTableRows: row -> " + - Bytes.toStringBinary(result.getRow())); + Bytes.toStringBinary(result.getRow()) + info); rows.add(result.getRow()); } } @@ -1232,8 +1464,8 @@ public HRegionServer getRSForFirstRegionInTable(byte[] tableName) Bytes.toString(tableName)); byte [] firstrow = metaRows.get(0); LOG.debug("FirstRow=" + Bytes.toString(firstrow)); - int index = hbaseCluster.getServerWith(firstrow); - return hbaseCluster.getRegionServerThreads().get(index).getRegionServer(); + int index = getMiniHBaseCluster().getServerWith(firstrow); + return getMiniHBaseCluster().getRegionServerThreads().get(index).getRegionServer(); } /** @@ -1254,15 +1486,52 @@ public void startMiniMapReduceCluster() throws IOException { */ public void startMiniMapReduceCluster(final int servers) throws IOException { LOG.info("Starting mini mapreduce cluster..."); + if (dataTestDir == null) { + setupDataTestDir(); + } // These are needed for the new and improved Map/Reduce framework - conf.set("mapred.output.dir", conf.get("hadoop.tmp.dir")); - mrCluster = new MiniMRCluster(servers, - FileSystem.get(conf).getUri().toString(), 1); + Configuration c = getConfiguration(); + String logDir = c.get("hadoop.log.dir"); + String tmpDir = c.get("hadoop.tmp.dir"); + if (logDir == null) { + logDir = tmpDir; + } + System.setProperty("hadoop.log.dir", logDir); + c.set("mapred.output.dir", tmpDir); + + // Tests were failing because this process used 6GB of virtual memory and was getting killed. + // we up the VM usable so that processes don't get killed. + conf.setFloat("yarn.nodemanager.vmem-pmem-ratio", 8.0f); + + mrCluster = new MiniMRCluster(0, 0, servers, + FileSystem.get(conf).getUri().toString(), 1, null, null, null, new JobConf(conf)); + + JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster); + if (jobConf == null) { + jobConf = mrCluster.createJobConf(); + } + jobConf.set("mapred.local.dir", + conf.get("mapred.local.dir")); //Hadoop MiniMR overwrites this while it should not + LOG.info("Mini mapreduce cluster started"); - conf.set("mapred.job.tracker", - mrCluster.createJobConf().get("mapred.job.tracker")); + JobConf mrClusterJobConf = mrCluster.createJobConf(); + c.set("mapred.job.tracker", mrClusterJobConf.get("mapred.job.tracker")); /* this for mrv2 support */ conf.set("mapreduce.framework.name", "yarn"); + conf.setBoolean("yarn.is.minicluster", true); + String rmAdress = mrClusterJobConf.get("yarn.resourcemanager.address"); + if (rmAdress != null) { + conf.set("yarn.resourcemanager.address", rmAdress); + } + String historyAddress = jobConf.get("mapreduce.jobhistory.address"); + if (historyAddress != null) { + conf.set("mapreduce.jobhistory.address", historyAddress); + } + String schedulerAdress = + mrClusterJobConf.get("yarn.resourcemanager.scheduler.address"); + if (schedulerAdress != null) { + conf.set("yarn.resourcemanager.scheduler.address", schedulerAdress); + } } /** @@ -1298,8 +1567,8 @@ public void enableDebug(Class clazz) { * @throws Exception */ public void expireMasterSession() throws Exception { - HMaster master = hbaseCluster.getMaster(); - expireSession(master.getZooKeeper(), master); + HMaster master = getMiniHBaseCluster().getMaster(); + expireSession(master.getZooKeeper(), false); } /** @@ -1308,17 +1577,44 @@ public void expireMasterSession() throws Exception { * @throws Exception */ public void expireRegionServerSession(int index) throws Exception { - HRegionServer rs = hbaseCluster.getRegionServer(index); - expireSession(rs.getZooKeeper(), rs); + HRegionServer rs = getMiniHBaseCluster().getRegionServer(index); + expireSession(rs.getZooKeeper(), false); + decrementMinRegionServerCount(); } - public void expireSession(ZooKeeperWatcher nodeZK, Server server) - throws Exception { - expireSession(nodeZK, server, false); + private void decrementMinRegionServerCount() { + // decrement the count for this.conf, for newly spwaned master + // this.hbaseCluster shares this configuration too + decrementMinRegionServerCount(getConfiguration()); + + // each master thread keeps a copy of configuration + for (MasterThread master : getHBaseCluster().getMasterThreads()) { + decrementMinRegionServerCount(master.getMaster().getConfiguration()); + } + } + + private void decrementMinRegionServerCount(Configuration conf) { + int currentCount = conf.getInt( + ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, -1); + if (currentCount != -1) { + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, + Math.max(currentCount - 1, 1)); + } } - public void expireSession(ZooKeeperWatcher nodeZK, Server server, - boolean checkStatus) throws Exception { + /** + * Expire a ZooKeeper session as recommended in ZooKeeper documentation + * http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A4 + * There are issues when doing this: + * [1] http://www.mail-archive.com/dev@zookeeper.apache.org/msg01942.html + * [2] https://issues.apache.org/jira/browse/ZOOKEEPER-1105 + * + * @param nodeZK - the ZK to make expiry + * @param checkStatus - true to check if the we can create a HTable with the + * current configuration. + */ + public void expireSession(ZooKeeperWatcher nodeZK, boolean checkStatus) + throws Exception { Configuration c = new Configuration(this.conf); String quorumServers = ZKConfig.getZKQuorumServersString(c); int sessionTimeout = 500; @@ -1326,27 +1622,56 @@ public void expireSession(ZooKeeperWatcher nodeZK, Server server, byte[] password = zk.getSessionPasswd(); long sessionID = zk.getSessionId(); + // Expiry seems to be asynchronous (see comment from P. Hunt in [1]), + // so we create a first watcher to be sure that the + // event was sent. We expect that if our watcher receives the event + // other watchers on the same machine will get is as well. + // When we ask to close the connection, ZK does not close it before + // we receive all the events, so don't have to capture the event, just + // closing the connection should be enough. + ZooKeeper monitor = new ZooKeeper(quorumServers, + 1000, new org.apache.zookeeper.Watcher(){ + @Override + public void process(WatchedEvent watchedEvent) { + LOG.info("Monitor ZKW received event="+watchedEvent); + } + } , sessionID, password); + + // Making it expire ZooKeeper newZK = new ZooKeeper(quorumServers, sessionTimeout, EmptyWatcher.instance, sessionID, password); newZK.close(); - final long sleep = 7000; // 7s seems enough to manage the timeout - LOG.info("ZK Closed Session 0x" + Long.toHexString(sessionID) + - "; sleeping=" + sleep); + LOG.info("ZK Closed Session 0x" + Long.toHexString(sessionID)); - Thread.sleep(sleep); + // Now closing & waiting to be sure that the clients get it. + monitor.close(); if (checkStatus) { new HTable(new Configuration(conf), HConstants.META_TABLE_NAME).close(); } } - /** - * Get the HBase cluster. + * Get the Mini HBase cluster. * * @return hbase cluster + * @see #getHBaseClusterInterface() */ public MiniHBaseCluster getHBaseCluster() { + return getMiniHBaseCluster(); + } + + /** + * Returns the HBaseCluster instance. + *

Returned object can be any of the subclasses of HBaseCluster, and the + * tests referring this should not assume that the cluster is a mini cluster or a + * distributed one. If the test only works on a mini cluster, then specific + * method {@link #getMiniHBaseCluster()} can be used instead w/o the + * need to type-cast. + */ + public HBaseCluster getHBaseClusterInterface() { + //implementation note: we should rename this method as #getHBaseCluster(), + //but this would require refactoring 90+ calls. return hbaseCluster; } @@ -1433,7 +1758,7 @@ public void setDFSCluster(MiniDFSCluster cluster) throws IOException { } public FileSystem getTestFileSystem() throws IOException { - return FileSystem.get(conf); + return HFileSystem.get(conf); } /** @@ -1479,8 +1804,21 @@ public void waitTableAvailable(byte[] table, long timeoutMillis) throws InterruptedException, IOException { long startWait = System.currentTimeMillis(); while (!getHBaseAdmin().isTableAvailable(table)) { - assertTrue("Timed out waiting for table " + Bytes.toStringBinary(table), - System.currentTimeMillis() - startWait < timeoutMillis); + assertTrue("Timed out waiting for table to become available " + + Bytes.toStringBinary(table), + System.currentTimeMillis() - startWait < timeoutMillis); + Thread.sleep(200); + } + } + + public void waitTableEnabled(byte[] table, long timeoutMillis) + throws InterruptedException, IOException { + long startWait = System.currentTimeMillis(); + while (!getHBaseAdmin().isTableAvailable(table) && + !getHBaseAdmin().isTableEnabled(table)) { + assertTrue("Timed out waiting for table to become available and enabled " + + Bytes.toStringBinary(table), + System.currentTimeMillis() - startWait < timeoutMillis); Thread.sleep(200); } } @@ -1489,14 +1827,14 @@ public void waitTableAvailable(byte[] table, long timeoutMillis) * Make sure that at least the specified number of region servers * are running * @param num minimum number of region servers that should be running - * @return True if we started some servers + * @return true if we started some servers * @throws IOException */ public boolean ensureSomeRegionServersAvailable(final int num) throws IOException { boolean startedServer = false; - - for (int i=hbaseCluster.getLiveRegionServerThreads().size(); i dfsClazz = dfsField.getType(); + final DFSClient dfs = DFSClient.class.cast(dfsField.get(fs)); + + // expose the method for creating direct RPC connections. + final Method createRPCNamenode = dfsClazz.getDeclaredMethod("createRPCNamenode", InetSocketAddress.class, Configuration.class, UserGroupInformation.class); + createRPCNamenode.setAccessible(true); + + // grab the DFSClient instance's backing connection information + final Field nnField = dfsClazz.getDeclaredField("nnAddress"); + nnField.setAccessible(true); + final InetSocketAddress nnAddress = InetSocketAddress.class.cast(nnField.get(dfs)); + final Field confField = dfsClazz.getDeclaredField("conf"); + confField.setAccessible(true); + final Configuration conf = Configuration.class.cast(confField.get(dfs)); + final Field ugiField = dfsClazz.getDeclaredField("ugi"); + ugiField.setAccessible(true); + final UserGroupInformation ugi = UserGroupInformation.class.cast(ugiField.get(dfs)); + + // replace the proxy for the namenode rpc with a direct instance + final Field namenodeField = dfsClazz.getDeclaredField("namenode"); + namenodeField.setAccessible(true); + namenodeField.set(dfs, createRPCNamenode.invoke(null, nnAddress, conf, ugi)); + LOG.debug("Set DSFClient namenode to bare RPC"); + } catch (Exception exception) { + LOG.info("Could not alter DFSClient to be non-retrying.", exception); + } + } /** - * Wait until countOfRegion in .META. have a non-empty - * info:server. This means all regions have been deployed, master has been - * informed and updated .META. with the regions deployed server. - * @param countOfRegions How many regions in .META. + * Wait until all regions for a table in .META. have a non-empty + * info:server, up to 60 seconds. This means all regions have been deployed, + * master has been informed and updated .META. with the regions deployed + * server. + * @param tableName the table name * @throws IOException */ - public void waitUntilAllRegionsAssigned(final int countOfRegions) - throws IOException { + public void waitUntilAllRegionsAssigned(final byte[] tableName) throws IOException { + waitUntilAllRegionsAssigned(tableName, 60000); + } + + /** + * Wait until all regions for a table in .META. have a non-empty + * info:server, or until timeout. This means all regions have been + * deployed, master has been informed and updated .META. with the regions + * deployed server. + * @param tableName the table name + * @param timeout timeout, in milliseconds + * @throws IOException + */ + public void waitUntilAllRegionsAssigned(final byte[] tableName, final long timeout) + throws IOException { + long deadline = System.currentTimeMillis() + timeout; HTable meta = new HTable(getConfiguration(), HConstants.META_TABLE_NAME); - while (true) { - int rows = 0; - Scan scan = new Scan(); - scan.addColumn(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER); - ResultScanner s = meta.getScanner(scan); - for (Result r = null; (r = s.next()) != null;) { - byte [] b = - r.getValue(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER); - if (b == null || b.length <= 0) { - break; + try { + while (true) { + boolean allRegionsAssigned = true; + Scan scan = new Scan(); + scan.addFamily(HConstants.CATALOG_FAMILY); + ResultScanner s = meta.getScanner(scan); + try { + Result r; + while ((r = s.next()) != null) { + byte [] b = r.getValue(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER); + HRegionInfo info = Writables.getHRegionInfoOrNull(b); + if (info != null && Bytes.equals(info.getTableName(), tableName)) { + b = r.getValue(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER); + allRegionsAssigned &= (b != null); + } + } + } finally { + s.close(); + } + if (allRegionsAssigned) { + return; + } + long now = System.currentTimeMillis(); + if (now > deadline) { + throw new IOException("Timeout waiting for all regions of " + + Bytes.toStringBinary(tableName) + " to be assigned"); + } + try { + Thread.sleep(deadline - now < 200 ? deadline - now : 200); + } catch (InterruptedException e) { + throw new IOException(e); } - rows++; - } - s.close(); - // If I get to here and all rows have a Server, then all have been assigned. - if (rows == countOfRegions) { - break; } - LOG.info("Found=" + rows); - Threads.sleep(200); + } finally { + meta.close(); } } @@ -1768,7 +2207,9 @@ public HTable createRandomTable(String tableName, Bytes.toBytes(String.format(keyFormat, splitStartKey)), Bytes.toBytes(String.format(keyFormat, splitEndKey)), numRegions); - hbaseCluster.flushcache(HConstants.META_TABLE_NAME); + if (hbaseCluster != null) { + getMiniHBaseCluster().flushcache(HConstants.META_TABLE_NAME); + } for (int iFlush = 0; iFlush < numFlushes; ++iFlush) { for (int iRow = 0; iRow < numRowsPerFlush; ++iRow) { @@ -1803,7 +2244,9 @@ public HTable createRandomTable(String tableName, } LOG.info("Initiating flush #" + iFlush + " for table " + tableName); table.flushCommits(); - hbaseCluster.flushcache(tableNameBytes); + if (hbaseCluster != null) { + getMiniHBaseCluster().flushcache(tableNameBytes); + } } return table; @@ -1873,12 +2316,23 @@ public static int createPreSplitLoadTestTable(Configuration conf, HColumnDescriptor hcd = new HColumnDescriptor(columnFamily); hcd.setDataBlockEncoding(dataBlockEncoding); hcd.setCompressionType(compression); - desc.addFamily(hcd); + return createPreSplitLoadTestTable(conf, desc, hcd); + } + + /** + * Creates a pre-split table for load testing. If the table already exists, + * logs a warning and continues. + * @return the number of regions the table was split into + */ + public static int createPreSplitLoadTestTable(Configuration conf, + HTableDescriptor desc, HColumnDescriptor hcd) throws IOException { + if (!desc.hasFamily(hcd.getName())) { + desc.addFamily(hcd); + } int totalNumberOfRegions = 0; + HBaseAdmin admin = new HBaseAdmin(conf); try { - HBaseAdmin admin = new HBaseAdmin(conf); - // create a table a pre-splits regions. // The number of splits is set as: // region servers * regions per region server). @@ -1901,8 +2355,10 @@ public static int createPreSplitLoadTestTable(Configuration conf, LOG.error("Master not running", e); throw new IOException(e); } catch (TableExistsException e) { - LOG.warn("Table " + Bytes.toStringBinary(tableName) + + LOG.warn("Table " + Bytes.toStringBinary(desc.getName()) + " already exists, continuing"); + } finally { + admin.close(); } return totalNumberOfRegions; } @@ -1925,4 +2381,75 @@ public HRegion createTestRegion(String tableName, HColumnDescriptor hcd) return region; } + /** + * Create region split keys between startkey and endKey + * + * @param startKey + * @param endKey + * @param numRegions the number of regions to be created. it has to be greater than 3. + * @return + */ + public byte[][] getRegionSplitStartKeys(byte[] startKey, byte[] endKey, int numRegions){ + assertTrue(numRegions>3); + byte [][] tmpSplitKeys = Bytes.split(startKey, endKey, numRegions - 3); + byte [][] result = new byte[tmpSplitKeys.length+1][]; + for (int i=0;i generateColumnDescriptors() { + return generateColumnDescriptors(""); + } + + /** + * Create a set of column descriptors with the combination of compression, + * encoding, bloom codecs available. + * @param prefix family names prefix + * @return the list of column descriptors + */ + public static List generateColumnDescriptors(final String prefix) { + List htds = new ArrayList(); + long familyId = 0; + for (Compression.Algorithm compressionType: getSupportedCompressionAlgorithms()) { + for (DataBlockEncoding encodingType: DataBlockEncoding.values()) { + for (StoreFile.BloomType bloomType: StoreFile.BloomType.values()) { + String name = String.format("%s-cf-!@#&-%d!@#", prefix, familyId); + HColumnDescriptor htd = new HColumnDescriptor(name); + htd.setCompressionType(compressionType); + htd.setDataBlockEncoding(encodingType); + htd.setBloomFilterType(bloomType); + htds.add(htd); + familyId++; + } + } + } + return htds; + } + + /** + * Get supported compression algorithms. + * @return supported compression algorithms. + */ + public static Compression.Algorithm[] getSupportedCompressionAlgorithms() { + String[] allAlgos = HFile.getSupportedCompressionAlgorithms(); + List supportedAlgos = new ArrayList(); + for (String algoName : allAlgos) { + try { + Compression.Algorithm algo = Compression.getCompressionAlgorithmByName(algoName); + algo.getCompressor(); + supportedAlgos.add(algo); + } catch (Throwable t) { + // this algo is not available + } + } + return supportedAlgos.toArray(new Compression.Algorithm[0]); + } } diff --git a/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java b/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java index a3b7ed99851b..8c492d3e451b 100644 --- a/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java +++ b/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -355,7 +354,8 @@ void doRow(int i) throws Exception { private byte [] getGaussianRandomRowBytes() { int r = (int) randomData.nextGaussian((double)totalRows / 2.0, (double)totalRows / 10.0); - return format(r); + // make sure r falls into [0,totalRows) + return format(Math.min(totalRows, Math.max(r,0))); } } diff --git a/src/test/java/org/apache/hadoop/hbase/HServerLoad092.java b/src/test/java/org/apache/hadoop/hbase/HServerLoad092.java new file mode 100644 index 000000000000..203dcd2fde38 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/HServerLoad092.java @@ -0,0 +1,693 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; +import java.util.Map; +import java.util.Set; +import java.util.TreeMap; +import java.util.TreeSet; + +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Strings; +import org.apache.hadoop.io.VersionedWritable; +import org.apache.hadoop.io.WritableComparable; + +/** + * This class is used to export current state of load on a RegionServer. + * This is the version of HServerLoad that we had in 0.92. + */ +public class HServerLoad092 extends VersionedWritable +implements WritableComparable { + private static final byte VERSION = 2; + // Empty load instance. + public static final HServerLoad092 EMPTY_HSERVERLOAD = new HServerLoad092(); + + /** Number of requests per second since last report. + */ + // TODO: Instead build this up out of region counters. + private int numberOfRequests = 0; + + /** Total Number of requests from the start of the region server. + */ + private int totalNumberOfRequests = 0; + + /** the amount of used heap, in MB */ + private int usedHeapMB = 0; + + /** the maximum allowable size of the heap, in MB */ + private int maxHeapMB = 0; + + // Regionserver-level coprocessors, e.g., WALObserver implementations. + // Region-level coprocessors, on the other hand, are stored inside RegionLoad + // objects. + private Set coprocessors = + new TreeSet(); + + /** + * HBASE-4070: Improve region server metrics to report loaded coprocessors. + * + * @return Returns the set of all coprocessors on this + * regionserver, where this set is the union of the + * regionserver-level coprocessors on one hand, and all of the region-level + * coprocessors, on the other. + * + * We must iterate through all regions loaded on this regionserver to + * obtain all of the region-level coprocessors. + */ + public String[] getCoprocessors() { + TreeSet returnValue = new TreeSet(coprocessors); + for (Map.Entry rls: getRegionsLoad().entrySet()) { + for (String coprocessor: rls.getValue().getCoprocessors()) { + returnValue.add(coprocessor); + } + } + return returnValue.toArray(new String[0]); + } + + /** per-region load metrics */ + private Map regionLoad = + new TreeMap(Bytes.BYTES_COMPARATOR); + + /** @return the object version number */ + public byte getVersion() { + return VERSION; + } + + /** + * Encapsulates per-region loading metrics. + */ + public static class RegionLoad extends VersionedWritable { + private static final byte VERSION = 1; + + /** @return the object version number */ + public byte getVersion() { + return VERSION; + } + + /** the region name */ + private byte[] name; + /** the number of stores for the region */ + private int stores; + /** the number of storefiles for the region */ + private int storefiles; + /** the total size of the store files for the region, uncompressed, in MB */ + private int storeUncompressedSizeMB; + /** the current total size of the store files for the region, in MB */ + private int storefileSizeMB; + /** the current size of the memstore for the region, in MB */ + private int memstoreSizeMB; + + /** + * The current total size of root-level store file indexes for the region, + * in MB. The same as {@link #rootIndexSizeKB} but in MB. + */ + private int storefileIndexSizeMB; + /** the current total read requests made to region */ + private int readRequestsCount; + /** the current total write requests made to region */ + private int writeRequestsCount; + /** the total compacting key values in currently running compaction */ + private long totalCompactingKVs; + /** the completed count of key values in currently running compaction */ + private long currentCompactedKVs; + + /** The current total size of root-level indexes for the region, in KB. */ + private int rootIndexSizeKB; + + /** The total size of all index blocks, not just the root level, in KB. */ + private int totalStaticIndexSizeKB; + + /** + * The total size of all Bloom filter blocks, not just loaded into the + * block cache, in KB. + */ + private int totalStaticBloomSizeKB; + + // Region-level coprocessors. + Set coprocessors = + new TreeSet(); + + /** + * Constructor, for Writable + */ + public RegionLoad() { + super(); + } + + /** + * @param name + * @param stores + * @param storefiles + * @param storeUncompressedSizeMB + * @param storefileSizeMB + * @param memstoreSizeMB + * @param storefileIndexSizeMB + * @param readRequestsCount + * @param writeRequestsCount + * @param totalCompactingKVs + * @param currentCompactedKVs + * @param coprocessors + */ + public RegionLoad(final byte[] name, final int stores, + final int storefiles, final int storeUncompressedSizeMB, + final int storefileSizeMB, + final int memstoreSizeMB, final int storefileIndexSizeMB, + final int rootIndexSizeKB, final int totalStaticIndexSizeKB, + final int totalStaticBloomSizeKB, + final int readRequestsCount, final int writeRequestsCount, + final long totalCompactingKVs, final long currentCompactedKVs, + final Set coprocessors) { + this.name = name; + this.stores = stores; + this.storefiles = storefiles; + this.storeUncompressedSizeMB = storeUncompressedSizeMB; + this.storefileSizeMB = storefileSizeMB; + this.memstoreSizeMB = memstoreSizeMB; + this.storefileIndexSizeMB = storefileIndexSizeMB; + this.rootIndexSizeKB = rootIndexSizeKB; + this.totalStaticIndexSizeKB = totalStaticIndexSizeKB; + this.totalStaticBloomSizeKB = totalStaticBloomSizeKB; + this.readRequestsCount = readRequestsCount; + this.writeRequestsCount = writeRequestsCount; + this.totalCompactingKVs = totalCompactingKVs; + this.currentCompactedKVs = currentCompactedKVs; + this.coprocessors = coprocessors; + } + + // Getters + private String[] getCoprocessors() { + return coprocessors.toArray(new String[0]); + } + + /** + * @return the region name + */ + public byte[] getName() { + return name; + } + + /** + * @return the region name as a string + */ + public String getNameAsString() { + return Bytes.toString(name); + } + + /** + * @return the number of stores + */ + public int getStores() { + return stores; + } + + /** + * @return the number of storefiles + */ + public int getStorefiles() { + return storefiles; + } + + /** + * @return the total size of the storefiles, in MB + */ + public int getStorefileSizeMB() { + return storefileSizeMB; + } + + /** + * @return the memstore size, in MB + */ + public int getMemStoreSizeMB() { + return memstoreSizeMB; + } + + /** + * @return the approximate size of storefile indexes on the heap, in MB + */ + public int getStorefileIndexSizeMB() { + return storefileIndexSizeMB; + } + + /** + * @return the number of requests made to region + */ + public long getRequestsCount() { + return readRequestsCount + writeRequestsCount; + } + + /** + * @return the number of read requests made to region + */ + public long getReadRequestsCount() { + return readRequestsCount; + } + + /** + * @return the number of read requests made to region + */ + public long getWriteRequestsCount() { + return writeRequestsCount; + } + + /** + * @return the total number of kvs in current compaction + */ + public long getTotalCompactingKVs() { + return totalCompactingKVs; + } + + /** + * @return the number of already compacted kvs in current compaction + */ + public long getCurrentCompactedKVs() { + return currentCompactedKVs; + } + + // Setters + + /** + * @param name the region name + */ + public void setName(byte[] name) { + this.name = name; + } + + /** + * @param stores the number of stores + */ + public void setStores(int stores) { + this.stores = stores; + } + + /** + * @param storefiles the number of storefiles + */ + public void setStorefiles(int storefiles) { + this.storefiles = storefiles; + } + + /** + * @param memstoreSizeMB the memstore size, in MB + */ + public void setMemStoreSizeMB(int memstoreSizeMB) { + this.memstoreSizeMB = memstoreSizeMB; + } + + /** + * @param storefileIndexSizeMB the approximate size of storefile indexes + * on the heap, in MB + */ + public void setStorefileIndexSizeMB(int storefileIndexSizeMB) { + this.storefileIndexSizeMB = storefileIndexSizeMB; + } + + /** + * @param requestsCount the number of read requests to region + */ + public void setReadRequestsCount(int requestsCount) { + this.readRequestsCount = requestsCount; + } + + /** + * @param requestsCount the number of write requests to region + */ + public void setWriteRequestsCount(int requestsCount) { + this.writeRequestsCount = requestsCount; + } + + /** + * @param totalCompactingKVs the number of kvs total in current compaction + */ + public void setTotalCompactingKVs(long totalCompactingKVs) { + this.totalCompactingKVs = totalCompactingKVs; + } + + /** + * @param currentCompactedKVs the number of kvs already compacted in + * current compaction + */ + public void setCurrentCompactedKVs(long currentCompactedKVs) { + this.currentCompactedKVs = currentCompactedKVs; + } + + // Writable + public void readFields(DataInput in) throws IOException { + super.readFields(in); + int version = in.readByte(); + if (version > VERSION) throw new IOException("Version mismatch; " + version); + int namelen = in.readInt(); + this.name = new byte[namelen]; + in.readFully(this.name); + this.stores = in.readInt(); + this.storefiles = in.readInt(); + this.storeUncompressedSizeMB = in.readInt(); + this.storefileSizeMB = in.readInt(); + this.memstoreSizeMB = in.readInt(); + this.storefileIndexSizeMB = in.readInt(); + this.readRequestsCount = in.readInt(); + this.writeRequestsCount = in.readInt(); + this.rootIndexSizeKB = in.readInt(); + this.totalStaticIndexSizeKB = in.readInt(); + this.totalStaticBloomSizeKB = in.readInt(); + this.totalCompactingKVs = in.readLong(); + this.currentCompactedKVs = in.readLong(); + int coprocessorsSize = in.readInt(); + coprocessors = new TreeSet(); + for (int i = 0; i < coprocessorsSize; i++) { + coprocessors.add(in.readUTF()); + } + } + + public void write(DataOutput out) throws IOException { + super.write(out); + out.writeByte(VERSION); + out.writeInt(name.length); + out.write(name); + out.writeInt(stores); + out.writeInt(storefiles); + out.writeInt(storeUncompressedSizeMB); + out.writeInt(storefileSizeMB); + out.writeInt(memstoreSizeMB); + out.writeInt(storefileIndexSizeMB); + out.writeInt(readRequestsCount); + out.writeInt(writeRequestsCount); + out.writeInt(rootIndexSizeKB); + out.writeInt(totalStaticIndexSizeKB); + out.writeInt(totalStaticBloomSizeKB); + out.writeLong(totalCompactingKVs); + out.writeLong(currentCompactedKVs); + out.writeInt(coprocessors.size()); + for (String coprocessor: coprocessors) { + out.writeUTF(coprocessor); + } + } + + /** + * @see java.lang.Object#toString() + */ + @Override + public String toString() { + StringBuilder sb = Strings.appendKeyValue(new StringBuilder(), "numberOfStores", + Integer.valueOf(this.stores)); + sb = Strings.appendKeyValue(sb, "numberOfStorefiles", + Integer.valueOf(this.storefiles)); + sb = Strings.appendKeyValue(sb, "storefileUncompressedSizeMB", + Integer.valueOf(this.storeUncompressedSizeMB)); + sb = Strings.appendKeyValue(sb, "storefileSizeMB", + Integer.valueOf(this.storefileSizeMB)); + if (this.storeUncompressedSizeMB != 0) { + sb = Strings.appendKeyValue(sb, "compressionRatio", + String.format("%.4f", (float)this.storefileSizeMB/ + (float)this.storeUncompressedSizeMB)); + } + sb = Strings.appendKeyValue(sb, "memstoreSizeMB", + Integer.valueOf(this.memstoreSizeMB)); + sb = Strings.appendKeyValue(sb, "storefileIndexSizeMB", + Integer.valueOf(this.storefileIndexSizeMB)); + sb = Strings.appendKeyValue(sb, "readRequestsCount", + Long.valueOf(this.readRequestsCount)); + sb = Strings.appendKeyValue(sb, "writeRequestsCount", + Long.valueOf(this.writeRequestsCount)); + sb = Strings.appendKeyValue(sb, "rootIndexSizeKB", + Integer.valueOf(this.rootIndexSizeKB)); + sb = Strings.appendKeyValue(sb, "totalStaticIndexSizeKB", + Integer.valueOf(this.totalStaticIndexSizeKB)); + sb = Strings.appendKeyValue(sb, "totalStaticBloomSizeKB", + Integer.valueOf(this.totalStaticBloomSizeKB)); + sb = Strings.appendKeyValue(sb, "totalCompactingKVs", + Long.valueOf(this.totalCompactingKVs)); + sb = Strings.appendKeyValue(sb, "currentCompactedKVs", + Long.valueOf(this.currentCompactedKVs)); + float compactionProgressPct = Float.NaN; + if( this.totalCompactingKVs > 0 ) { + compactionProgressPct = Float.valueOf( + this.currentCompactedKVs / this.totalCompactingKVs); + } + sb = Strings.appendKeyValue(sb, "compactionProgressPct", + compactionProgressPct); + String coprocessors = Arrays.toString(getCoprocessors()); + if (coprocessors != null) { + sb = Strings.appendKeyValue(sb, "coprocessors", + Arrays.toString(getCoprocessors())); + } + return sb.toString(); + } + } + + /* + * TODO: Other metrics that might be considered when the master is actually + * doing load balancing instead of merely trying to decide where to assign + * a region: + *

    + *
  • # of CPUs, heap size (to determine the "class" of machine). For + * now, we consider them to be homogeneous.
    • + *
  • #requests per region (Map<{String|HRegionInfo}, Integer>)
    • + *
  • #compactions and/or #splits (churn)
    • + *
  • server death rate (maybe there is something wrong with this server)
    • + *
+ */ + + /** default constructor (used by Writable) */ + public HServerLoad092() { + super(); + } + + /** + * Constructor + * @param numberOfRequests + * @param usedHeapMB + * @param maxHeapMB + * @param coprocessors : coprocessors loaded at the regionserver-level + */ + public HServerLoad092(final int totalNumberOfRequests, + final int numberOfRequests, final int usedHeapMB, final int maxHeapMB, + final Map regionLoad, + final Set coprocessors) { + this.numberOfRequests = numberOfRequests; + this.usedHeapMB = usedHeapMB; + this.maxHeapMB = maxHeapMB; + this.regionLoad = regionLoad; + this.totalNumberOfRequests = totalNumberOfRequests; + this.coprocessors = coprocessors; + } + + /** + * Constructor + * @param hsl the template HServerLoad + */ + public HServerLoad092(final HServerLoad092 hsl) { + this(hsl.totalNumberOfRequests, hsl.numberOfRequests, hsl.usedHeapMB, + hsl.maxHeapMB, hsl.getRegionsLoad(), hsl.coprocessors); + for (Map.Entry e : hsl.regionLoad.entrySet()) { + this.regionLoad.put(e.getKey(), e.getValue()); + } + } + + /** + * Originally, this method factored in the effect of requests going to the + * server as well. However, this does not interact very well with the current + * region rebalancing code, which only factors number of regions. For the + * interim, until we can figure out how to make rebalancing use all the info + * available, we're just going to make load purely the number of regions. + * + * @return load factor for this server + */ + public int getLoad() { + // int load = numberOfRequests == 0 ? 1 : numberOfRequests; + // load *= numberOfRegions == 0 ? 1 : numberOfRegions; + // return load; + return this.regionLoad.size(); + } + + /** + * @see java.lang.Object#toString() + */ + @Override + public String toString() { + return toString(1); + } + + /** + * Returns toString() with the number of requests divided by the message + * interval in seconds + * @param msgInterval + * @return The load as a String + */ + public String toString(int msgInterval) { + int numberOfRegions = this.regionLoad.size(); + StringBuilder sb = new StringBuilder(); + sb = Strings.appendKeyValue(sb, "requestsPerSecond", + Integer.valueOf(numberOfRequests/msgInterval)); + sb = Strings.appendKeyValue(sb, "numberOfOnlineRegions", + Integer.valueOf(numberOfRegions)); + sb = Strings.appendKeyValue(sb, "usedHeapMB", + Integer.valueOf(this.usedHeapMB)); + sb = Strings.appendKeyValue(sb, "maxHeapMB", Integer.valueOf(maxHeapMB)); + return sb.toString(); + } + + /** + * @see java.lang.Object#equals(java.lang.Object) + */ + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null) { + return false; + } + if (getClass() != o.getClass()) { + return false; + } + return compareTo((HServerLoad092)o) == 0; + } + + // Getters + + /** + * @return the numberOfRegions + */ + public int getNumberOfRegions() { + return this.regionLoad.size(); + } + + /** + * @return the numberOfRequests per second. + */ + public int getNumberOfRequests() { + return numberOfRequests; + } + + /** + * @return the numberOfRequests + */ + public int getTotalNumberOfRequests() { + return totalNumberOfRequests; + } + + /** + * @return the amount of heap in use, in MB + */ + public int getUsedHeapMB() { + return usedHeapMB; + } + + /** + * @return the maximum allowable heap size, in MB + */ + public int getMaxHeapMB() { + return maxHeapMB; + } + + /** + * @return region load metrics + */ + public Map getRegionsLoad() { + return Collections.unmodifiableMap(regionLoad); + } + + /** + * @return Count of storefiles on this regionserver + */ + public int getStorefiles() { + int count = 0; + for (RegionLoad info: regionLoad.values()) + count += info.getStorefiles(); + return count; + } + + /** + * @return Total size of store files in MB + */ + public int getStorefileSizeInMB() { + int count = 0; + for (RegionLoad info: regionLoad.values()) + count += info.getStorefileSizeMB(); + return count; + } + + /** + * @return Size of memstores in MB + */ + public int getMemStoreSizeInMB() { + int count = 0; + for (RegionLoad info: regionLoad.values()) + count += info.getMemStoreSizeMB(); + return count; + } + + /** + * @return Size of store file indexes in MB + */ + public int getStorefileIndexSizeInMB() { + int count = 0; + for (RegionLoad info: regionLoad.values()) + count += info.getStorefileIndexSizeMB(); + return count; + } + + // Writable + + public void readFields(DataInput in) throws IOException { + super.readFields(in); + int version = in.readByte(); + if (version > VERSION) throw new IOException("Version mismatch; " + version); + numberOfRequests = in.readInt(); + usedHeapMB = in.readInt(); + maxHeapMB = in.readInt(); + int numberOfRegions = in.readInt(); + for (int i = 0; i < numberOfRegions; i++) { + RegionLoad rl = new RegionLoad(); + rl.readFields(in); + regionLoad.put(rl.getName(), rl); + } + totalNumberOfRequests = in.readInt(); + int coprocessorsSize = in.readInt(); + for(int i = 0; i < coprocessorsSize; i++) { + coprocessors.add(in.readUTF()); + } + } + + public void write(DataOutput out) throws IOException { + super.write(out); + out.writeByte(VERSION); + out.writeInt(numberOfRequests); + out.writeInt(usedHeapMB); + out.writeInt(maxHeapMB); + out.writeInt(this.regionLoad.size()); + for (RegionLoad rl: regionLoad.values()) + rl.write(out); + out.writeInt(totalNumberOfRequests); + out.writeInt(coprocessors.size()); + for (String coprocessor: coprocessors) { + out.writeUTF(coprocessor); + } + } + + // Comparable + + public int compareTo(HServerLoad092 o) { + return this.getLoad() - o.getLoad(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/IngestIntegrationTestBase.java b/src/test/java/org/apache/hadoop/hbase/IngestIntegrationTestBase.java new file mode 100644 index 000000000000..9c603ae09ea9 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/IngestIntegrationTestBase.java @@ -0,0 +1,123 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import java.io.IOException; + +import junit.framework.Assert; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.LoadTestTool; + +/** + * A base class for tests that do something with the cluster while running + * {@link LoadTestTool} to write and verify some data. + */ +public abstract class IngestIntegrationTestBase { + private static String tableName = null; + + /** A soft limit on how long we should run */ + private static final String RUN_TIME_KEY = "hbase.%s.runtime"; + + protected static final Log LOG = LogFactory.getLog(IngestIntegrationTestBase.class); + protected IntegrationTestingUtility util; + protected HBaseCluster cluster; + private LoadTestTool loadTool; + + protected void setUp(int numSlavesBase) throws Exception { + tableName = this.getClass().getSimpleName(); + util = new IntegrationTestingUtility(); + LOG.info("Initializing cluster with " + numSlavesBase + " servers"); + util.initializeCluster(numSlavesBase); + LOG.info("Done initializing cluster"); + cluster = util.getHBaseClusterInterface(); + deleteTableIfNecessary(); + loadTool = new LoadTestTool(); + loadTool.setConf(util.getConfiguration()); + // Initialize load test tool before we start breaking things; + // LoadTestTool init, even when it is a no-op, is very fragile. + int ret = loadTool.run(new String[] { "-tn", tableName, "-init_only" }); + Assert.assertEquals("Failed to initialize LoadTestTool", 0, ret); + } + + protected void tearDown() throws Exception { + LOG.info("Restoring the cluster"); + util.restoreCluster(); + LOG.info("Done restoring the cluster"); + } + + private void deleteTableIfNecessary() throws IOException { + if (util.getHBaseAdmin().tableExists(tableName)) { + util.deleteTable(Bytes.toBytes(tableName)); + } + } + + protected void runIngestTest(long defaultRunTime, int keysPerServerPerIter, + int colsPerKey, int recordSize, int writeThreads) throws Exception { + LOG.info("Running ingest"); + LOG.info("Cluster size:" + util.getHBaseClusterInterface().getClusterStatus().getServersSize()); + + long start = System.currentTimeMillis(); + String runtimeKey = String.format(RUN_TIME_KEY, this.getClass().getSimpleName()); + long runtime = util.getConfiguration().getLong(runtimeKey, defaultRunTime); + long startKey = 0; + + long numKeys = getNumKeys(keysPerServerPerIter); + while (System.currentTimeMillis() - start < 0.9 * runtime) { + LOG.info("Intended run time: " + (runtime/60000) + " min, left:" + + ((runtime - (System.currentTimeMillis() - start))/60000) + " min"); + + int ret = loadTool.run(new String[] { + "-tn", tableName, + "-write", String.format("%d:%d:%d", colsPerKey, recordSize, writeThreads), + "-start_key", String.valueOf(startKey), + "-num_keys", String.valueOf(numKeys), + "-skip_init" + }); + if (0 != ret) { + String errorMsg = "Load failed with error code " + ret; + LOG.error(errorMsg); + Assert.fail(errorMsg); + } + + ret = loadTool.run(new String[] { + "-tn", tableName, + "-read", "100:20", + "-start_key", String.valueOf(startKey), + "-num_keys", String.valueOf(numKeys), + "-skip_init" + }); + if (0 != ret) { + String errorMsg = "Verification failed with error code " + ret; + LOG.error(errorMsg); + Assert.fail(errorMsg); + } + startKey += numKeys; + } + } + + /** Estimates a data size based on the cluster size */ + private long getNumKeys(int keysPerServer) + throws IOException { + int numRegionServers = cluster.getClusterStatus().getServersSize(); + return keysPerServer * numRegionServers; + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/IntegrationTestDataIngestSlowDeterministic.java b/src/test/java/org/apache/hadoop/hbase/IntegrationTestDataIngestSlowDeterministic.java new file mode 100644 index 000000000000..57415a19962b --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/IntegrationTestDataIngestSlowDeterministic.java @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import org.apache.hadoop.hbase.util.ChaosMonkey; +import org.apache.hadoop.hbase.util.ChaosMonkey.BatchRestartRs; +import org.apache.hadoop.hbase.util.ChaosMonkey.RestartActiveMaster; +import org.apache.hadoop.hbase.util.ChaosMonkey.RestartRandomRs; +import org.apache.hadoop.hbase.util.ChaosMonkey.RestartRsHoldingMeta; +import org.apache.hadoop.hbase.util.ChaosMonkey.RestartRsHoldingRoot; +import org.apache.hadoop.hbase.util.ChaosMonkey.RollingBatchRestartRs; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * A system test which does large data ingestion and verify using {@link LoadTestTool}. + * It performs a set of actions deterministically using ChaosMonkey, then starts killing + * things randomly. You can configure how long should the load test run by using + * "hbase.IntegrationTestDataIngestSlowDeterministic.runtime" configuration parameter. + */ +@Category(IntegrationTests.class) +public class IntegrationTestDataIngestSlowDeterministic extends IngestIntegrationTestBase { + private static final int SERVER_COUNT = 3; // number of slaves for the smallest cluster + private static final long DEFAULT_RUN_TIME = 30 * 60 * 1000; + private static final long CHAOS_EVERY_MS = 150 * 1000; // Chaos every 2.5 minutes. + + private ChaosMonkey monkey; + + @Before + public void setUp() throws Exception { + super.setUp(SERVER_COUNT); + ChaosMonkey.Action[] actions = new ChaosMonkey.Action[] { + new RestartRandomRs(60000), + new BatchRestartRs(5000, 0.5f), + new RestartActiveMaster(5000), + new RollingBatchRestartRs(5000, 1.0f), + new RestartRsHoldingMeta(35000), + new RestartRsHoldingRoot(35000) + }; + monkey = new ChaosMonkey(util, new ChaosMonkey.CompositeSequentialPolicy( + new ChaosMonkey.DoActionsOncePolicy(CHAOS_EVERY_MS, actions), + new ChaosMonkey.PeriodicRandomActionPolicy(CHAOS_EVERY_MS, actions))); + monkey.start(); + } + + @After + public void tearDown() throws Exception { + if (monkey != null) { + monkey.stop("tearDown"); + monkey.waitForStop(); + } + super.tearDown(); + } + + @Test + public void testDataIngest() throws Exception { + runIngestTest(DEFAULT_RUN_TIME, 2500, 10, 100, 5); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/IntegrationTestDataIngestWithChaosMonkey.java b/src/test/java/org/apache/hadoop/hbase/IntegrationTestDataIngestWithChaosMonkey.java new file mode 100644 index 000000000000..72891b1849c1 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/IntegrationTestDataIngestWithChaosMonkey.java @@ -0,0 +1,63 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import org.apache.hadoop.hbase.util.ChaosMonkey; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * A system test which does large data ingestion and verify using {@link LoadTestTool}, + * while killing the region servers and the master(s) randomly. You can configure how long + * should the load test run by using "hbase.IntegrationTestDataIngestWithChaosMonkey.runtime" + * configuration parameter. + */ +@Category(IntegrationTests.class) +public class IntegrationTestDataIngestWithChaosMonkey extends IngestIntegrationTestBase { + + private static final int NUM_SLAVES_BASE = 4; //number of slaves for the smallest cluster + + // run for 5 min by default + private static final long DEFAULT_RUN_TIME = 5 * 60 * 1000; + + private ChaosMonkey monkey; + + @Before + public void setUp() throws Exception { + super.setUp(NUM_SLAVES_BASE); + monkey = new ChaosMonkey(util, ChaosMonkey.EVERY_MINUTE_RANDOM_ACTION_POLICY); + monkey.start(); + } + + @After + public void tearDown() throws Exception { + if (monkey != null) { + monkey.stop("test has finished, that's why"); + monkey.waitForStop(); + } + super.tearDown(); + } + + @Test + public void testDataIngest() throws Exception { + runIngestTest(DEFAULT_RUN_TIME, 2500, 10, 100, 20); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/IntegrationTestingUtility.java b/src/test/java/org/apache/hadoop/hbase/IntegrationTestingUtility.java new file mode 100644 index 000000000000..45cbc01e7097 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/IntegrationTestingUtility.java @@ -0,0 +1,135 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.util.ReflectionUtils; + +import java.io.IOException; + +/** + * Facility for integration/system tests. This extends {@link HBaseTestingUtility} + * and adds-in the functionality needed by integration and system tests. This class understands + * distributed and pseudo-distributed/local cluster deployments, and abstracts those from the tests + * in this module. + *

+ * IntegrationTestingUtility is constructed and used by the integration tests, but the tests + * themselves should not assume a particular deployment. They can rely on the methods in this + * class and HBaseCluster. Before the testing begins, the test should initialize the cluster by + * calling {@link #initializeCluster(int)}. + *

+ * The cluster that is used defaults to a mini cluster, but it can be forced to use a distributed + * cluster by calling {@link #setUseDistributedCluster(Configuration)}. This method is invoked by + * test drivers (maven, IntegrationTestsDriver, etc) before initializing the cluster + * via {@link #initializeCluster(int)}. Individual tests should not directly call + * {@link #setUseDistributedCluster(Configuration)}. + */ +public class IntegrationTestingUtility extends HBaseTestingUtility { + + public IntegrationTestingUtility() { + this(HBaseConfiguration.create()); + } + + public IntegrationTestingUtility(Configuration conf) { + super(conf); + } + + /** + * Configuration that controls whether this utility assumes a running/deployed cluster. + * This is different than "hbase.cluster.distributed" since that parameter indicates whether the + * cluster is in an actual distributed environment, while this shows that there is a + * deployed (distributed or pseudo-distributed) cluster running, and we do not need to + * start a mini-cluster for tests. + */ + public static final String IS_DISTRIBUTED_CLUSTER = "hbase.test.cluster.distributed"; + + /** + * Initializes the state of the cluster. It starts a new in-process mini cluster, OR + * if we are given an already deployed distributed cluster it initializes the state. + * @param numSlaves Number of slaves to start up if we are booting a mini cluster. Otherwise + * we check whether this many nodes are available and throw an exception if not. + */ + public void initializeCluster(int numSlaves) throws Exception { + if (isDistributedCluster()) { + createDistributedHBaseCluster(); + checkNodeCount(numSlaves); + } else { + startMiniCluster(numSlaves); + } + } + + /** + * Checks whether we have more than numSlaves nodes. Throws an + * exception otherwise. + */ + public void checkNodeCount(int numSlaves) throws Exception { + HBaseCluster cluster = getHBaseClusterInterface(); + if (cluster.getClusterStatus().getServers().size() < numSlaves) { + throw new Exception("Cluster does not have enough nodes:" + numSlaves); + } + } + + /** + * Restores the cluster to the initial state if it is a distributed cluster, otherwise, shutdowns the + * mini cluster. + */ + public void restoreCluster() throws IOException { + if (isDistributedCluster()) { + getHBaseClusterInterface().restoreInitialStatus(); + } else { + getMiniHBaseCluster().shutdown(); + } + } + + /** + * Sets the configuration property to use a distributed cluster for the integration tests. Test drivers + * should use this to enforce cluster deployment. + */ + public static void setUseDistributedCluster(Configuration conf) { + conf.setBoolean(IS_DISTRIBUTED_CLUSTER, true); + System.setProperty(IS_DISTRIBUTED_CLUSTER, "true"); + } + + /** + * @return whether we are interacting with a distributed cluster as opposed to and in-process mini + * cluster or a local cluster. + * @see IntegrationTestingUtility#setUseDistributedCluster(Configuration) + */ + private boolean isDistributedCluster() { + Configuration conf = getConfiguration(); + boolean isDistributedCluster = false; + isDistributedCluster = Boolean.parseBoolean(System.getProperty(IS_DISTRIBUTED_CLUSTER, "false")); + if (!isDistributedCluster) { + isDistributedCluster = conf.getBoolean(IS_DISTRIBUTED_CLUSTER, false); + } + return isDistributedCluster; + } + + private void createDistributedHBaseCluster() throws IOException { + Configuration conf = getConfiguration(); + Class clusterManagerClass = conf.getClass( + HConstants.HBASE_CLUSTER_MANAGER_CLASS, HBaseClusterManager.class, + ClusterManager.class); + ClusterManager clusterManager = ReflectionUtils.newInstance( + clusterManagerClass, conf); + setHBaseCluster(new DistributedHBaseCluster(conf, clusterManager)); + getHBaseAdmin(); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/IntegrationTests.java b/src/test/java/org/apache/hadoop/hbase/IntegrationTests.java new file mode 100644 index 000000000000..d429e2405101 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/IntegrationTests.java @@ -0,0 +1,39 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +/** + * Tag a test as 'integration/system' test, meaning that the test class has the following + * characteristics:

    + *
  • Possibly takes hours to complete
    • + *
  • Can be run on a mini cluster or an actual cluster
    • + *
  • Can make changes to the given cluster (starting stopping daemons, etc)
    • + *
  • Should not be run in parallel of other integration tests
    • + *
+ * + * Integration / System tests should have a class name starting with "IntegrationTest", and + * should be annotated with @Category(IntegrationTests.class). Integration tests can be run + * using the IntegrationTestsDriver class or from mvn verify. + * + * @see SmallTests + * @see MediumTests + * @see LargeTests + */ +public interface IntegrationTests { +} diff --git a/src/test/java/org/apache/hadoop/hbase/IntegrationTestsDriver.java b/src/test/java/org/apache/hadoop/hbase/IntegrationTestsDriver.java new file mode 100644 index 000000000000..a6f3760bc44e --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/IntegrationTestsDriver.java @@ -0,0 +1,107 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import java.io.IOException; +import java.util.List; +import java.util.Set; +import java.util.regex.Pattern; + +import org.apache.commons.cli.CommandLine; +import org.apache.hadoop.hbase.util.AbstractHBaseTool; +import org.apache.hadoop.util.ToolRunner; +import org.junit.internal.TextListener; +import org.junit.runner.JUnitCore; +import org.junit.runner.Result; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; + +/** + * This class drives the Integration test suite execution. Executes all + * tests having @Category(IntegrationTests.class) annotation against an + * already deployed distributed cluster. + */ +public class IntegrationTestsDriver extends AbstractHBaseTool { + private static final String TESTS_ARG = "test"; + private static final Log LOG = LogFactory.getLog(IntegrationTestsDriver.class); + private IntegrationTestFilter intTestFilter = new IntegrationTestFilter(); + + public static void main(String[] args) throws Exception { + int ret = ToolRunner.run(new IntegrationTestsDriver(), args); + System.exit(ret); + } + + private class IntegrationTestFilter extends ClassTestFinder.TestClassFilter { + private Pattern testFilterRe = Pattern.compile(".*"); + public IntegrationTestFilter() { + super(IntegrationTests.class); + } + + public void setPattern(String pattern) { + testFilterRe = Pattern.compile(pattern); + } + + @Override + public boolean isCandidateClass(Class c) { + return super.isCandidateClass(c) && testFilterRe.matcher(c.getName()).find(); + } + } + + @Override + protected void addOptions() { + addOptWithArg(TESTS_ARG, "a Java regular expression to filter tests on"); + } + + @Override + protected void processOptions(CommandLine cmd) { + String testFilterString = cmd.getOptionValue(TESTS_ARG, null); + if (testFilterString != null) { + intTestFilter.setPattern(testFilterString); + } + } + + /** + * Returns test classes annotated with @Category(IntegrationTests.class), + * according to the filter specific on the command line (if any). + */ + private Class[] findIntegrationTestClasses() + throws ClassNotFoundException, LinkageError, IOException { + ClassTestFinder.TestFileNameFilter nameFilter = new ClassTestFinder.TestFileNameFilter(); + ClassFinder classFinder = new ClassFinder(nameFilter, intTestFilter); + Set> classes = classFinder.findClasses(true); + return classes.toArray(new Class[classes.size()]); + } + + + @Override + protected int doWork() throws Exception { + //this is called from the command line, so we should set to use the distributed cluster + IntegrationTestingUtility.setUseDistributedCluster(conf); + Class[] classes = findIntegrationTestClasses(); + LOG.info("Found " + classes.length + " integration tests to run"); + + JUnitCore junit = new JUnitCore(); + junit.addListener(new TextListener(System.out)); + Result result = junit.run(classes); + + return result.wasSuccessful() ? 0 : 1; + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/KeyValueTestUtil.java b/src/test/java/org/apache/hadoop/hbase/KeyValueTestUtil.java index 36d768a9fa16..af9032bf96f9 100644 --- a/src/test/java/org/apache/hadoop/hbase/KeyValueTestUtil.java +++ b/src/test/java/org/apache/hadoop/hbase/KeyValueTestUtil.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/LargeTests.java b/src/test/java/org/apache/hadoop/hbase/LargeTests.java index f1b46fa92ea7..958ffd71c7ef 100644 --- a/src/test/java/org/apache/hadoop/hbase/LargeTests.java +++ b/src/test/java/org/apache/hadoop/hbase/LargeTests.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -33,6 +32,7 @@ * * @see SmallTests * @see MediumTests + * @see IntegrationTests */ public interface LargeTests { } diff --git a/src/test/java/org/apache/hadoop/hbase/MapFilePerformanceEvaluation.java b/src/test/java/org/apache/hadoop/hbase/MapFilePerformanceEvaluation.java index 76359498b7fc..ec47318a2d0c 100644 --- a/src/test/java/org/apache/hadoop/hbase/MapFilePerformanceEvaluation.java +++ b/src/test/java/org/apache/hadoop/hbase/MapFilePerformanceEvaluation.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/MediumTests.java b/src/test/java/org/apache/hadoop/hbase/MediumTests.java index bbbde7cdcca0..a51a2c9be6e7 100644 --- a/src/test/java/org/apache/hadoop/hbase/MediumTests.java +++ b/src/test/java/org/apache/hadoop/hbase/MediumTests.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -32,6 +31,7 @@ * * @see SmallTests * @see LargeTests + * @see IntegrationTests */ public interface MediumTests { } diff --git a/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java b/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java index b36d12ca283f..5be589d6ed3a 100644 --- a/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java +++ b/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java @@ -1,5 +1,4 @@ /** - * Copyright 2008 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -28,13 +27,18 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.hbase.ipc.HRegionInterface; +import org.apache.hadoop.hbase.ipc.HMasterInterface; import org.apache.hadoop.hbase.client.HConnectionManager; import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.master.ServerManager; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.HRegionServer; import org.apache.hadoop.hbase.security.User; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.JVMClusterUtil; +import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread; +import org.apache.hadoop.hbase.util.JVMClusterUtil.RegionServerThread; import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.io.MapWritable; @@ -44,9 +48,8 @@ * if we are running on DistributedFilesystem, create a FileSystem instance * each and will close down their instance on the way out. */ -public class MiniHBaseCluster { +public class MiniHBaseCluster extends HBaseCluster { static final Log LOG = LogFactory.getLog(MiniHBaseCluster.class.getName()); - private Configuration conf; public LocalHBaseCluster hbaseCluster; private static int index; @@ -69,11 +72,19 @@ public MiniHBaseCluster(Configuration conf, int numRegionServers) * @throws IOException */ public MiniHBaseCluster(Configuration conf, int numMasters, - int numRegionServers) - throws IOException, InterruptedException { - this.conf = conf; + int numRegionServers) + throws IOException, InterruptedException { + this(conf, numMasters, numRegionServers, null, null); + } + + public MiniHBaseCluster(Configuration conf, int numMasters, int numRegionServers, + Class masterClass, + Class regionserverClass) + throws IOException, InterruptedException { + super(conf); conf.set(HConstants.MASTER_PORT, "0"); - init(numMasters, numRegionServers); + init(numMasters, numRegionServers, masterClass, regionserverClass); + this.initialClusterStatus = getClusterStatus(); } public Configuration getConfiguration() { @@ -178,12 +189,21 @@ public void run() { } } - private void init(final int nMasterNodes, final int nRegionNodes) + private void init(final int nMasterNodes, final int nRegionNodes, + Class masterClass, + Class regionserverClass) throws IOException, InterruptedException { try { + if (masterClass == null){ + masterClass = HMaster.class; + } + if (regionserverClass == null){ + regionserverClass = MiniHBaseCluster.MiniHBaseClusterRegionServer.class; + } + // start up a LocalHBaseCluster hbaseCluster = new LocalHBaseCluster(conf, nMasterNodes, 0, - HMaster.class, MiniHBaseCluster.MiniHBaseClusterRegionServer.class); + masterClass, regionserverClass); // manually add the regionservers as other users for (int i=0; i mts; - while (!(mts = getMasterThreads()).isEmpty()) { + long start = System.currentTimeMillis(); + while (!(mts = getMasterThreads()).isEmpty() + && (System.currentTimeMillis() - start) < timeout) { for (JVMClusterUtil.MasterThread mt : mts) { - if (mt.getMaster().isActiveMaster() && mt.getMaster().isInitialized()) { + ServerManager serverManager = mt.getMaster().getServerManager(); + if (mt.getMaster().isActiveMaster() && mt.getMaster().isInitialized() + && !serverManager.areDeadServersInProgress()) { return true; } } - Thread.sleep(100); + + Threads.sleep(100); } return false; } @@ -415,7 +493,17 @@ public void shutdown() throws IOException { if (this.hbaseCluster != null) { this.hbaseCluster.shutdown(); } - HConnectionManager.deleteAllConnections(false); + HConnectionManager.deleteAllConnections(); + } + + @Override + public void close() throws IOException { + } + + @Override + public ClusterStatus getClusterStatus() throws IOException { + HMaster master = getMaster(); + return master == null ? null : master.getClusterStatus(); } /** @@ -446,6 +534,34 @@ public void flushcache(byte [] tableName) throws IOException { } } + /** + * Call flushCache on all regions on all participating regionservers. + * @throws IOException + */ + public void compact(boolean major) throws IOException { + for (JVMClusterUtil.RegionServerThread t: + this.hbaseCluster.getRegionServers()) { + for(HRegion r: t.getRegionServer().getOnlineRegionsLocalContext()) { + r.compactStores(major); + } + } + } + + /** + * Call flushCache on all regions of the specified table. + * @throws IOException + */ + public void compact(byte [] tableName, boolean major) throws IOException { + for (JVMClusterUtil.RegionServerThread t: + this.hbaseCluster.getRegionServers()) { + for(HRegion r: t.getRegionServer().getOnlineRegionsLocalContext()) { + if(Bytes.equals(r.getTableDesc().getName(), tableName)) { + r.compactStores(major); + } + } + } + } + /** * @return List of region server threads. */ @@ -512,6 +628,15 @@ public int getServerWith(byte[] regionName) { return index; } + @Override + public ServerName getServerHoldingRegion(byte[] regionName) throws IOException { + int index = getServerWith(regionName); + if (index < 0) { + return null; + } + return getRegionServer(index).getServerName(); + } + /** * Counts the total numbers of regions being served by the currently online * region servers by asking each how many regions they have. Does not look @@ -525,4 +650,30 @@ public long countServedRegions() { } return count; } + + @Override + public void waitUntilShutDown() { + this.hbaseCluster.join(); + } + + protected int getRegionServerIndex(ServerName serverName) { + //we have a small number of region servers, this should be fine for now. + List servers = getRegionServerThreads(); + for (int i=0; i < servers.size(); i++) { + if (servers.get(i).getRegionServer().getServerName().equals(serverName)) { + return i; + } + } + return -1; + } + + protected int getMasterIndex(ServerName serverName) { + List masters = getMasterThreads(); + for (int i = 0; i < masters.size(); i++) { + if (masters.get(i).getMaster().getServerName().equals(serverName)) { + return i; + } + } + return -1; + } } diff --git a/src/test/java/org/apache/hadoop/hbase/MultithreadedTestUtil.java b/src/test/java/org/apache/hadoop/hbase/MultithreadedTestUtil.java index f9a00dded996..8758f9eb7098 100644 --- a/src/test/java/org/apache/hadoop/hbase/MultithreadedTestUtil.java +++ b/src/test/java/org/apache/hadoop/hbase/MultithreadedTestUtil.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java b/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java index 16c7653506c6..6d4a65c0d24a 100644 --- a/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java +++ b/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -21,9 +20,12 @@ import java.io.DataInput; import java.io.DataOutput; +import java.io.File; import java.io.IOException; import java.io.PrintStream; -import java.io.File; +import java.math.BigDecimal; +import java.math.MathContext; +import java.text.DecimalFormat; import java.text.SimpleDateFormat; import java.util.ArrayList; import java.util.Date; @@ -39,6 +41,7 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; @@ -50,12 +53,15 @@ import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.filter.FilterAllFilter; +import org.apache.hadoop.hbase.filter.FilterList; import org.apache.hadoop.hbase.filter.PageFilter; import org.apache.hadoop.hbase.filter.WhileMatchFilter; -import org.apache.hadoop.hbase.filter.Filter; import org.apache.hadoop.hbase.filter.SingleColumnValueFilter; import org.apache.hadoop.hbase.filter.CompareFilter; import org.apache.hadoop.hbase.filter.BinaryComparator; +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; +import org.apache.hadoop.hbase.io.hfile.Compression; import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.FSUtils; @@ -78,6 +84,8 @@ import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.mapreduce.lib.reduce.LongSumReducer; import org.apache.hadoop.util.LineReader; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; /** * Script used evaluating HBase performance and scalability. Runs a HBase @@ -95,45 +103,53 @@ *

If number of clients > 1, we start up a MapReduce job. Each map task * runs an individual client. Each client does about 1GB of data. */ -public class PerformanceEvaluation { +public class PerformanceEvaluation extends Configured implements Tool { protected static final Log LOG = LogFactory.getLog(PerformanceEvaluation.class.getName()); - private static final int ROW_LENGTH = 1000; - private static final int ONE_GB = 1024 * 1024 * 1000; - private static final int ROWS_PER_GB = ONE_GB / ROW_LENGTH; - + public static final byte[] COMPRESSION = Bytes.toBytes("NONE"); public static final byte[] TABLE_NAME = Bytes.toBytes("TestTable"); public static final byte[] FAMILY_NAME = Bytes.toBytes("info"); public static final byte[] QUALIFIER_NAME = Bytes.toBytes("data"); + public static final int VALUE_LENGTH = 1000; + public static final int ROW_LENGTH = 26; - protected static final HTableDescriptor TABLE_DESCRIPTOR; - static { - TABLE_DESCRIPTOR = new HTableDescriptor(TABLE_NAME); - TABLE_DESCRIPTOR.addFamily(new HColumnDescriptor(FAMILY_NAME)); - } + private static final int ONE_GB = 1024 * 1024 * 1000; + private static final int ROWS_PER_GB = ONE_GB / VALUE_LENGTH; + private static final DecimalFormat FMT = new DecimalFormat("0.##"); + private static final MathContext CXT = MathContext.DECIMAL64; + private static final BigDecimal MS_PER_SEC = BigDecimal.valueOf(1000); + private static final BigDecimal BYTES_PER_MB = BigDecimal.valueOf(1024 * 1024); + protected static HTableDescriptor TABLE_DESCRIPTOR; protected Map commands = new TreeMap(); - volatile Configuration conf; private boolean miniCluster = false; private boolean nomapred = false; private int N = 1; private int R = ROWS_PER_GB; + private byte[] tableName = TABLE_NAME; + private Compression.Algorithm compression = Compression.Algorithm.NONE; + private DataBlockEncoding blockEncoding = DataBlockEncoding.NONE; + private float sampleRate = 1.0f; private boolean flushCommits = true; private boolean writeToWAL = true; + private boolean reportLatency = false; + private boolean filterAll = false; private int presplitRegions = 0; private static final Path PERF_EVAL_DIR = new Path("performance_evaluation"); - /** - * Regex to parse lines in input file passed to mapreduce task. - */ + + /** Regex to parse lines in input file passed to mapreduce task. */ public static final Pattern LINE_PATTERN = Pattern.compile("startRow=(\\d+),\\s+" + "perClientRunRows=(\\d+),\\s+" + "totalRows=(\\d+),\\s+" + + "sampleRate=([-+]?[0-9]*\\.?[0-9]+),\\s+" + "clients=(\\d+),\\s+" + "flushCommits=(\\w+),\\s+" + - "writeToWAL=(\\w+)"); + "writeToWAL=(\\w+),\\s+" + + "reportLatency=(\\w+),\\s+" + + "filterAll=(\\w+)"); /** * Enum for map metrics. Keep it out here rather than inside in the Map @@ -143,15 +159,15 @@ protected static enum Counter { /** elapsed time */ ELAPSED_TIME, /** number of rows */ - ROWS} - + ROWS + } /** * Constructor - * @param c Configuration object + * @param conf Configuration object */ - public PerformanceEvaluation(final Configuration c) { - this.conf = c; + public PerformanceEvaluation(final Configuration conf) { + super(conf); addCommandDescriptor(RandomReadTest.class, "randomRead", "Run random read test"); @@ -206,27 +222,27 @@ public static class PeInputSplit extends InputSplit implements Writable { private int startRow = 0; private int rows = 0; private int totalRows = 0; + private float sampleRate = 1.0f; private int clients = 0; private boolean flushCommits = false; private boolean writeToWAL = true; + private boolean reportLatency = false; + private boolean filterAll = false; - public PeInputSplit() { - this.startRow = 0; - this.rows = 0; - this.totalRows = 0; - this.clients = 0; - this.flushCommits = false; - this.writeToWAL = true; - } + public PeInputSplit() {} - public PeInputSplit(int startRow, int rows, int totalRows, int clients, - boolean flushCommits, boolean writeToWAL) { + public PeInputSplit(int startRow, int rows, int totalRows, float sampleRate, + int clients, boolean flushCommits, boolean writeToWAL, boolean reportLatency, + boolean filterAll) { this.startRow = startRow; this.rows = rows; this.totalRows = totalRows; + this.sampleRate = sampleRate; this.clients = clients; this.flushCommits = flushCommits; this.writeToWAL = writeToWAL; + this.reportLatency = reportLatency; + this.filterAll = filterAll; } @Override @@ -234,9 +250,12 @@ public void readFields(DataInput in) throws IOException { this.startRow = in.readInt(); this.rows = in.readInt(); this.totalRows = in.readInt(); + this.sampleRate = in.readFloat(); this.clients = in.readInt(); this.flushCommits = in.readBoolean(); this.writeToWAL = in.readBoolean(); + this.reportLatency = in.readBoolean(); + this.filterAll = in.readBoolean(); } @Override @@ -244,9 +263,12 @@ public void write(DataOutput out) throws IOException { out.writeInt(startRow); out.writeInt(rows); out.writeInt(totalRows); + out.writeFloat(sampleRate); out.writeInt(clients); out.writeBoolean(flushCommits); out.writeBoolean(writeToWAL); + out.writeBoolean(reportLatency); + out.writeBoolean(filterAll); } @Override @@ -271,6 +293,10 @@ public int getTotalRows() { return totalRows; } + public float getSampleRate() { + return sampleRate; + } + public int getClients() { return clients; } @@ -282,6 +308,14 @@ public boolean isFlushCommits() { public boolean isWriteToWAL() { return writeToWAL; } + + public boolean isReportLatency() { + return reportLatency; + } + + public boolean isFilterAll() { + return filterAll; + } } /** @@ -312,21 +346,27 @@ public List getSplits(JobContext job) throws IOException { int startRow = Integer.parseInt(m.group(1)); int rows = Integer.parseInt(m.group(2)); int totalRows = Integer.parseInt(m.group(3)); - int clients = Integer.parseInt(m.group(4)); - boolean flushCommits = Boolean.parseBoolean(m.group(5)); - boolean writeToWAL = Boolean.parseBoolean(m.group(6)); + float sampleRate = Float.parseFloat(m.group(4)); + int clients = Integer.parseInt(m.group(5)); + boolean flushCommits = Boolean.parseBoolean(m.group(6)); + boolean writeToWAL = Boolean.parseBoolean(m.group(7)); + boolean reportLatency = Boolean.parseBoolean(m.group(8)); + boolean filterAll = Boolean.parseBoolean(m.group(9)); LOG.debug("split["+ splitList.size() + "] " + " startRow=" + startRow + " rows=" + rows + " totalRows=" + totalRows + + " sampleRate=" + sampleRate + " clients=" + clients + " flushCommits=" + flushCommits + - " writeToWAL=" + writeToWAL); + " writeToWAL=" + writeToWAL + + " reportLatency=" + reportLatency + + " filterAll=" + filterAll); PeInputSplit newSplit = - new PeInputSplit(startRow, rows, totalRows, clients, - flushCommits, writeToWAL); + new PeInputSplit(startRow, rows, totalRows, sampleRate, clients, + flushCommits, writeToWAL, reportLatency, filterAll); splitList.add(newSplit); } } @@ -363,7 +403,7 @@ public boolean nextKeyValue() throws IOException, InterruptedException { } key = NullWritable.get(); - value = (PeInputSplit)split; + value = split; readOver = true; return true; @@ -446,9 +486,9 @@ public void setStatus(String msg) { // Evaluation task long elapsedTime = this.pe.runOneClient(this.cmd, value.getStartRow(), - value.getRows(), value.getTotalRows(), - value.isFlushCommits(), value.isWriteToWAL(), - status); + value.getRows(), value.getTotalRows(), value.getSampleRate(), + value.isFlushCommits(), value.isWriteToWAL(), value.isReportLatency(), + value.isFilterAll(), status); // Collect how much time the thing took. Report as map output and // to the ELAPSED_TIME counter. context.getCounter(Counter.ELAPSED_TIME).increment(elapsedTime); @@ -487,11 +527,17 @@ private boolean checkTable(HBaseAdmin admin) throws IOException { LOG.info("Table " + tableDescriptor + " created"); } } - boolean tableExists = admin.tableExists(tableDescriptor.getName()); - return tableExists; + return admin.tableExists(tableDescriptor.getName()); } protected HTableDescriptor getTableDescriptor() { + if (TABLE_DESCRIPTOR == null) { + TABLE_DESCRIPTOR = new HTableDescriptor(tableName); + HColumnDescriptor family = new HColumnDescriptor(FAMILY_NAME); + family.setDataBlockEncoding(blockEncoding); + family.setCompressionType(compression); + TABLE_DESCRIPTOR.addFamily(family); + } return TABLE_DESCRIPTOR; } @@ -504,10 +550,11 @@ protected byte[][] getSplits() { if (this.presplitRegions == 0) return new byte [0][]; - byte[][] splits = new byte[this.presplitRegions][]; - int jump = this.R / this.presplitRegions; - for (int i=0; i cmd) throws IOException, InterruptedException, ClassNotFoundException { - checkTable(new HBaseAdmin(conf)); + checkTable(new HBaseAdmin(getConf())); if (this.nomapred) { doMultipleClients(cmd); } else { @@ -542,12 +589,12 @@ private void doMultipleClients(final Class cmd) throws IOExcepti @Override public void run() { super.run(); - PerformanceEvaluation pe = new PerformanceEvaluation(conf); + PerformanceEvaluation pe = new PerformanceEvaluation(getConf()); int index = Integer.parseInt(getName()); try { long elapsedTime = pe.runOneClient(cmd, index * perClientRows, - perClientRows, R, - flushCommits, writeToWAL, new Status() { + perClientRows, R, sampleRate, flushCommits, writeToWAL, + reportLatency, filterAll, new Status() { public void setStatus(final String msg) throws IOException { LOG.info("client-" + getName() + " " + msg); } @@ -584,10 +631,11 @@ public void setStatus(final String msg) throws IOException { */ private void doMapReduce(final Class cmd) throws IOException, InterruptedException, ClassNotFoundException { - Path inputDir = writeInputFile(this.conf); - this.conf.set(EvaluationMapTask.CMD_KEY, cmd.getName()); - this.conf.set(EvaluationMapTask.PE_KEY, getClass().getName()); - Job job = new Job(this.conf); + Configuration conf = getConf(); + Path inputDir = writeInputFile(conf); + conf.set(EvaluationMapTask.CMD_KEY, cmd.getName()); + conf.set(EvaluationMapTask.PE_KEY, getClass().getName()); + Job job = new Job(conf); job.setJarByClass(PerformanceEvaluation.class); job.setJobName("HBase Performance Evaluation"); @@ -641,9 +689,12 @@ private Path writeInputFile(final Configuration c) throws IOException { String s = "startRow=" + ((j * perClientRows) + (i * (perClientRows/10))) + ", perClientRunRows=" + (perClientRows / 10) + ", totalRows=" + this.R + + ", sampleRate=" + this.sampleRate + ", clients=" + this.N + ", flushCommits=" + this.flushCommits + - ", writeToWAL=" + this.writeToWAL; + ", writeToWAL=" + this.writeToWAL + + ", reportLatency=" + this.reportLatency + + ", filterAll=" + this.filterAll; int hash = h.hash(Bytes.toBytes(s)); m.put(hash, s); } @@ -692,20 +743,27 @@ static class TestOptions { private int startRow; private int perClientRunRows; private int totalRows; + private float sampleRate; private byte[] tableName; private boolean flushCommits; private boolean writeToWAL = true; + private boolean reportLatency; + private boolean filterAll; - TestOptions() { - } + TestOptions() {} - TestOptions(int startRow, int perClientRunRows, int totalRows, byte[] tableName, boolean flushCommits, boolean writeToWAL) { + TestOptions(int startRow, int perClientRunRows, int totalRows, float sampleRate, + byte[] tableName, boolean flushCommits, boolean writeToWAL, boolean reportLatency, + boolean filterAll) { this.startRow = startRow; this.perClientRunRows = perClientRunRows; this.totalRows = totalRows; + this.sampleRate = sampleRate; this.tableName = tableName; this.flushCommits = flushCommits; this.writeToWAL = writeToWAL; + this.reportLatency = reportLatency; + this.filterAll = filterAll; } public int getStartRow() { @@ -720,6 +778,10 @@ public int getTotalRows() { return totalRows; } + public float getSampleRate() { + return sampleRate; + } + public byte[] getTableName() { return tableName; } @@ -731,6 +793,14 @@ public boolean isFlushCommits() { public boolean isWriteToWAL() { return writeToWAL; } + + public boolean isReportLatency() { + return reportLatency; + } + + public boolean isFilterAll() { + return filterAll; + } } /* @@ -750,6 +820,7 @@ private static long nextRandomSeed() { protected final int startRow; protected final int perClientRunRows; protected final int totalRows; + protected final float sampleRate; private final Status status; protected byte[] tableName; protected HBaseAdmin admin; @@ -757,6 +828,8 @@ private static long nextRandomSeed() { protected volatile Configuration conf; protected boolean flushCommits; protected boolean writeToWAL; + protected boolean reportlatency; + protected boolean filterAll; /** * Note that all subclasses of this class must provide a public contructor @@ -767,12 +840,15 @@ private static long nextRandomSeed() { this.startRow = options.getStartRow(); this.perClientRunRows = options.getPerClientRunRows(); this.totalRows = options.getTotalRows(); + this.sampleRate = options.getSampleRate(); this.status = status; this.tableName = options.getTableName(); this.table = null; this.conf = conf; this.flushCommits = options.isFlushCommits(); this.writeToWAL = options.isWriteToWAL(); + this.reportlatency = options.isReportLatency(); + this.filterAll = options.isFilterAll(); } private String generateStatus(final int sr, final int i, final int lr) { @@ -781,7 +857,7 @@ private String generateStatus(final int sr, final int i, final int lr) { protected int getReportingPeriod() { int period = this.perClientRunRows / 10; - return period == 0? this.perClientRunRows: period; + return period == 0 ? this.perClientRunRows : period; } void testSetup() throws IOException { @@ -848,19 +924,21 @@ static class RandomSeekScanTest extends Test { void testRow(final int i) throws IOException { Scan scan = new Scan(getRandomRow(this.rand, this.totalRows)); scan.addColumn(FAMILY_NAME, QUALIFIER_NAME); - scan.setFilter(new WhileMatchFilter(new PageFilter(120))); - ResultScanner s = this.table.getScanner(scan); - //int count = 0; - for (Result rr = null; (rr = s.next()) != null;) { - // LOG.info("" + count++ + " " + rr.toString()); + FilterList list = new FilterList(); + if (this.filterAll) { + list.addFilter(new FilterAllFilter()); } + list.addFilter(new WhileMatchFilter(new PageFilter(120))); + scan.setFilter(list); + ResultScanner s = this.table.getScanner(scan); + for (Result rr; (rr = s.next()) != null;) ; s.close(); } @Override protected int getReportingPeriod() { int period = this.perClientRunRows / 100; - return period == 0? this.perClientRunRows: period; + return period == 0 ? this.perClientRunRows : period; } } @@ -876,9 +954,12 @@ void testRow(final int i) throws IOException { Pair startAndStopRow = getStartAndStopRow(); Scan scan = new Scan(startAndStopRow.getFirst(), startAndStopRow.getSecond()); scan.addColumn(FAMILY_NAME, QUALIFIER_NAME); + if (this.filterAll) { + scan.setFilter(new FilterAllFilter()); + } ResultScanner s = this.table.getScanner(scan); int count = 0; - for (Result rr = null; (rr = s.next()) != null;) { + for (Result rr; (rr = s.next()) != null;) { count++; } @@ -951,23 +1032,52 @@ protected Pair getStartAndStopRow() { } static class RandomReadTest extends Test { + private final int everyN; + private final boolean reportLatency; + private final float[] times; + int idx = 0; + RandomReadTest(Configuration conf, TestOptions options, Status status) { super(conf, options, status); + everyN = (int) (this.totalRows / (this.totalRows * this.sampleRate)); + LOG.info("Sampling 1 every " + everyN + " out of " + perClientRunRows + " total rows."); + this.reportLatency = options.isReportLatency(); + if (this.reportLatency) { + times = new float[(int) Math.ceil(this.perClientRunRows * this.sampleRate)]; + } else { + times = null; + } } @Override void testRow(final int i) throws IOException { - Get get = new Get(getRandomRow(this.rand, this.totalRows)); - get.addColumn(FAMILY_NAME, QUALIFIER_NAME); - this.table.get(get); + if (i % everyN == 0) { + Get get = new Get(getRandomRow(this.rand, this.totalRows)); + get.addColumn(FAMILY_NAME, QUALIFIER_NAME); + if (this.filterAll) { + get.setFilter(new FilterAllFilter()); + } + long start = System.nanoTime(); + this.table.get(get); + if (this.reportLatency) { + times[idx++] = (float) ((System.nanoTime() - start) / 1000000.0); + } + } } @Override protected int getReportingPeriod() { int period = this.perClientRunRows / 100; - return period == 0? this.perClientRunRows: period; + return period == 0 ? this.perClientRunRows : period; } + @Override + protected void testTakedown() throws IOException { + super.testTakedown(); + if (this.reportLatency) { + LOG.info("randomRead latency log (ms): " + Arrays.toString(times)); + } + } } static class RandomWriteTest extends Test { @@ -1012,6 +1122,9 @@ void testRow(final int i) throws IOException { if (this.testScanner == null) { Scan scan = new Scan(format(this.startRow)); scan.addColumn(FAMILY_NAME, QUALIFIER_NAME); + if (this.filterAll) { + scan.setFilter(new FilterAllFilter()); + } this.testScanner = table.getScanner(scan); } testScanner.next(); @@ -1028,9 +1141,11 @@ static class SequentialReadTest extends Test { void testRow(final int i) throws IOException { Get get = new Get(format(i)); get.addColumn(FAMILY_NAME, QUALIFIER_NAME); + if (this.filterAll) { + get.setFilter(new FilterAllFilter()); + } table.get(get); } - } static class SequentialWriteTest extends Test { @@ -1046,7 +1161,6 @@ void testRow(final int i) throws IOException { put.setWriteToWAL(writeToWAL); table.put(put); } - } static class FilteredScanTest extends Test { @@ -1071,25 +1185,46 @@ void testRow(int i) throws IOException { } protected Scan constructScan(byte[] valuePrefix) throws IOException { - Filter filter = new SingleColumnValueFilter( - FAMILY_NAME, QUALIFIER_NAME, CompareFilter.CompareOp.EQUAL, - new BinaryComparator(valuePrefix) - ); Scan scan = new Scan(); scan.addColumn(FAMILY_NAME, QUALIFIER_NAME); - scan.setFilter(filter); + FilterList list = new FilterList(); + list.addFilter(new SingleColumnValueFilter( + FAMILY_NAME, QUALIFIER_NAME, CompareFilter.CompareOp.EQUAL, + new BinaryComparator(valuePrefix) + )); + if (this.filterAll) { + list.addFilter(new FilterAllFilter()); + } + scan.setFilter(list); return scan; } } + /** + * Compute a throughput rate in MB/s. + * @param rows Number of records consumed. + * @param timeMs Time taken in milliseconds. + * @return String value with label, ie '123.76 MB/s' + */ + private static String calculateMbps(int rows, long timeMs) { + // MB/s = ((totalRows * ROW_SIZE_BYTES) / totalTimeMS) + // * 1000 MS_PER_SEC / (1024 * 1024) BYTES_PER_MB + BigDecimal rowSize = + BigDecimal.valueOf(VALUE_LENGTH + VALUE_LENGTH + FAMILY_NAME.length + QUALIFIER_NAME.length); + BigDecimal mbps = BigDecimal.valueOf(rows).multiply(rowSize, CXT) + .divide(BigDecimal.valueOf(timeMs), CXT).multiply(MS_PER_SEC, CXT) + .divide(BYTES_PER_MB, CXT); + return FMT.format(mbps) + " MB/s"; + } + /* * Format passed integer. * @param number - * @return Returns zero-prefixed 10-byte wide decimal version of passed + * @return Returns zero-prefixed ROW_LENGTH-byte wide decimal version of passed * number (Does absolute in case number is negative). */ public static byte [] format(final int number) { - byte [] b = new byte[10]; + byte [] b = new byte[ROW_LENGTH]; int d = Math.abs(number); for (int i = b.length - 1; i >= 0; i--) { b[i] = (byte)((d % 10) + '0'); @@ -1105,8 +1240,24 @@ protected Scan constructScan(byte[] valuePrefix) throws IOException { * @return Generated random value to insert into a table cell. */ public static byte[] generateValue(final Random r) { - byte [] b = new byte [ROW_LENGTH]; - r.nextBytes(b); + byte [] b = new byte [VALUE_LENGTH]; + int i = 0; + + for(i = 0; i < (ROW_LENGTH-8); i += 8) { + b[i] = (byte) (65 + r.nextInt(26)); + b[i+1] = b[i]; + b[i+2] = b[i]; + b[i+3] = b[i]; + b[i+4] = b[i]; + b[i+5] = b[i]; + b[i+6] = b[i]; + b[i+7] = b[i]; + } + + byte a = (byte) (65 + r.nextInt(26)); + for(; i < ROW_LENGTH; i++) { + b[i] = a; + } return b; } @@ -1115,21 +1266,22 @@ public static byte[] generateValue(final Random r) { } long runOneClient(final Class cmd, final int startRow, - final int perClientRunRows, final int totalRows, - boolean flushCommits, boolean writeToWAL, - final Status status) + final int perClientRunRows, final int totalRows, final float sampleRate, + boolean flushCommits, boolean writeToWAL, boolean reportLatency, + boolean filterAll, final Status status) throws IOException { status.setStatus("Start " + cmd + " at offset " + startRow + " for " + perClientRunRows + " rows"); long totalElapsedTime = 0; Test t = null; - TestOptions options = new TestOptions(startRow, perClientRunRows, - totalRows, getTableDescriptor().getName(), flushCommits, writeToWAL); + TestOptions options = new TestOptions(startRow, perClientRunRows, totalRows, + sampleRate, getTableDescriptor().getName(), flushCommits, writeToWAL, reportLatency, + filterAll); try { Constructor constructor = cmd.getDeclaredConstructor( Configuration.class, TestOptions.class, Status.class); - t = constructor.newInstance(this.conf, options, status); + t = constructor.newInstance(getConf(), options, status); } catch (NoSuchMethodException e) { throw new IllegalArgumentException("Invalid command class: " + cmd.getName() + ". It does not provide a constructor as described by" + @@ -1141,11 +1293,12 @@ long runOneClient(final Class cmd, final int startRow, totalElapsedTime = t.test(); status.setStatus("Finished " + cmd + " in " + totalElapsedTime + - "ms at offset " + startRow + " for " + perClientRunRows + " rows"); + "ms at offset " + startRow + " for " + perClientRunRows + " rows" + + " (" + calculateMbps((int)(perClientRunRows * sampleRate), totalElapsedTime) + ")"); return totalElapsedTime; } - private void runNIsOne(final Class cmd) { + private void runNIsOne(final Class cmd) throws IOException { Status status = new Status() { public void setStatus(String msg) throws IOException { LOG.info(msg); @@ -1154,12 +1307,14 @@ public void setStatus(String msg) throws IOException { HBaseAdmin admin = null; try { - admin = new HBaseAdmin(this.conf); + admin = new HBaseAdmin(getConf()); checkTable(admin); - runOneClient(cmd, 0, this.R, this.R, this.flushCommits, this.writeToWAL, - status); + runOneClient(cmd, 0, this.R, this.R, this.sampleRate, this.flushCommits, + this.writeToWAL, this.reportLatency, this.filterAll, status); } catch (Exception e) { LOG.error("Failed", e); + } finally { + if (admin != null) admin.close(); } } @@ -1168,6 +1323,7 @@ private void runTest(final Class cmd) throws IOException, MiniHBaseCluster hbaseMiniCluster = null; MiniDFSCluster dfsCluster = null; MiniZooKeeperCluster zooKeeperCluster = null; + Configuration conf = getConf(); if (this.miniCluster) { dfsCluster = new MiniDFSCluster(conf, 2, true, (String[])null); zooKeeperCluster = new MiniZooKeeperCluster(); @@ -1182,7 +1338,7 @@ private void runTest(final Class cmd) throws IOException, conf.set(HConstants.HBASE_DIR, parentdir.toString()); fs.mkdirs(parentdir); FSUtils.setVersion(fs, parentdir); - hbaseMiniCluster = new MiniHBaseCluster(this.conf, N); + hbaseMiniCluster = new MiniHBaseCluster(conf, N); } try { @@ -1212,16 +1368,33 @@ protected void printUsage(final String message) { System.err.println(message); } System.err.println("Usage: java " + this.getClass().getName() + " \\"); - System.err.println(" [--miniCluster] [--nomapred] [--rows=ROWS] "); + System.err.println(" [--miniCluster] [--nomapred] [--rows=ROWS] [--table=NAME] \\"); + System.err.println(" [--compress=TYPE] [--blockEncoding=TYPE] [-D]* "); System.err.println(); System.err.println("Options:"); System.err.println(" miniCluster Run the test on an HBaseMiniCluster"); System.err.println(" nomapred Run multiple clients using threads " + "(rather than use mapreduce)"); System.err.println(" rows Rows each client runs. Default: One million"); - System.err.println(" flushCommits Used to determine if the test should flush the table. Default: false"); + System.err.println(" table Alternate table name. Default: 'TestTable'"); + System.err.println(" compress Compression type to use (GZ, LZO, ...). Default: 'NONE'"); + System.err.println(" sampleRate Execute test on a sample of total " + + "rows. Only supported by randomRead. Default: 1.0"); + System.err.println(" flushCommits Used to determine if the test should flush the table. " + + "Default: false"); System.err.println(" writeToWAL Set writeToWAL on puts. Default: True"); - System.err.println(" presplit Create presplit table. Recommended for accurate perf analysis (see guide). Default: disabled"); + System.err.println(" presplit Create presplit table. Recommended for accurate perf " + + "analysis (see guide). Default: disabled"); + System.err.println(" filterAll Helps to filter out all the rows on the server side" + + " there by not returning any thing back to the client. Helps to check the server side" + + " performance. Uses FilterAllFilter internally. "); + System.err.println(" latency Set to report operation latencies. " + + "Currently only supported by randomRead test. Default: False"); + System.err.println(); + System.err.println(" Note: -D properties will be applied to the conf used. "); + System.err.println(" For example: "); + System.err.println(" -Dmapred.output.compress=true"); + System.err.println(" -Dmapreduce.task.timeout=60000"); System.err.println(); System.err.println("Command:"); for (CmdDescriptor command : commands.values()) { @@ -1250,7 +1423,7 @@ private void getArgs(final int start, final String[] args) { this.R = this.R * N; } - public int doCommandLine(final String[] args) { + public int run(String[] args) throws Exception { // Process command-line args. TODO: Better cmd-line processing // (but hopefully something not as painful as cli options). int errCode = -1; @@ -1286,6 +1459,30 @@ public int doCommandLine(final String[] args) { continue; } + final String sampleRate = "--sampleRate="; + if (cmd.startsWith(sampleRate)) { + this.sampleRate = Float.parseFloat(cmd.substring(sampleRate.length())); + continue; + } + + final String table = "--table="; + if (cmd.startsWith(table)) { + this.tableName = Bytes.toBytes(cmd.substring(table.length())); + continue; + } + + final String compress = "--compress="; + if (cmd.startsWith(compress)) { + this.compression = Compression.Algorithm.valueOf(cmd.substring(compress.length())); + continue; + } + + final String blockEncoding = "--blockEncoding="; + if (cmd.startsWith(blockEncoding)) { + this.blockEncoding = DataBlockEncoding.valueOf(cmd.substring(blockEncoding.length())); + continue; + } + final String flushCommits = "--flushCommits="; if (cmd.startsWith(flushCommits)) { this.flushCommits = Boolean.parseBoolean(cmd.substring(flushCommits.length())); @@ -1304,6 +1501,18 @@ public int doCommandLine(final String[] args) { continue; } + final String latency = "--latency"; + if (cmd.startsWith(latency)) { + this.reportLatency = true; + continue; + } + + final String filterOutAll = "--filterAll"; + if (cmd.startsWith(filterOutAll)) { + this.filterAll = true; + continue; + } + Class cmdClass = determineCommandClass(cmd); if (cmdClass != null) { getArgs(i + 1, args); @@ -1327,11 +1536,8 @@ private Class determineCommandClass(String cmd) { return descriptor != null ? descriptor.getCmdClass() : null; } - /** - * @param args - */ - public static void main(final String[] args) { - Configuration c = HBaseConfiguration.create(); - System.exit(new PerformanceEvaluation(c).doCommandLine(args)); + public static void main(final String[] args) throws Exception { + int res = ToolRunner.run(new PerformanceEvaluation(HBaseConfiguration.create()), args); + System.exit(res); } } diff --git a/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluationCommons.java b/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluationCommons.java index eac7207cb0e8..86fe1360e23a 100644 --- a/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluationCommons.java +++ b/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluationCommons.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java b/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java index 8b9b1bb5a159..118a0633cf2e 100644 --- a/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java +++ b/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -38,6 +37,11 @@ public class ResourceChecker { private static final Log LOG = LogFactory.getLog(ResourceChecker.class); + enum Phase { + INITIAL, INTERMEDIATE, END + } + private static Set initialThreadNames = new HashSet(); + /** * On unix, we know how to get the number of open file descriptor */ @@ -45,8 +49,14 @@ private static class ResourceAnalyzer { private static final OperatingSystemMXBean osStats; private static final UnixOperatingSystemMXBean unixOsStats; - public long getThreadsCount() { - return Thread.getAllStackTraces().size(); + public long getThreadsCount(Phase phase) { + Map stackTraces = Thread.getAllStackTraces(); + if (phase == Phase.INITIAL) { + for (Thread t : stackTraces.keySet()) { + initialThreadNames.add(t.getName()); + } + } + return stackTraces.size(); } public long getOpenFileDescriptorCount() { @@ -102,14 +112,14 @@ public long getConnectionCount(){ public boolean checkThreads(String tagLine) { boolean isOk = true; + long threadCount = rc.getThreadsCount(Phase.INTERMEDIATE); - if (rc.getThreadsCount() > MAX_THREADS_COUNT) { + if (threadCount > MAX_THREADS_COUNT) { LOG.error( tagLine + ": too many threads used. We use " + - rc.getThreadsCount() + " our max is " + MAX_THREADS_COUNT); + threadCount + " our max is " + MAX_THREADS_COUNT); isOk = false; } - return isOk; } @@ -132,19 +142,20 @@ public final void init(String tagLine) { rc.getMaxFileDescriptorCount() + " our is " + MAX_FILE_HANDLES_COUNT); } - logInfo(tagLine); + logInfo(Phase.INITIAL, tagLine); - initialThreadsCount = rc.getThreadsCount(); + initialThreadsCount = rc.getThreadsCount(Phase.INITIAL); initialFileHandlesCount = rc.getOpenFileDescriptorCount(); initialConnectionCount= rc.getConnectionCount(); check(tagLine); } - public void logInfo(String tagLine) { + public void logInfo(Phase phase, String tagLine) { + long threadCount = rc.getThreadsCount(phase); LOG.info( - tagLine + ": " + - rc.getThreadsCount() + " threads" + + tagLine + ": " + + threadCount + " threads" + (initialThreadsCount > 0 ? " (was " + initialThreadsCount + "), " : ", ") + rc.getOpenFileDescriptorCount() + " file descriptors" + @@ -153,7 +164,7 @@ public void logInfo(String tagLine) { rc.getConnectionCount() + " connections" + (initialConnectionCount > 0 ? " (was " + initialConnectionCount + "), " : ", ") + - (initialThreadsCount > 0 && rc.getThreadsCount() > initialThreadsCount ? + (initialThreadsCount > 0 && threadCount > initialThreadsCount ? " -thread leak?- " : "") + (initialFileHandlesCount > 0 && rc.getOpenFileDescriptorCount() > initialFileHandlesCount ? @@ -162,6 +173,20 @@ public void logInfo(String tagLine) { rc.getConnectionCount() > initialConnectionCount ? " -connection leak?- " : "" ) ); + if (phase == Phase.END) { + Map stackTraces = Thread.getAllStackTraces(); + if (stackTraces.size() > initialThreadNames.size()) { + for (Thread t : stackTraces.keySet()) { + if (!initialThreadNames.contains(t.getName())) { + LOG.info(tagLine + ": potentially hanging thread - " + t.getName()); + StackTraceElement[] stackElements = stackTraces.get(t); + for (StackTraceElement ele : stackElements) { + LOG.info("\t" + ele); + } + } + } + } + } } diff --git a/src/test/java/org/apache/hadoop/hbase/ResourceCheckerJUnitRule.java b/src/test/java/org/apache/hadoop/hbase/ResourceCheckerJUnitRule.java index 6d1ab258c70c..86fe4c11bb22 100644 --- a/src/test/java/org/apache/hadoop/hbase/ResourceCheckerJUnitRule.java +++ b/src/test/java/org/apache/hadoop/hbase/ResourceCheckerJUnitRule.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -48,7 +47,7 @@ private void start(String testName) { private void end(String testName) { if (!endDone) { endDone = true; - cu.logInfo("after " + testName); + cu.logInfo(ResourceChecker.Phase.END, "after " + testName); cu.check("after "+testName); } } diff --git a/src/test/java/org/apache/hadoop/hbase/SmallTests.java b/src/test/java/org/apache/hadoop/hbase/SmallTests.java index c702f5a1fd5e..6953667d6998 100644 --- a/src/test/java/org/apache/hadoop/hbase/SmallTests.java +++ b/src/test/java/org/apache/hadoop/hbase/SmallTests.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -29,6 +28,7 @@ * * @see MediumTests * @see LargeTests + * @see IntegrationTests */ public interface SmallTests { } diff --git a/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java b/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java index 4643e6af909d..2e43387ccec3 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java +++ b/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -27,30 +26,34 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.MultithreadedTestUtil.TestContext; import org.apache.hadoop.hbase.MultithreadedTestUtil.RepeatingTestThread; +import org.apache.hadoop.hbase.MultithreadedTestUtil.TestContext; import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy; import org.apache.hadoop.hbase.util.Bytes; -import org.junit.Ignore; +import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; import org.junit.Test; +import org.junit.experimental.categories.Category; import com.google.common.collect.Lists; -import org.junit.experimental.categories.Category; /** * Test case that uses multiple threads to read and write multifamily rows * into a table, verifying that reads never see partially-complete writes. - * + * * This can run as a junit test, or with a main() function which runs against * a real cluster (eg for testing with failures, region movement, etc) */ @Category(MediumTests.class) -public class TestAcidGuarantees { +public class TestAcidGuarantees implements Tool { protected static final Log LOG = LogFactory.getLog(TestAcidGuarantees.class); public static final byte [] TABLE_NAME = Bytes.toBytes("TestAcidGuarantees"); public static final byte [] FAMILY_A = Bytes.toBytes("A"); @@ -65,6 +68,9 @@ public class TestAcidGuarantees { public static int NUM_COLS_TO_CHECK = 50; + // when run as main + private Configuration conf; + private void createTableIfMissing() throws IOException { try { @@ -77,9 +83,12 @@ public TestAcidGuarantees() { // Set small flush size for minicluster so we exercise reseeking scanners Configuration conf = HBaseConfiguration.create(); conf.set(HConstants.HREGION_MEMSTORE_FLUSH_SIZE, String.valueOf(128*1024)); + // prevent aggressive region split + conf.set(HConstants.HBASE_REGION_SPLIT_POLICY_KEY, + ConstantSizeRegionSplitPolicy.class.getName()); util = new HBaseTestingUtility(conf); } - + /** * Thread that does random full-row writes into a table. */ @@ -90,7 +99,7 @@ public static class AtomicityWriter extends RepeatingTestThread { byte targetFamilies[][]; HTable table; AtomicLong numWritten = new AtomicLong(); - + public AtomicityWriter(TestContext ctx, byte targetRows[][], byte targetFamilies[][]) throws IOException { super(ctx); @@ -101,7 +110,7 @@ public AtomicityWriter(TestContext ctx, byte targetRows[][], public void doAnAction() throws Exception { // Pick a random row to write into byte[] targetRow = targetRows[rand.nextInt(targetRows.length)]; - Put p = new Put(targetRow); + Put p = new Put(targetRow); rand.nextBytes(data); for (byte[] family : targetFamilies) { @@ -114,7 +123,7 @@ public void doAnAction() throws Exception { numWritten.getAndIncrement(); } } - + /** * Thread that does single-row reads in a table, looking for partially * completed rows. @@ -144,7 +153,7 @@ public void doAnAction() throws Exception { // ignore this action return; } - + for (byte[] family : targetFamilies) { for (int i = 0; i < NUM_COLS_TO_CHECK; i++) { byte qualifier[] = Bytes.toBytes("col" + i); @@ -173,7 +182,7 @@ private void gotFailure(byte[] expected, Result res) { throw new RuntimeException(msg.toString()); } } - + /** * Thread that does full scans of the table looking for any partially completed * rows. @@ -197,10 +206,10 @@ public void doAnAction() throws Exception { s.addFamily(family); } ResultScanner scanner = table.getScanner(s); - + for (Result res : scanner) { byte[] gotValue = null; - + for (byte[] family : targetFamilies) { for (int i = 0; i < NUM_COLS_TO_CHECK; i++) { byte qualifier[] = Bytes.toBytes("col" + i); @@ -231,20 +240,28 @@ private void gotFailure(byte[] expected, Result res) { } } - public void runTestAtomicity(long millisToRun, int numWriters, int numGetters, int numScanners, int numUniqueRows) throws Exception { + runTestAtomicity(millisToRun, numWriters, numGetters, numScanners, numUniqueRows, false); + } + + public void runTestAtomicity(long millisToRun, + int numWriters, + int numGetters, + int numScanners, + int numUniqueRows, + final boolean systemTest) throws Exception { createTableIfMissing(); TestContext ctx = new TestContext(util.getConfiguration()); - + byte rows[][] = new byte[numUniqueRows][]; for (int i = 0; i < numUniqueRows; i++) { rows[i] = Bytes.toBytes("test_row_" + i); } - + List writers = Lists.newArrayList(); for (int i = 0; i < numWriters; i++) { AtomicityWriter writer = new AtomicityWriter( @@ -254,8 +271,22 @@ public void runTestAtomicity(long millisToRun, } // Add a flusher ctx.addThread(new RepeatingTestThread(ctx) { + HBaseAdmin admin = new HBaseAdmin(util.getConfiguration()); public void doAnAction() throws Exception { - util.flush(); + try { + admin.flush(TABLE_NAME); + } catch(IOException ioe) { + LOG.warn("Ignoring exception while flushing: " + StringUtils.stringifyException(ioe)); + } + // Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally, + // we would flush as often as possible. On a running cluster, this isn't practical: + // (1) we will cause a lot of load due to all the flushing and compacting + // (2) we cannot change the flushing/compacting related Configuration options to try to + // alleviate this + // (3) it is an unrealistic workload, since no one would actually flush that often. + // Therefore, let's flush every minute to have more flushes than usual, but not overload + // the running cluster. + if (systemTest) Thread.sleep(60000); } }); @@ -266,18 +297,18 @@ public void doAnAction() throws Exception { getters.add(getter); ctx.addThread(getter); } - + List scanners = Lists.newArrayList(); for (int i = 0; i < numScanners; i++) { AtomicScanReader scanner = new AtomicScanReader(ctx, FAMILIES); scanners.add(scanner); ctx.addThread(scanner); } - + ctx.startThreads(); ctx.waitFor(millisToRun); ctx.stop(); - + LOG.info("Finished test. Writers:"); for (AtomicityWriter writer : writers) { LOG.info(" wrote " + writer.numWritten.get()); @@ -300,7 +331,7 @@ public void testGetAtomicity() throws Exception { runTestAtomicity(20000, 5, 5, 0, 3); } finally { util.shutdownMiniCluster(); - } + } } @Test @@ -310,7 +341,7 @@ public void testScanAtomicity() throws Exception { runTestAtomicity(20000, 5, 0, 5, 3); } finally { util.shutdownMiniCluster(); - } + } } @Test @@ -320,19 +351,48 @@ public void testMixedAtomicity() throws Exception { runTestAtomicity(20000, 5, 2, 2, 3); } finally { util.shutdownMiniCluster(); - } + } + } + + //////////////////////////////////////////////////////////////////////////// + // Tool interface + //////////////////////////////////////////////////////////////////////////// + @Override + public Configuration getConf() { + return conf; + } + + @Override + public void setConf(Configuration c) { + this.conf = c; + this.util = new HBaseTestingUtility(c); + } + + @Override + public int run(String[] arg0) throws Exception { + Configuration c = getConf(); + int millis = c.getInt("millis", 5000); + int numWriters = c.getInt("numWriters", 50); + int numGetters = c.getInt("numGetters", 2); + int numScanners = c.getInt("numScanners", 2); + int numUniqueRows = c.getInt("numUniqueRows", 3); + runTestAtomicity(millis, numWriters, numGetters, numScanners, numUniqueRows, true); + return 0; } public static void main(String args[]) throws Exception { Configuration c = HBaseConfiguration.create(); - TestAcidGuarantees test = new TestAcidGuarantees(); - test.setConf(c); - test.runTestAtomicity(5000, 50, 2, 2, 3); + int status; + try { + TestAcidGuarantees test = new TestAcidGuarantees(); + status = ToolRunner.run(c, test, args); + } catch (Exception e) { + LOG.error("Exiting due to error", e); + status = -1; + } + System.exit(status); } - private void setConf(Configuration c) { - util = new HBaseTestingUtility(c); - } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/TestCheckTestClasses.java b/src/test/java/org/apache/hadoop/hbase/TestCheckTestClasses.java new file mode 100644 index 000000000000..b5413bc95a19 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/TestCheckTestClasses.java @@ -0,0 +1,61 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import static junit.framework.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.FileFilter; +import java.io.IOException; +import java.lang.reflect.Method; +import java.lang.reflect.Modifier; +import java.net.URL; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.regex.Pattern; + +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.junit.runners.Suite; + + +/** + * Checks tests are categorized. + */ +@Category(SmallTests.class) +public class TestCheckTestClasses { + /** + * Throws an assertion if we find a test class without category (small/medium/large/integration). + * List all the test classes without category in the assertion message. + */ + @Test + public void checkClasses() throws Exception { + List> badClasses = new java.util.ArrayList>(); + ClassTestFinder classFinder = new ClassTestFinder(); + for (Class c : classFinder.findClasses(false)) { + if (ClassTestFinder.getCategoryAnnotations(c).length == 0) { + badClasses.add(c); + } + } + assertTrue("There are " + badClasses.size() + " test classes without category: " + + badClasses, badClasses.isEmpty()); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/TestClassFinder.java b/src/test/java/org/apache/hadoop/hbase/TestClassFinder.java new file mode 100644 index 000000000000..3bd8e655eda8 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/TestClassFinder.java @@ -0,0 +1,345 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import static org.junit.Assert.*; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.PrintStream; +import java.lang.reflect.Method; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.atomic.AtomicLong; +import java.util.jar.*; +import javax.tools.*; + +import org.apache.hadoop.hbase.SmallTests; + +import org.junit.experimental.categories.Category; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; + +import org.apache.commons.io.FileUtils; + +@Category(SmallTests.class) +public class TestClassFinder { + private static final HBaseTestingUtility testUtil = new HBaseTestingUtility(); + private static final String BASEPKG = "tfcpkg"; + // Use unique jar/class/package names in each test case with the help + // of these global counters; we are mucking with ClassLoader in this test + // and we don't want individual test cases to conflict via it. + private static AtomicLong testCounter = new AtomicLong(0); + private static AtomicLong jarCounter = new AtomicLong(0); + + private static String basePath = null; + + // Default name/class filters for testing. + private static final ClassFinder.FileNameFilter trueNameFilter = + new ClassFinder.FileNameFilter() { + @Override + public boolean isCandidateFile(String fileName, String absFilePath) { + return true; + } + }; + private static final ClassFinder.ClassFilter trueClassFilter = + new ClassFinder.ClassFilter() { + @Override + public boolean isCandidateClass(Class c) { + return true; + } + }; + + @BeforeClass + public static void createTestDir() throws IOException { + basePath = testUtil.getDataTestDir(TestClassFinder.class.getSimpleName()).toString(); + if (!basePath.endsWith("/")) { + basePath += "/"; + } + // Make sure we get a brand new directory. + File testDir = new File(basePath); + if (testDir.exists()) { + deleteTestDir(); + } + assertTrue(testDir.mkdirs()); + } + + @AfterClass + public static void deleteTestDir() throws IOException { + testUtil.cleanupTestDir(TestClassFinder.class.getSimpleName()); + } + + @Test + public void testClassFinderCanFindClassesInJars() throws Exception { + long counter = testCounter.incrementAndGet(); + FileAndPath c1 = compileTestClass(counter, "", "c1"); + FileAndPath c2 = compileTestClass(counter, ".nested", "c2"); + FileAndPath c3 = compileTestClass(counter, "", "c3"); + packageAndLoadJar(c1, c3); + packageAndLoadJar(c2); + + ClassFinder allClassesFinder = new ClassFinder(trueNameFilter, trueClassFilter); + Set> allClasses = allClassesFinder.findClasses( + makePackageName("", counter), false); + assertEquals(3, allClasses.size()); + } + + @Test + public void testClassFinderHandlesConflicts() throws Exception { + long counter = testCounter.incrementAndGet(); + FileAndPath c1 = compileTestClass(counter, "", "c1"); + FileAndPath c2 = compileTestClass(counter, "", "c2"); + packageAndLoadJar(c1, c2); + packageAndLoadJar(c1); + + ClassFinder allClassesFinder = new ClassFinder(trueNameFilter, trueClassFilter); + Set> allClasses = allClassesFinder.findClasses( + makePackageName("", counter), false); + assertEquals(2, allClasses.size()); + } + + @Test + public void testClassFinderHandlesNestedPackages() throws Exception { + final String NESTED = ".nested"; + final String CLASSNAME1 = "c2"; + final String CLASSNAME2 = "c3"; + long counter = testCounter.incrementAndGet(); + FileAndPath c1 = compileTestClass(counter, "", "c1"); + FileAndPath c2 = compileTestClass(counter, NESTED, CLASSNAME1); + FileAndPath c3 = compileTestClass(counter, NESTED, CLASSNAME2); + packageAndLoadJar(c1, c2); + packageAndLoadJar(c3); + + ClassFinder allClassesFinder = new ClassFinder(trueNameFilter, trueClassFilter); + Set> nestedClasses = allClassesFinder.findClasses( + makePackageName(NESTED, counter), false); + assertEquals(2, nestedClasses.size()); + Class nestedClass1 = makeClass(NESTED, CLASSNAME1, counter); + assertTrue(nestedClasses.contains(nestedClass1)); + Class nestedClass2 = makeClass(NESTED, CLASSNAME2, counter); + assertTrue(nestedClasses.contains(nestedClass2)); + } + + @Test + public void testClassFinderFiltersByNameInJar() throws Exception { + final String CLASSNAME = "c1"; + final String CLASSNAMEEXCPREFIX = "c2"; + long counter = testCounter.incrementAndGet(); + FileAndPath c1 = compileTestClass(counter, "", CLASSNAME); + FileAndPath c2 = compileTestClass(counter, "", CLASSNAMEEXCPREFIX + "1"); + FileAndPath c3 = compileTestClass(counter, "", CLASSNAMEEXCPREFIX + "2"); + packageAndLoadJar(c1, c2, c3); + + ClassFinder.FileNameFilter notExcNameFilter = new ClassFinder.FileNameFilter() { + @Override + public boolean isCandidateFile(String fileName, String absFilePath) { + return !fileName.startsWith(CLASSNAMEEXCPREFIX); + } + }; + ClassFinder incClassesFinder = new ClassFinder(notExcNameFilter, trueClassFilter); + Set> incClasses = incClassesFinder.findClasses( + makePackageName("", counter), false); + assertEquals(1, incClasses.size()); + Class incClass = makeClass("", CLASSNAME, counter); + assertTrue(incClasses.contains(incClass)); + } + + @Test + public void testClassFinderFiltersByClassInJar() throws Exception { + final String CLASSNAME = "c1"; + final String CLASSNAMEEXCPREFIX = "c2"; + long counter = testCounter.incrementAndGet(); + FileAndPath c1 = compileTestClass(counter, "", CLASSNAME); + FileAndPath c2 = compileTestClass(counter, "", CLASSNAMEEXCPREFIX + "1"); + FileAndPath c3 = compileTestClass(counter, "", CLASSNAMEEXCPREFIX + "2"); + packageAndLoadJar(c1, c2, c3); + + final ClassFinder.ClassFilter notExcClassFilter = new ClassFinder.ClassFilter() { + @Override + public boolean isCandidateClass(Class c) { + return !c.getSimpleName().startsWith(CLASSNAMEEXCPREFIX); + } + }; + ClassFinder incClassesFinder = new ClassFinder(trueNameFilter, notExcClassFilter); + Set> incClasses = incClassesFinder.findClasses( + makePackageName("", counter), false); + assertEquals(1, incClasses.size()); + Class incClass = makeClass("", CLASSNAME, counter); + assertTrue(incClasses.contains(incClass)); + } + + @Test + public void testClassFinderCanFindClassesInDirs() throws Exception { + // Well, technically, we are not guaranteed that the classes will + // be in dirs, but during normal build they would be. + ClassFinder allClassesFinder = new ClassFinder(trueNameFilter, trueClassFilter); + Set> allClasses = allClassesFinder.findClasses( + this.getClass().getPackage().getName(), false); + assertTrue(allClasses.contains(this.getClass())); + assertTrue(allClasses.contains(ClassFinder.class)); + } + + @Test + public void testClassFinderFiltersByNameInDirs() throws Exception { + final String thisName = this.getClass().getSimpleName(); + ClassFinder.FileNameFilter notThisFilter = new ClassFinder.FileNameFilter() { + @Override + public boolean isCandidateFile(String fileName, String absFilePath) { + return !fileName.equals(thisName + ".class"); + } + }; + String thisPackage = this.getClass().getPackage().getName(); + ClassFinder allClassesFinder = new ClassFinder(trueNameFilter, trueClassFilter); + Set> allClasses = allClassesFinder.findClasses(thisPackage, false); + ClassFinder notThisClassFinder = new ClassFinder(notThisFilter, trueClassFilter); + Set> notAllClasses = notThisClassFinder.findClasses(thisPackage, false); + assertFalse(notAllClasses.contains(this.getClass())); + assertEquals(allClasses.size() - 1, notAllClasses.size()); + } + + @Test + public void testClassFinderFiltersByClassInDirs() throws Exception { + ClassFinder.ClassFilter notThisFilter = new ClassFinder.ClassFilter() { + @Override + public boolean isCandidateClass(Class c) { + return c != TestClassFinder.class; + } + }; + String thisPackage = this.getClass().getPackage().getName(); + ClassFinder allClassesFinder = new ClassFinder(trueNameFilter, trueClassFilter); + Set> allClasses = allClassesFinder.findClasses(thisPackage, false); + ClassFinder notThisClassFinder = new ClassFinder(trueNameFilter, notThisFilter); + Set> notAllClasses = notThisClassFinder.findClasses(thisPackage, false); + assertFalse(notAllClasses.contains(this.getClass())); + assertEquals(allClasses.size() - 1, notAllClasses.size()); + } + + @Test + public void testClassFinderDefaultsToOwnPackage() throws Exception { + // Correct handling of nested packages is tested elsewhere, so here we just assume + // pkgClasses is the correct answer that we don't have to check. + ClassFinder allClassesFinder = new ClassFinder(trueNameFilter, trueClassFilter); + Set> pkgClasses = allClassesFinder.findClasses( + ClassFinder.class.getPackage().getName(), false); + Set> defaultClasses = allClassesFinder.findClasses(false); + assertArrayEquals(pkgClasses.toArray(), defaultClasses.toArray()); + } + + private static class FileAndPath { + String path; + File file; + public FileAndPath(String path, File file) { + this.file = file; + this.path = path; + } + } + + private static Class makeClass(String nestedPkgSuffix, + String className, long counter) throws ClassNotFoundException { + return Class.forName( + makePackageName(nestedPkgSuffix, counter) + "." + className + counter); + } + + private static String makePackageName(String nestedSuffix, long counter) { + return BASEPKG + counter + nestedSuffix; + } + + /** + * Compiles the test class with bogus code into a .class file. + * Unfortunately it's very tedious. + * @param counter Unique test counter. + * @param packageNameSuffix Package name suffix (e.g. ".suffix") for nesting, or "". + * @return The resulting .class file and the location in jar it is supposed to go to. + */ + private static FileAndPath compileTestClass(long counter, + String packageNameSuffix, String classNamePrefix) throws Exception { + classNamePrefix = classNamePrefix + counter; + String packageName = makePackageName(packageNameSuffix, counter); + String javaPath = basePath + classNamePrefix + ".java"; + String classPath = basePath + classNamePrefix + ".class"; + PrintStream source = new PrintStream(javaPath); + source.println("package " + packageName + ";"); + source.println("public class " + classNamePrefix + + " { public static void main(String[] args) { } };"); + source.close(); + JavaCompiler jc = ToolProvider.getSystemJavaCompiler(); + int result = jc.run(null, null, null, javaPath); + assertEquals(0, result); + File classFile = new File(classPath); + assertTrue(classFile.exists()); + return new FileAndPath(packageName.replace('.', '/') + '/', classFile); + } + + /** + * Makes a jar out of some class files. Unfortunately it's very tedious. + * @param filesInJar Files created via compileTestClass. + */ + private static void packageAndLoadJar(FileAndPath... filesInJar) throws Exception { + // First, write the bogus jar file. + String path = basePath + "jar" + jarCounter.incrementAndGet() + ".jar"; + Manifest manifest = new Manifest(); + manifest.getMainAttributes().put(Attributes.Name.MANIFEST_VERSION, "1.0"); + FileOutputStream fos = new FileOutputStream(path); + JarOutputStream jarOutputStream = new JarOutputStream(fos, manifest); + // Directory entries for all packages have to be added explicitly for + // resources to be findable via ClassLoader. Directory entries must end + // with "/"; the initial one is expected to, also. + Set pathsInJar = new HashSet(); + for (FileAndPath fileAndPath : filesInJar) { + String pathToAdd = fileAndPath.path; + while (pathsInJar.add(pathToAdd)) { + int ix = pathToAdd.lastIndexOf('/', pathToAdd.length() - 2); + if (ix < 0) { + break; + } + pathToAdd = pathToAdd.substring(0, ix); + } + } + for (String pathInJar : pathsInJar) { + jarOutputStream.putNextEntry(new JarEntry(pathInJar)); + jarOutputStream.closeEntry(); + } + for (FileAndPath fileAndPath : filesInJar) { + File file = fileAndPath.file; + jarOutputStream.putNextEntry( + new JarEntry(fileAndPath.path + file.getName())); + byte[] allBytes = new byte[(int)file.length()]; + FileInputStream fis = new FileInputStream(file); + fis.read(allBytes); + fis.close(); + jarOutputStream.write(allBytes); + jarOutputStream.closeEntry(); + } + jarOutputStream.close(); + fos.close(); + + // Add the file to classpath. + File jarFile = new File(path); + assertTrue(jarFile.exists()); + URLClassLoader urlClassLoader = (URLClassLoader)ClassLoader.getSystemClassLoader(); + Method method = URLClassLoader.class + .getDeclaredMethod("addURL", new Class[] { URL.class }); + method.setAccessible(true); + method.invoke(urlClassLoader, new Object[] { jarFile.toURI().toURL() }); + } +}; diff --git a/src/test/java/org/apache/hadoop/hbase/TestClusterBootOrder.java b/src/test/java/org/apache/hadoop/hbase/TestClusterBootOrder.java new file mode 100644 index 000000000000..b150d135cc54 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/TestClusterBootOrder.java @@ -0,0 +1,121 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase; + +import static org.junit.Assert.assertTrue; + +import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread; +import org.apache.hadoop.hbase.util.JVMClusterUtil.RegionServerThread; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Tests the boot order indifference between regionserver and master + */ +@Category(MediumTests.class) +public class TestClusterBootOrder { + + private static final long SLEEP_INTERVAL = 1000; + private static final long SLEEP_TIME = 4000; + + private HBaseTestingUtility testUtil; + private LocalHBaseCluster cluster; + private RegionServerThread rs; + private MasterThread master; + + @Before + public void setUp() throws Exception { + testUtil = new HBaseTestingUtility(); + testUtil.startMiniDFSCluster(1); + testUtil.startMiniZKCluster(1); + testUtil.createRootDir(); //manually setup hbase dir to point to minidfscluster + cluster = new LocalHBaseCluster(testUtil.getConfiguration(), 0, 0); + } + + @After + public void tearDown() throws Exception { + cluster.shutdown(); + cluster.join(); + testUtil.shutdownMiniZKCluster(); + testUtil.shutdownMiniDFSCluster(); + } + + private void startRegionServer() throws Exception { + rs = cluster.addRegionServer(); + rs.start(); + + for (int i=0; i * SLEEP_INTERVAL < SLEEP_TIME ;i++) { + //we cannot block on wait for rs at this point , since master is not up. + Thread.sleep(SLEEP_INTERVAL); + assertTrue(rs.isAlive()); + } + } + + private void startMaster() throws Exception { + master = cluster.addMaster(); + master.start(); + + for (int i=0; i * SLEEP_INTERVAL < SLEEP_TIME ;i++) { + Thread.sleep(SLEEP_INTERVAL); + assertTrue(master.isAlive()); + } + } + + private void waitForClusterOnline() { + while (true) { + if (master.getMaster().isInitialized()) { + break; + } + try { + Thread.sleep(100); + } catch (InterruptedException ignored) { + // Keep waiting + } + } + rs.waitForServerOnline(); + } + + /** + * Tests launching the cluster by first starting regionserver, and then the master + * to ensure that it does not matter which is started first. + */ + @Test + public void testBootRegionServerFirst() throws Exception { + startRegionServer(); + startMaster(); + waitForClusterOnline(); + } + + /** + * Tests launching the cluster by first starting master, and then the regionserver + * to ensure that it does not matter which is started first. + */ + @Test + public void testBootMasterFirst() throws Exception { + startMaster(); + startRegionServer(); + waitForClusterOnline(); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java b/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java index d1b8a1b353b0..4679cf27e351 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java +++ b/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.util.FSUtils; import org.apache.hadoop.hbase.util.FSTableDescriptors; import org.apache.hadoop.hbase.util.Threads; +import org.apache.hadoop.hbase.zookeeper.ZKAssign; import org.apache.hadoop.hbase.zookeeper.ZKUtil; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.apache.zookeeper.KeeperException; @@ -53,14 +54,16 @@ public class TestDrainingServer { private static final byte [] TABLENAME = Bytes.toBytes("t"); private static final byte [] FAMILY = Bytes.toBytes("f"); private static final int COUNT_OF_REGIONS = HBaseTestingUtility.KEYS.length; + private static final int NB_SLAVES = 5; /** * Spin up a cluster with a bunch of regions on it. */ @BeforeClass public static void setUpBeforeClass() throws Exception { - TEST_UTIL.startMiniCluster(5); + TEST_UTIL.startMiniCluster(NB_SLAVES); TEST_UTIL.getConfiguration().setBoolean("hbase.master.enabletable.roundrobin", true); + ZooKeeperWatcher zkw = HBaseTestingUtility.getZooKeeperWatcher(TEST_UTIL); HTableDescriptor htd = new HTableDescriptor(TABLENAME); htd.addFamily(new HColumnDescriptor(FAMILY)); TEST_UTIL.createMultiRegionsInMeta(TEST_UTIL.getConfiguration(), htd, @@ -71,13 +74,25 @@ public static void setUpBeforeClass() throws Exception { createTableDescriptor(fs, FSUtils.getRootDir(TEST_UTIL.getConfiguration()), htd); // Assign out the regions we just created. HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + MiniHBaseCluster cluster = TEST_UTIL.getMiniHBaseCluster(); admin.disableTable(TABLENAME); admin.enableTable(TABLENAME); - // Assert that every regionserver has some regions on it. - MiniHBaseCluster cluster = TEST_UTIL.getMiniHBaseCluster(); - for (int i = 0; i < cluster.getRegionServerThreads().size(); i++) { - HRegionServer hrs = cluster.getRegionServer(i); - Assert.assertFalse(hrs.getOnlineRegions().isEmpty()); + boolean ready = false; + while (!ready) { + ZKAssign.blockUntilNoRIT(zkw); + // Assert that every regionserver has some regions on it, else invoke the balancer. + ready = true; + for (int i = 0; i < NB_SLAVES; i++) { + HRegionServer hrs = cluster.getRegionServer(i); + if (hrs.getOnlineRegions().isEmpty()) { + ready = false; + break; + } + } + if (!ready) { + admin.balancer(); + Thread.sleep(100); + } } } diff --git a/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java b/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java index 6bc7c32b3439..d7fff04726f8 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java +++ b/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/TestFullLogReconstruction.java b/src/test/java/org/apache/hadoop/hbase/TestFullLogReconstruction.java index e725d434577d..4c379b4c7631 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestFullLogReconstruction.java +++ b/src/test/java/org/apache/hadoop/hbase/TestFullLogReconstruction.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java b/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java index ad77e0a12ace..7615479523a1 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java +++ b/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java b/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java new file mode 100644 index 000000000000..6fac0245116e --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java @@ -0,0 +1,229 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import static org.junit.Assert.*; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.net.URI; +import java.util.UUID; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataInputStream; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.ManualEnvironmentEdge; +import org.apache.hadoop.util.Progressable; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(MediumTests.class) +public class TestHBaseFileSystem { + public static final Log LOG = LogFactory.getLog(TestHBaseFileSystem.class); + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static Configuration conf; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + conf = TEST_UTIL.getConfiguration(); + conf.setBoolean("dfs.support.append", true); + // The below config supported by 0.20-append and CDH3b2 + conf.setInt("dfs.client.block.recovery.retries", 2); + TEST_UTIL.startMiniDFSCluster(3); + Path hbaseRootDir = + TEST_UTIL.getDFSCluster().getFileSystem().makeQualified(new Path("/hbase")); + LOG.info("hbase.rootdir=" + hbaseRootDir); + conf.set(HConstants.HBASE_DIR, hbaseRootDir.toString()); + conf.setInt("hdfs.client.retries.number", 10); + HBaseFileSystem.setRetryCounts(conf); + } + + + @Test + public void testNonIdempotentOpsWithRetries() throws IOException { + LOG.info("testNonIdempotentOpsWithRetries"); + + Path rootDir = new Path(TestHBaseFileSystem.conf.get(HConstants.HBASE_DIR)); + FileSystem fs = TEST_UTIL.getTestFileSystem(); + // Create a Region + assertTrue(HBaseFileSystem.createPathOnFileSystem(fs, rootDir, + true) != null); + + boolean result = HBaseFileSystem.makeDirOnFileSystem(new MockFileSystemForCreate(), + new Path("/a")); + assertTrue("Couldn't create the directory", result); + + try { + HBaseFileSystem.createPathOnFileSystem(new MockFileSystemForCreate(), + new Path("/A"), false); + assertTrue(false);// control should not come here. + } catch (Exception e) { + LOG.info(e); + } + + result = HBaseFileSystem.renameDirForFileSystem(new MockFileSystem(), new Path("/a"), + new Path("/b")); + assertTrue("Couldn't rename the directory", result); + + result = HBaseFileSystem.deleteDirFromFileSystem(new MockFileSystem(), + new Path("/a")); + + assertTrue("Couldn't delete the directory", result); + fs.delete(rootDir, true); + } + + @Test + public void testRenameAndSetModifyTime() throws Exception { + assertTrue(FSUtils.isHDFS(conf)); + + FileSystem fs = FileSystem.get(conf); + Path testDir = TEST_UTIL.getDataTestDir("testArchiveFile"); + + String file = UUID.randomUUID().toString(); + Path p = new Path(testDir, file); + + FSDataOutputStream out = fs.create(p); + out.close(); + assertTrue("The created file should be present", FSUtils.isExists(fs, p)); + + long expect = System.currentTimeMillis() + 1000; + assertFalse(expect == fs.getFileStatus(p).getModificationTime()); + + ManualEnvironmentEdge mockEnv = new ManualEnvironmentEdge(); + mockEnv.setValue(expect); + EnvironmentEdgeManager.injectEdge(mockEnv); + + String dstFile = UUID.randomUUID().toString(); + Path dst = new Path(testDir , dstFile); + + assertTrue(HBaseFileSystem.renameAndSetModifyTime(fs, p, dst)); + assertFalse("The moved file should not be present", FSUtils.isExists(fs, + p)); + assertTrue("The dst file should be present", FSUtils.isExists(fs, dst)); + + assertEquals(expect, fs.getFileStatus(dst).getModificationTime()); + } + + + static class MockFileSystemForCreate extends MockFileSystem { + @Override + public boolean exists(Path path) { + if ("/A".equals(path.toString())) return true; + return false; + } + } + + /** + * a mock fs which throws exception for first 3 times, and then process the call (returns the + * excepted result). + */ + static class MockFileSystem extends FileSystem { + int retryCount; + final static int successRetryCount = 3; + + public MockFileSystem() { + retryCount = 0; + } + + @Override + public FSDataOutputStream append(Path arg0, int arg1, Progressable arg2) throws IOException { + throw new IOException(""); + } + + @Override + public FSDataOutputStream create(Path arg0, FsPermission arg1, boolean arg2, int arg3, + short arg4, long arg5, Progressable arg6) throws IOException { + LOG.debug("Create, " + retryCount); + if (retryCount++ < successRetryCount) throw new IOException("Something bad happen"); + return null; + } + + @Override + public boolean delete(Path arg0) throws IOException { + if (retryCount++ < successRetryCount) throw new IOException("Something bad happen"); + return true; + } + + @Override + public boolean delete(Path arg0, boolean arg1) throws IOException { + if (retryCount++ < successRetryCount) throw new IOException("Something bad happen"); + return true; + } + + @Override + public FileStatus getFileStatus(Path arg0) throws IOException { + FileStatus fs = new FileStatus(); + return fs; + } + + @Override + public boolean exists(Path path) { + return true; + } + + @Override + public URI getUri() { + throw new RuntimeException("Something bad happen"); + } + + @Override + public Path getWorkingDirectory() { + throw new RuntimeException("Something bad happen"); + } + + @Override + public FileStatus[] listStatus(Path arg0) throws IOException { + throw new IOException("Something bad happen"); + } + + @Override + public boolean mkdirs(Path arg0, FsPermission arg1) throws IOException { + LOG.debug("mkdirs, " + retryCount); + if (retryCount++ < successRetryCount) throw new IOException("Something bad happen"); + return true; + } + + @Override + public FSDataInputStream open(Path arg0, int arg1) throws IOException { + throw new IOException("Something bad happen"); + } + + @Override + public boolean rename(Path arg0, Path arg1) throws IOException { + LOG.debug("rename, " + retryCount); + if (retryCount++ < successRetryCount) throw new IOException("Something bad happen"); + return true; + } + + @Override + public void setWorkingDirectory(Path arg0) { + throw new RuntimeException("Something bad happen"); + } + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java b/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java index 3ed5d52e008d..dad29883a820 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java +++ b/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java b/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java new file mode 100644 index 000000000000..ea694067c782 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java @@ -0,0 +1,69 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import java.util.HashMap; +import java.util.Map; + +import static junit.framework.Assert.assertEquals; + +@Category(SmallTests.class) +public class TestHDFSBlocksDistribution { + @Test + public void testAddHostsAndBlockWeight() throws Exception { + HDFSBlocksDistribution distribution = new HDFSBlocksDistribution(); + distribution.addHostsAndBlockWeight(null, 100); + assertEquals("Expecting no hosts weights", 0, distribution.getHostAndWeights().size()); + distribution.addHostsAndBlockWeight(new String[0], 100); + assertEquals("Expecting no hosts weights", 0, distribution.getHostAndWeights().size()); + distribution.addHostsAndBlockWeight(new String[] {"test"}, 101); + assertEquals("Should be one host", 1, distribution.getHostAndWeights().size()); + distribution.addHostsAndBlockWeight(new String[] {"test"}, 202); + assertEquals("Should be one host", 1, distribution.getHostAndWeights().size()); + assertEquals("test host should have weight 303", 303, + distribution.getHostAndWeights().get("test").getWeight()); + distribution.addHostsAndBlockWeight(new String[] {"testTwo"}, 222); + assertEquals("Should be two hosts", 2, distribution.getHostAndWeights().size()); + assertEquals("Total weight should be 525", 525, distribution.getUniqueBlocksTotalWeight()); + } + + public class MockHDFSBlocksDistribution extends HDFSBlocksDistribution { + public Map getHostAndWeights() { + HashMap map = new HashMap(); + map.put("test", new HostAndWeight(null, 100)); + return map; + } + + } + + @Test + public void testAdd() throws Exception { + HDFSBlocksDistribution distribution = new HDFSBlocksDistribution(); + distribution.add(new MockHDFSBlocksDistribution()); + assertEquals("Expecting no hosts weights", 0, distribution.getHostAndWeights().size()); + distribution.addHostsAndBlockWeight(new String[]{"test"}, 10); + assertEquals("Should be one host", 1, distribution.getHostAndWeights().size()); + distribution.add(new MockHDFSBlocksDistribution()); + assertEquals("Should be one host", 1, distribution.getHostAndWeights().size()); + assertEquals("Total weight should be 10", 10, distribution.getUniqueBlocksTotalWeight()); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/TestHRegionLocation.java b/src/test/java/org/apache/hadoop/hbase/TestHRegionLocation.java index a9b8f9458be4..98f070e9cfe4 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestHRegionLocation.java +++ b/src/test/java/org/apache/hadoop/hbase/TestHRegionLocation.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/TestHServerAddress.java b/src/test/java/org/apache/hadoop/hbase/TestHServerAddress.java index 1a6933752cf3..a86a52a4ad44 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestHServerAddress.java +++ b/src/test/java/org/apache/hadoop/hbase/TestHServerAddress.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java b/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java index f7c0ccaec0a2..586b7af50c6d 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java +++ b/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java @@ -20,8 +20,15 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.util.regex.Pattern; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver; +import org.apache.hadoop.hbase.coprocessor.SampleRegionWALObserver; +import org.apache.hadoop.hbase.util.Bytes; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -30,6 +37,7 @@ */ @Category(SmallTests.class) public class TestHTableDescriptor { + final static Log LOG = LogFactory.getLog(TestHTableDescriptor.class); /** * Test cps in the table description @@ -48,6 +56,43 @@ public void testGetSetRemoveCP() throws Exception { assertFalse(desc.hasCoprocessor(className)); } + /** + * Test cps in the table description + * @throws Exception + */ + @Test + public void testSetListRemoveCP() throws Exception { + HTableDescriptor desc = new HTableDescriptor("testGetSetRemoveCP"); + // simple CP + String className1 = BaseRegionObserver.class.getName(); + String className2 = SampleRegionWALObserver.class.getName(); + // Check that any coprocessor is present. + assertTrue(desc.getCoprocessors().size() == 0); + + // Add the 1 coprocessor and check if present. + desc.addCoprocessor(className1); + assertTrue(desc.getCoprocessors().size() == 1); + assertTrue(desc.getCoprocessors().contains(className1)); + + // Add the 2nd coprocessor and check if present. + // remove it and check that it is gone + desc.addCoprocessor(className2); + assertTrue(desc.getCoprocessors().size() == 2); + assertTrue(desc.getCoprocessors().contains(className2)); + + // Remove one and check + desc.removeCoprocessor(className1); + assertTrue(desc.getCoprocessors().size() == 1); + assertFalse(desc.getCoprocessors().contains(className1)); + assertTrue(desc.getCoprocessors().contains(className2)); + + // Remove the last and check + desc.removeCoprocessor(className2); + assertTrue(desc.getCoprocessors().size() == 0); + assertFalse(desc.getCoprocessors().contains(className1)); + assertFalse(desc.getCoprocessors().contains(className2)); + } + /** * Test that we add and remove strings from settings properly. * @throws Exception @@ -63,4 +108,63 @@ public void testRemoveString() throws Exception { assertEquals(null, desc.getValue(key)); } + /** + * Test default value handling for maxFileSize + */ + @Test + public void testGetMaxFileSize() { + HTableDescriptor desc = new HTableDescriptor("table"); + assertEquals(-1, desc.getMaxFileSize()); + desc.setMaxFileSize(1111L); + assertEquals(1111L, desc.getMaxFileSize()); + } + + /** + * Test default value handling for memStoreFlushSize + */ + @Test + public void testGetMemStoreFlushSize() { + HTableDescriptor desc = new HTableDescriptor("table"); + assertEquals(-1, desc.getMemStoreFlushSize()); + desc.setMemStoreFlushSize(1111L); + assertEquals(1111L, desc.getMemStoreFlushSize()); + } + + String legalTableNames[] = { "foo", "with-dash_under.dot", "_under_start_ok", }; + String illegalTableNames[] = { ".dot_start_illegal", "-dash_start_illegal", "spaces not ok" }; + + @Test + public void testLegalHTableNames() { + for (String tn : legalTableNames) { + HTableDescriptor.isLegalTableName(Bytes.toBytes(tn)); + } + } + + @Test + public void testIllegalHTableNames() { + for (String tn : illegalTableNames) { + try { + HTableDescriptor.isLegalTableName(Bytes.toBytes(tn)); + fail("invalid tablename " + tn + " should have failed"); + } catch (Exception e) { + // expected + } + } + } + + @Test + public void testLegalHTableNamesRegex() { + for (String tn : legalTableNames) { + LOG.info("Testing: '" + tn + "'"); + assertTrue(Pattern.matches(HTableDescriptor.VALID_USER_TABLE_REGEX, tn)); + } + } + + @Test + public void testIllegalHTableNamesRegex() { + for (String tn : illegalTableNames) { + LOG.info("Testing: '" + tn + "'"); + assertFalse(Pattern.matches(HTableDescriptor.VALID_USER_TABLE_REGEX, tn)); + } + } } diff --git a/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java b/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java index adb5365cbeb3..f257a605fbd5 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java +++ b/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -26,6 +25,7 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.util.Bytes; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -48,6 +48,7 @@ public static void beforeClass() throws Exception { // Set them to ephemeral ports so they will start UTIL.getConfiguration().setInt("hbase.master.info.port", 0); UTIL.getConfiguration().setInt("hbase.regionserver.info.port", 0); + UTIL.getConfiguration().setBoolean("hbase.master.ui.readonly", true); UTIL.startMiniCluster(); } @@ -64,12 +65,12 @@ public void testInfoServersRedirect() throws Exception { // give the cluster time to start up new HTable(UTIL.getConfiguration(), ".META.").close(); int port = UTIL.getHBaseCluster().getMaster().getInfoServer().getPort(); - assertHasExpectedContent(new URL("http://localhost:" + port + - "/index.html"), "master-status"); + assertContainsContent(new URL("http://localhost:" + port + + "/index.html"), "master-status"); port = UTIL.getHBaseCluster().getRegionServerThreads().get(0).getRegionServer(). getInfoServer().getPort(); - assertHasExpectedContent(new URL("http://localhost:" + port + - "/index.html"), "rs-status"); + assertContainsContent(new URL("http://localhost:" + port + + "/index.html"), "rs-status"); } /** @@ -84,17 +85,49 @@ public void testInfoServersStatusPages() throws Exception { // give the cluster time to start up new HTable(UTIL.getConfiguration(), ".META.").close(); int port = UTIL.getHBaseCluster().getMaster().getInfoServer().getPort(); - assertHasExpectedContent(new URL("http://localhost:" + port + - "/master-status"), "META"); + assertContainsContent(new URL("http://localhost:" + port + + "/master-status"), "META"); port = UTIL.getHBaseCluster().getRegionServerThreads().get(0).getRegionServer(). getInfoServer().getPort(); - assertHasExpectedContent(new URL("http://localhost:" + port + - "/rs-status"), "META"); + assertContainsContent(new URL("http://localhost:" + port + + "/rs-status"), "META"); } - private void assertHasExpectedContent(final URL u, final String expected) + @Test + public void testMasterServerReadOnly() throws Exception { + String sTableName = "testMasterServerReadOnly"; + byte[] tableName = Bytes.toBytes(sTableName); + byte[] cf = Bytes.toBytes("d"); + UTIL.createTable(tableName, cf); + new HTable(UTIL.getConfiguration(), tableName).close(); + int port = UTIL.getHBaseCluster().getMaster().getInfoServer().getPort(); + assertDoesNotContainContent( + new URL("http://localhost:" + port + "/table.jsp?name=" + sTableName + "&action=split&key="), + "Table action request accepted"); + assertDoesNotContainContent( + new URL("http://localhost:" + port + "/table.jsp?name=" + sTableName), + "Actions:"); + } + + private void assertContainsContent(final URL u, final String expected) throws IOException { LOG.info("Testing " + u.toString() + " has " + expected); + String content = getUrlContent(u); + assertTrue("expected=" + expected + ", content=" + content, + content.contains(expected)); + } + + + + private void assertDoesNotContainContent(final URL u, final String expected) + throws IOException { + LOG.info("Testing " + u.toString() + " has " + expected); + String content = getUrlContent(u); + assertTrue("Does Not Contain =" + expected + ", content=" + content, + !content.contains(expected)); + } + + private String getUrlContent(URL u) throws IOException { java.net.URLConnection c = u.openConnection(); c.connect(); StringBuilder sb = new StringBuilder(); @@ -105,8 +138,7 @@ private void assertHasExpectedContent(final URL u, final String expected) } bis.close(); String content = sb.toString(); - assertTrue("expected=" + expected + ", content=" + content, - content.contains(expected)); + return content; } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java b/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java index f97d2bab3df0..e92d0a63c52b 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java +++ b/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,6 +18,8 @@ */ package org.apache.hadoop.hbase; +import java.io.ByteArrayInputStream; +import java.io.DataInputStream; import java.io.IOException; import java.util.Set; import java.util.TreeSet; @@ -31,6 +32,7 @@ import org.apache.hadoop.hbase.KeyValue.MetaComparator; import org.apache.hadoop.hbase.KeyValue.Type; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.io.WritableUtils; import org.junit.experimental.categories.Category; @Category(SmallTests.class) @@ -338,6 +340,65 @@ private void assertKVLess(KeyValue.KVComparator c, assertTrue(cmp > 0); } + private void assertKVLessWithoutRow(KeyValue.KeyComparator c, int common, KeyValue less, + KeyValue greater) { + int cmp = c.compareIgnoringPrefix(common, less.getBuffer(), less.getOffset() + + KeyValue.ROW_OFFSET, less.getKeyLength(), greater.getBuffer(), + greater.getOffset() + KeyValue.ROW_OFFSET, greater.getKeyLength()); + assertTrue(cmp < 0); + cmp = c.compareIgnoringPrefix(common, greater.getBuffer(), greater.getOffset() + + KeyValue.ROW_OFFSET, greater.getKeyLength(), less.getBuffer(), + less.getOffset() + KeyValue.ROW_OFFSET, less.getKeyLength()); + assertTrue(cmp > 0); + } + + public void testCompareWithoutRow() { + final KeyValue.KeyComparator c = KeyValue.KEY_COMPARATOR; + byte[] row = Bytes.toBytes("row"); + + byte[] fa = Bytes.toBytes("fa"); + byte[] fami = Bytes.toBytes("fami"); + byte[] fami1 = Bytes.toBytes("fami1"); + + byte[] qual0 = Bytes.toBytes(""); + byte[] qual1 = Bytes.toBytes("qf1"); + byte[] qual2 = Bytes.toBytes("qf2"); + long ts = 1; + + // 'fa:' + KeyValue kv_0 = new KeyValue(row, fa, qual0, ts, Type.Put); + // 'fami:' + KeyValue kv0_0 = new KeyValue(row, fami, qual0, ts, Type.Put); + // 'fami:qf1' + KeyValue kv0_1 = new KeyValue(row, fami, qual1, ts, Type.Put); + // 'fami:qf2' + KeyValue kv0_2 = new KeyValue(row, fami, qual2, ts, Type.Put); + // 'fami1:' + KeyValue kv1_0 = new KeyValue(row, fami1, qual0, ts, Type.Put); + + // 'fami:qf1' < 'fami:qf2' + assertKVLessWithoutRow(c, 0, kv0_1, kv0_2); + // 'fami:qf1' < 'fami1:' + assertKVLessWithoutRow(c, 0, kv0_1, kv1_0); + + // Test comparison by skipping the same prefix bytes. + /*** + * KeyValue Format and commonLength: + * |_keyLen_|_valLen_|_rowLen_|_rowKey_|_famiLen_|_fami_|_Quali_|.... + * ------------------|-------commonLength--------|-------------- + */ + int commonLength = KeyValue.ROW_LENGTH_SIZE + KeyValue.FAMILY_LENGTH_SIZE + + row.length; + // 'fa:' < 'fami:'. They have commonPrefix + 2 same prefix bytes. + assertKVLessWithoutRow(c, commonLength + 2, kv_0, kv0_0); + // 'fami:' < 'fami:qf1'. They have commonPrefix + 4 same prefix bytes. + assertKVLessWithoutRow(c, commonLength + 4, kv0_0, kv0_1); + // 'fami:qf1' < 'fami1:'. They have commonPrefix + 4 same prefix bytes. + assertKVLessWithoutRow(c, commonLength + 4, kv0_1, kv1_0); + // 'fami:qf1' < 'fami:qf2'. They have commonPrefix + 6 same prefix bytes. + assertKVLessWithoutRow(c, commonLength + 6, kv0_1, kv0_2); + } + public void testFirstLastOnRow() { final KVComparator c = KeyValue.COMPARATOR; long ts = 1; @@ -404,7 +465,41 @@ public void testCreateKeyValueFromKey() { System.err.println("kv=" + kv); System.err.println("kvFromKey=" + kvFromKey); assertEquals(kvFromKey.toString(), - kv.toString().replaceAll("=[0-9]+$", "=0")); + kv.toString().replaceAll("=[0-9]+", "=0")); + } + + /** + * The row cache is cleared and re-read for the new value + * + * @throws IOException + */ + public void testReadFields() throws IOException { + KeyValue kv1 = new KeyValue(Bytes.toBytes("row1"), Bytes.toBytes("cf1"), + Bytes.toBytes("qualifier1"), 12345L, Bytes.toBytes("value1")); + kv1.getRow(); // set row cache of kv1 + KeyValue kv2 = new KeyValue(Bytes.toBytes("row2"), Bytes.toBytes("cf2"), + Bytes.toBytes("qualifier2"), 12345L, Bytes.toBytes("value2")); + kv1.readFields(new DataInputStream(new ByteArrayInputStream(WritableUtils + .toByteArray(kv2)))); + // check equality + assertEquals(kv1, kv2); + // check cache state (getRow() return the cached value if the cache is set) + assertTrue(Bytes.equals(kv1.getRow(), kv2.getRow())); + } + + /** + * Tests that getTimestamp() does always return the proper timestamp, even after updating it. + * See HBASE-6265. + */ + public void testGetTimestamp() { + KeyValue kv = new KeyValue(Bytes.toBytes("myRow"), Bytes.toBytes("myCF"), + Bytes.toBytes("myQualifier"), HConstants.LATEST_TIMESTAMP, + Bytes.toBytes("myValue")); + long time1 = kv.getTimestamp(); + kv.updateLatestStamp(Bytes.toBytes(12345L)); + long time2 = kv.getTimestamp(); + assertEquals(HConstants.LATEST_TIMESTAMP, time1); + assertEquals(12345L, time2); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/TestLocalHBaseCluster.java b/src/test/java/org/apache/hadoop/hbase/TestLocalHBaseCluster.java new file mode 100644 index 000000000000..8b1c01848df3 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/TestLocalHBaseCluster.java @@ -0,0 +1,130 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import static org.junit.Assert.fail; + +import java.io.IOException; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster; +import org.apache.zookeeper.KeeperException; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(MediumTests.class) +public class TestLocalHBaseCluster { + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + /** + * Check that we can start a local HBase cluster specifying a custom master + * and regionserver class and then cast back to those classes; also that + * the cluster will launch and terminate cleanly. See HBASE-6011. + */ + @Test + public void testLocalHBaseCluster() throws Exception { + Configuration conf = TEST_UTIL.getConfiguration(); + conf.set(HConstants.HBASE_DIR, TEST_UTIL.getDataTestDir("hbase.rootdir"). + makeQualified(TEST_UTIL.getTestFileSystem().getUri(), + TEST_UTIL.getTestFileSystem().getWorkingDirectory()).toString()); + MiniZooKeeperCluster zkCluster = TEST_UTIL.startMiniZKCluster(); + conf.set(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(zkCluster.getClientPort())); + LocalHBaseCluster cluster = new LocalHBaseCluster(conf, 1, 1, MyHMaster.class, + MyHRegionServer.class); + // Can we cast back to our master class? + try { + ((MyHMaster)cluster.getMaster(0)).setZKCluster(zkCluster); + } catch (ClassCastException e) { + fail("Could not cast master to our class"); + } + // Can we cast back to our regionserver class? + try { + ((MyHRegionServer)cluster.getRegionServer(0)).echo(42); + } catch (ClassCastException e) { + fail("Could not cast regionserver to our class"); + } + // Does the cluster start successfully? + try { + cluster.startup(); + waitForClusterUp(conf); + } catch (IOException e) { + fail("LocalHBaseCluster did not start successfully"); + } finally { + cluster.shutdown(); + } + } + + private void waitForClusterUp(Configuration conf) throws IOException { + HTable t = new HTable(conf, HConstants.META_TABLE_NAME); + ResultScanner s = t.getScanner(new Scan()); + while (s.next() != null) { + continue; + } + s.close(); + t.close(); + } + + /** + * A private master class similar to that used by HMasterCommandLine when + * running in local mode. + */ + public static class MyHMaster extends HMaster { + private MiniZooKeeperCluster zkcluster = null; + + public MyHMaster(Configuration conf) throws IOException, KeeperException, + InterruptedException { + super(conf); + } + + @Override + public void run() { + super.run(); + if (this.zkcluster != null) { + try { + this.zkcluster.shutdown(); + } catch (IOException e) { + e.printStackTrace(); + } + } + } + + void setZKCluster(final MiniZooKeeperCluster zkcluster) { + this.zkcluster = zkcluster; + } + } + + /** + * A private regionserver class with a dummy method for testing casts + */ + public static class MyHRegionServer extends HRegionServer { + + public MyHRegionServer(Configuration conf) throws IOException, + InterruptedException { + super(conf); + } + + public int echo(int val) { + return val; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/TestNodeHealthCheckChore.java b/src/test/java/org/apache/hadoop/hbase/TestNodeHealthCheckChore.java new file mode 100644 index 000000000000..9ae974e33fd4 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/TestNodeHealthCheckChore.java @@ -0,0 +1,140 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertEquals; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.PrintWriter; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.Stoppable; +import org.apache.hadoop.hbase.HealthChecker.HealthCheckerExitStatus; +import org.junit.After; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(SmallTests.class) +public class TestNodeHealthCheckChore { + + private static final Log LOG = LogFactory.getLog(TestNodeHealthCheckChore.class); + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private File healthScriptFile; + + + @After + public void cleanUp() throws IOException { + UTIL.cleanupTestDir(); + } + + @Test + public void testHealthChecker() throws Exception { + Configuration config = getConfForNodeHealthScript(); + config.addResource(healthScriptFile.getName()); + String location = healthScriptFile.getAbsolutePath(); + long timeout = config.getLong(HConstants.HEALTH_SCRIPT_TIMEOUT, 200); + + HealthChecker checker = new HealthChecker(); + checker.init(location, timeout); + + String normalScript = "echo \"I am all fine\""; + createScript(normalScript, true); + HealthReport report = checker.checkHealth(); + assertEquals(HealthCheckerExitStatus.SUCCESS, report.getStatus()); + + LOG.info("Health Status:" + checker); + + String errorScript = "echo ERROR\n echo \"Node not healthy\""; + createScript(errorScript, true); + report = checker.checkHealth(); + assertEquals(HealthCheckerExitStatus.FAILED, report.getStatus()); + LOG.info("Health Status:" + report.getHealthReport()); + + String timeOutScript = "sleep 4\n echo\"I am fine\""; + createScript(timeOutScript, true); + report = checker.checkHealth(); + assertEquals(HealthCheckerExitStatus.TIMED_OUT, report.getStatus()); + LOG.info("Health Status:" + report.getHealthReport()); + + healthScriptFile.delete(); + } + + @Test + public void testNodeHealthChore() throws Exception{ + Stoppable stop = new StoppableImplementation(); + Configuration conf = getConfForNodeHealthScript(); + String errorScript = "echo ERROR\n echo \"Node not healthy\""; + createScript(errorScript, true); + HealthCheckChore rsChore = new HealthCheckChore(100, stop, conf); + //Default threshold is three. + rsChore.chore(); + rsChore.chore(); + assertFalse("Stoppable must not be stopped.", stop.isStopped()); + rsChore.chore(); + assertTrue("Stoppable must have been stopped.", stop.isStopped()); + } + + private void createScript(String scriptStr, boolean setExecutable) + throws Exception { + healthScriptFile.createNewFile(); + PrintWriter pw = new PrintWriter(new FileOutputStream(healthScriptFile)); + pw.println(scriptStr); + pw.flush(); + pw.close(); + healthScriptFile.setExecutable(setExecutable); + } + + private Configuration getConfForNodeHealthScript() { + Configuration conf = UTIL.getConfiguration(); + File tempDir = new File(UTIL.getDataTestDir().toString()); + tempDir.mkdirs(); + healthScriptFile = new File(tempDir.getAbsolutePath(), "HealthScript.sh"); + conf.set(HConstants.HEALTH_SCRIPT_LOC, + healthScriptFile.getAbsolutePath()); + conf.setLong(HConstants.HEALTH_FAILURE_THRESHOLD, 3); + conf.setLong(HConstants.HEALTH_SCRIPT_TIMEOUT, 200); + return conf; + } + + /** + * Simple helper class that just keeps track of whether or not its stopped. + */ + private static class StoppableImplementation implements Stoppable { + private volatile boolean stop = false; + + @Override + public void stop(String why) { + this.stop = true; + } + + @Override + public boolean isStopped() { + return this.stop; + } + + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java b/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java index e3b148f3bb9d..a5eba33f9679 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java +++ b/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java @@ -1,5 +1,4 @@ /** - * Copyright 2008 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -24,6 +23,7 @@ import java.io.IOException; import java.util.ArrayList; +import java.util.Arrays; import java.util.List; import org.apache.commons.logging.Log; @@ -78,7 +78,8 @@ public void before() { public void testRebalanceOnRegionServerNumberChange() throws IOException, InterruptedException { HBaseAdmin admin = new HBaseAdmin(UTIL.getConfiguration()); - admin.createTable(this.desc, HBaseTestingUtility.KEYS); + admin.createTable(this.desc, Arrays.copyOfRange(HBaseTestingUtility.KEYS, + 1, HBaseTestingUtility.KEYS.length)); this.table = new HTable(UTIL.getConfiguration(), this.desc.getName()); CatalogTracker ct = new CatalogTracker(UTIL.getConfiguration()); ct.start(); @@ -88,7 +89,7 @@ public void testRebalanceOnRegionServerNumberChange() ct.stop(); } assertEquals("Test table should have right number of regions", - HBaseTestingUtility.KEYS.length + 1/*One extra to account for start/end keys*/, + HBaseTestingUtility.KEYS.length, this.table.getStartKeys().length); // verify that the region assignments are balanced to start out @@ -216,9 +217,10 @@ private List getOnlineRegionServers() { * Wait until all the regions are assigned. */ private void waitForAllRegionsAssigned() throws IOException { - while (getRegionCount() < 22) { + int totalRegions = HBaseTestingUtility.KEYS.length+2; + while (getRegionCount() < totalRegions) { // while (!cluster.getMaster().allRegionsAssigned()) { - LOG.debug("Waiting for there to be 22 regions, but there are " + getRegionCount() + " right now."); + LOG.debug("Waiting for there to be "+ totalRegions +" regions, but there are " + getRegionCount() + " right now."); try { Thread.sleep(200); } catch (InterruptedException e) {} diff --git a/src/test/java/org/apache/hadoop/hbase/TestSerialization.java b/src/test/java/org/apache/hadoop/hbase/TestSerialization.java index 21c92fff72e9..977d59c28c75 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestSerialization.java +++ b/src/test/java/org/apache/hadoop/hbase/TestSerialization.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -22,14 +21,20 @@ import static org.junit.Assert.*; +import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; +import java.io.DataInputStream; import java.io.DataOutputStream; +import java.io.IOException; import java.util.ArrayList; +import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.NavigableSet; import java.util.Set; +import java.util.TreeMap; +import org.apache.hadoop.hbase.HServerLoad092.RegionLoad; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.Put; @@ -54,6 +59,17 @@ */ @Category(SmallTests.class) public class TestSerialization { + @Test + public void testHServerLoadVersioning() throws IOException { + Set cps = new HashSet(0); + Map regions = new TreeMap(Bytes.BYTES_COMPARATOR); + regions.put(HConstants.META_TABLE_NAME, + new HServerLoad092.RegionLoad(HConstants.META_TABLE_NAME, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, cps)); + HServerLoad092 hsl092 = new HServerLoad092(0, 0, 0, 0, regions, cps); + byte [] hsl092bytes = Writables.getBytes(hsl092); + HServerLoad hsl = (HServerLoad)Writables.getWritable(hsl092bytes, new HServerLoad()); + // TO BE CONTINUED + } @Test public void testCompareFilter() throws Exception { Filter f = new RowFilter(CompareOp.EQUAL, @@ -74,6 +90,51 @@ public class TestSerialization { assertTrue(KeyValue.COMPARATOR.compare(original, newone) == 0); } + @Test public void testCreateKeyValueInvalidNegativeLength() { + + KeyValue kv_0 = new KeyValue(Bytes.toBytes("myRow"), Bytes.toBytes("myCF"), // 51 bytes + Bytes.toBytes("myQualifier"), 12345L, Bytes.toBytes("my12345")); + + KeyValue kv_1 = new KeyValue(Bytes.toBytes("myRow"), Bytes.toBytes("myCF"), // 49 bytes + Bytes.toBytes("myQualifier"), 12345L, Bytes.toBytes("my123")); + + KeyValue kv_i = new KeyValue(); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + DataOutputStream dos = new DataOutputStream(baos); + + long l = 0; + try { + kv_0.write(dos); + l = dos.size(); + kv_1.write(dos); + l += dos.size(); + assertEquals(151L, l); + } catch (IOException e) { + fail("Unexpected IOException" + e.getMessage()); + } + + ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray()); + DataInputStream dis = new DataInputStream(bais); + + try { + kv_i.readFields(dis); + assertTrue(kv_0.equals(kv_1)); + } catch (Exception e) { + fail("Unexpected Exception" + e.getMessage()); + } + + // length -1 + try { + // even if we have a good kv now in dis we will just pass length with -1 for simplicity + kv_i.readFields(-1, dis); + fail("Expected corrupt stream"); + } catch (Exception e) { + assertEquals("Failed read -1 bytes, stream corrupt?", e.getMessage()); + } + + } + @SuppressWarnings("unchecked") @Test public void testHbaseMapWritable() throws Exception { HbaseMapWritable hmw = diff --git a/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java b/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java index d267824b467d..89537971ddf7 100644 --- a/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java +++ b/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -26,6 +25,12 @@ import static org.junit.Assert.fail; import java.io.IOException; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.SynchronousQueue; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -36,7 +41,9 @@ import org.apache.hadoop.hbase.client.HConnectionManager; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.master.HMaster; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.hbase.zookeeper.ZKConfig; import org.apache.hadoop.hbase.zookeeper.ZKUtil; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; @@ -58,12 +65,17 @@ public class TestZooKeeper { private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static HConnection persistentConnection; /** * @throws java.lang.Exception */ @BeforeClass public static void setUpBeforeClass() throws Exception { + // create a connection *before* the cluster is started, to validate that the + // connection's ZK trackers are initialized on demand + persistentConnection = HConnectionManager.createConnection(TEST_UTIL.getConfiguration()); + // Test we can first start the ZK cluster by itself TEST_UTIL.startMiniZKCluster(); TEST_UTIL.getConfiguration().setBoolean("dfs.support.append", true); @@ -75,6 +87,7 @@ public static void setUpBeforeClass() throws Exception { */ @AfterClass public static void tearDownAfterClass() throws Exception { + persistentConnection.close(); TEST_UTIL.shutdownMiniCluster(); } @@ -86,47 +99,45 @@ public void setUp() throws Exception { TEST_UTIL.ensureSomeRegionServersAvailable(2); } + private ZooKeeperWatcher getZooKeeperWatcher(HConnection c) throws + NoSuchMethodException, InvocationTargetException, IllegalAccessException { + + Method getterZK = c.getClass().getMethod("getKeepAliveZooKeeperWatcher"); + getterZK.setAccessible(true); + + return (ZooKeeperWatcher) getterZK.invoke(c); + } + /** * See HBASE-1232 and http://wiki.apache.org/hadoop/ZooKeeper/FAQ#4. * @throws IOException * @throws InterruptedException */ - @Test + // fails frequently, disabled for now, see HBASE-6406 + // @Test public void testClientSessionExpired() - throws IOException, InterruptedException { + throws Exception { LOG.info("testClientSessionExpired"); Configuration c = new Configuration(TEST_UTIL.getConfiguration()); new HTable(c, HConstants.META_TABLE_NAME).close(); - String quorumServers = ZKConfig.getZKQuorumServersString(c); - int sessionTimeout = 5 * 1000; // 5 seconds HConnection connection = HConnectionManager.getConnection(c); ZooKeeperWatcher connectionZK = connection.getZooKeeperWatcher(); - long sessionID = connectionZK.getRecoverableZooKeeper().getSessionId(); - byte[] password = connectionZK.getRecoverableZooKeeper().getSessionPasswd(); - ZooKeeper zk = new ZooKeeper(quorumServers, sessionTimeout, - EmptyWatcher.instance, sessionID, password); - LOG.info("Session timeout=" + zk.getSessionTimeout() + - ", original=" + sessionTimeout + - ", id=" + zk.getSessionId()); - zk.close(); - - Thread.sleep(sessionTimeout * 3L); + TEST_UTIL.expireSession(connectionZK, false); // provoke session expiration by doing something with ZK ZKUtil.dump(connectionZK); // Check that the old ZK connection is closed, means we did expire System.err.println("ZooKeeper should have timed out"); - String state = connectionZK.getRecoverableZooKeeper().getState().toString(); LOG.info("state=" + connectionZK.getRecoverableZooKeeper().getState()); Assert.assertTrue(connectionZK.getRecoverableZooKeeper().getState(). equals(States.CLOSED)); // Check that the client recovered ZooKeeperWatcher newConnectionZK = connection.getZooKeeperWatcher(); - LOG.info("state=" + newConnectionZK.getRecoverableZooKeeper().getState()); - Assert.assertTrue(newConnectionZK.getRecoverableZooKeeper().getState().equals( - States.CONNECTED)); + States state = newConnectionZK.getRecoverableZooKeeper().getState(); + LOG.info("state=" + state); + Assert.assertTrue(state.equals(States.CONNECTED) || state.equals(States.CONNECTING)); } @Test @@ -137,7 +148,9 @@ public void testRegionServerSessionExpired() throws Exception { testSanity(); } - @Test + // @Test Disabled because seems to make no sense expiring master session + // and then trying to create table (down in testSanity); on master side + // it will fail because the master's session has expired -- St.Ack 07/24/2012 public void testMasterSessionExpired() throws Exception { LOG.info("Starting testMasterSessionExpired"); TEST_UTIL.expireMasterSession(); @@ -148,21 +161,33 @@ public void testMasterSessionExpired() throws Exception { * Make sure we can use the cluster * @throws Exception */ - public void testSanity() throws Exception{ - HBaseAdmin admin = - new HBaseAdmin(new Configuration(TEST_UTIL.getConfiguration())); + private void testSanity() throws Exception { + String tableName = "test"+System.currentTimeMillis(); + HBaseAdmin admin = new HBaseAdmin(new Configuration(TEST_UTIL.getConfiguration())); + testAdminSanity(admin, tableName); + HTable table = new HTable(new Configuration(TEST_UTIL.getConfiguration()), tableName); + testTableSanity(table, tableName); + } + + private void testSanity(HConnection conn, ExecutorService pool) throws Exception { String tableName = "test"+System.currentTimeMillis(); + HBaseAdmin admin = new HBaseAdmin(persistentConnection); + testAdminSanity(admin, tableName); + HTable table = new HTable(Bytes.toBytes(tableName), persistentConnection, pool); + testTableSanity(table, tableName); + + } + private void testAdminSanity(HBaseAdmin admin, String tableName) throws Exception { HTableDescriptor desc = new HTableDescriptor(tableName); HColumnDescriptor family = new HColumnDescriptor("fam"); desc.addFamily(family); LOG.info("Creating table " + tableName); admin.createTable(desc); + } - HTable table = - new HTable(new Configuration(TEST_UTIL.getConfiguration()), tableName); + private void testTableSanity(HTable table, String tableName) throws Exception { Put put = new Put(Bytes.toBytes("testrow")); - put.add(Bytes.toBytes("fam"), - Bytes.toBytes("col"), Bytes.toBytes("testdata")); + put.add(Bytes.toBytes("fam"), Bytes.toBytes("col"), Bytes.toBytes("testdata")); LOG.info("Putting table " + tableName); table.put(put); table.close(); @@ -195,6 +220,27 @@ public void testMultipleZK() { } } + /** + * Create a znode with data + * @throws Exception + */ + @Test + public void testCreateWithParents() throws Exception { + ZooKeeperWatcher zkw = + new ZooKeeperWatcher(new Configuration(TEST_UTIL.getConfiguration()), + TestZooKeeper.class.getName(), null); + byte[] expectedData = new byte[] { 1, 2, 3 }; + ZKUtil.createWithParents(zkw, "/l1/l2/l3/l4/testCreateWithParents", expectedData); + byte[] data = ZKUtil.getData(zkw, "/l1/l2/l3/l4/testCreateWithParents"); + assertEquals(Bytes.equals(expectedData, data), true); + ZKUtil.deleteNodeRecursively(zkw, "/l1"); + + ZKUtil.createWithParents(zkw, "/testCreateWithParents", expectedData); + data = ZKUtil.getData(zkw, "/testCreateWithParents"); + assertEquals(Bytes.equals(expectedData, data), true); + ZKUtil.deleteNodeRecursively(zkw, "/testCreateWithParents"); + } + /** * Create a bunch of znodes in a hierarchy, try deleting one that has childs * (it will fail), then delete it recursively, then delete the last znode @@ -213,7 +259,12 @@ public void testZNodeDeletes() throws Exception { assertNotNull(ZKUtil.getDataNoWatch(zkw, "/l1/l2/l3/l4", null)); } ZKUtil.deleteNodeRecursively(zkw, "/l1/l2"); + // make sure it really is deleted assertNull(ZKUtil.getDataNoWatch(zkw, "/l1/l2/l3/l4", null)); + + // do the same delete again and make sure it doesn't crash + ZKUtil.deleteNodeRecursively(zkw, "/l1/l2"); + ZKUtil.deleteNode(zkw, "/l1"); assertNull(ZKUtil.getDataNoWatch(zkw, "/l1/l2", null)); } @@ -229,6 +280,16 @@ public void testClusterKey() throws Exception { } } + /** + * Test with a connection that existed before the cluster was started + */ + @Test + public void testPersistentConnection() throws Exception { + ExecutorService pool = new ThreadPoolExecutor(1, 10, 10, TimeUnit.SECONDS, + new SynchronousQueue()); + testSanity(persistentConnection, pool); + } + private void testKey(String ensemble, String port, String znode) throws IOException { Configuration conf = new Configuration(); @@ -266,16 +327,80 @@ public void testCreateSilentIsReallySilent() throws InterruptedException, // Assumes the root of the ZooKeeper space is writable as it creates a node // wherever the cluster home is defined. ZooKeeperWatcher zk2 = new ZooKeeperWatcher(TEST_UTIL.getConfiguration(), - "testMasterAddressManagerFromZK", - null); + "testMasterAddressManagerFromZK", null); // I set this acl after the attempted creation of the cluster home node. - zk.setACL("/", ZooDefs.Ids.CREATOR_ALL_ACL, -1); - zk.create(aclZnode, null, ZooDefs.Ids.CREATOR_ALL_ACL, CreateMode.PERSISTENT); - zk.close(); + // Add retries in case of retryable zk exceptions. + while (true) { + try { + zk.setACL("/", ZooDefs.Ids.CREATOR_ALL_ACL, -1); + break; + } catch (KeeperException e) { + switch (e.code()) { + case CONNECTIONLOSS: + case SESSIONEXPIRED: + case OPERATIONTIMEOUT: + LOG.warn("Possibly transient ZooKeeper exception: " + e); + Threads.sleep(100); + break; + default: + throw e; + } + } + } + while (true) { + try { + zk.create(aclZnode, null, ZooDefs.Ids.CREATOR_ALL_ACL, CreateMode.PERSISTENT); + break; + } catch (KeeperException e) { + switch (e.code()) { + case CONNECTIONLOSS: + case SESSIONEXPIRED: + case OPERATIONTIMEOUT: + LOG.warn("Possibly transient ZooKeeper exception: " + e); + Threads.sleep(100); + break; + default: + throw e; + } + } + } ZKUtil.createAndFailSilent(zk2, aclZnode); + + // reset /'s ACL for tests that follow + zk = new ZooKeeper(quorumServers, sessionTimeout, EmptyWatcher.instance); + zk.addAuthInfo("digest", "hbase:rox".getBytes()); + zk.setACL("/", ZooDefs.Ids.OPEN_ACL_UNSAFE, -1); + zk.close(); } + + /** + * Test should not fail with NPE when getChildDataAndWatchForNewChildren + * invoked with wrongNode + */ + @Test + public void testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE() + throws Exception { + ZooKeeperWatcher zkw = new ZooKeeperWatcher(TEST_UTIL.getConfiguration(), + "testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE", null); + ZKUtil.getChildDataAndWatchForNewChildren(zkw, "/wrongNode"); + } + + /** + * Master recovery when the znode already exists. Internally, this + * test differs from {@link #testMasterSessionExpired} because here + * the master znode will exist in ZK. + */ + @Test(timeout=60000) + public void testMasterZKSessionRecoveryFailure() throws Exception { + MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + HMaster m = cluster.getMaster(); + m.abort("Test recovery from zk session expired", + new KeeperException.SessionExpiredException()); + assertFalse(m.isStopped()); + testSanity(); + } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java b/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java new file mode 100644 index 000000000000..b5f277485902 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java @@ -0,0 +1,436 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.backup; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.PathFilter; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.Stoppable; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.master.cleaner.HFileCleaner; +import org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.HFileArchiveTestingUtil; +import org.apache.hadoop.hbase.util.HFileArchiveUtil; +import org.apache.hadoop.hbase.util.StoppableImplementation; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Assert; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test that the {@link HFileArchiver} correctly removes all the parts of a region when cleaning up + * a region + */ +@Category(MediumTests.class) +public class TestHFileArchiving { + + private static final Log LOG = LogFactory.getLog(TestHFileArchiving.class); + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static final byte[] TEST_FAM = Bytes.toBytes("fam"); + + /** + * Setup the config for the cluster + */ + @BeforeClass + public static void setupCluster() throws Exception { + setupConf(UTIL.getConfiguration()); + UTIL.startMiniCluster(); + + // We don't want the cleaner to remove files. The tests do that. + UTIL.getMiniHBaseCluster().getMaster().getHFileCleaner().interrupt(); + } + + private static void setupConf(Configuration conf) { + // disable the ui + conf.setInt("hbase.regionsever.info.port", -1); + // drop the memstore size so we get flushes + conf.setInt("hbase.hregion.memstore.flush.size", 25000); + // disable major compactions + conf.setInt(HConstants.MAJOR_COMPACTION_PERIOD, 0); + + // prevent aggressive region split + conf.set(HConstants.HBASE_REGION_SPLIT_POLICY_KEY, + ConstantSizeRegionSplitPolicy.class.getName()); + } + + @After + public void tearDown() throws Exception { + // cleanup the archive directory + try { + clearArchiveDirectory(); + } catch (IOException e) { + Assert.fail("Failure to delete archive directory:" + e.getMessage()); + } + } + + @AfterClass + public static void cleanupTest() throws Exception { + try { + UTIL.shutdownMiniCluster(); + } catch (Exception e) { + // NOOP; + } + } + + @Test + public void testRemovesRegionDirOnArchive() throws Exception { + byte[] TABLE_NAME = Bytes.toBytes("testRemovesRegionDirOnArchive"); + UTIL.createTable(TABLE_NAME, TEST_FAM); + + final HBaseAdmin admin = UTIL.getHBaseAdmin(); + + // get the current store files for the region + List servingRegions = UTIL.getHBaseCluster().getRegions(TABLE_NAME); + // make sure we only have 1 region serving this table + assertEquals(1, servingRegions.size()); + HRegion region = servingRegions.get(0); + + // and load the table + UTIL.loadRegion(region, TEST_FAM); + + // shutdown the table so we can manipulate the files + admin.disableTable(TABLE_NAME); + + FileSystem fs = UTIL.getTestFileSystem(); + + // now attempt to depose the region + Path regionDir = HRegion.getRegionDir(region.getTableDir().getParent(), region.getRegionInfo()); + + HFileArchiver.archiveRegion(UTIL.getConfiguration(), fs, region.getRegionInfo()); + + // check for the existence of the archive directory and some files in it + Path archiveDir = HFileArchiveTestingUtil.getRegionArchiveDir(UTIL.getConfiguration(), region); + assertTrue(fs.exists(archiveDir)); + + // check to make sure the store directory was copied + FileStatus[] stores = fs.listStatus(archiveDir); + assertTrue(stores.length == 1); + + // make sure we archived the store files + FileStatus[] storeFiles = fs.listStatus(stores[0].getPath()); + assertTrue(storeFiles.length > 0); + + // then ensure the region's directory isn't present + assertFalse(fs.exists(regionDir)); + + UTIL.deleteTable(TABLE_NAME); + } + + /** + * Test that the region directory is removed when we archive a region without store files, but + * still has hidden files. + * @throws Exception + */ + @Test + public void testDeleteRegionWithNoStoreFiles() throws Exception { + byte[] TABLE_NAME = Bytes.toBytes("testDeleteRegionWithNoStoreFiles"); + UTIL.createTable(TABLE_NAME, TEST_FAM); + + // get the current store files for the region + List servingRegions = UTIL.getHBaseCluster().getRegions(TABLE_NAME); + // make sure we only have 1 region serving this table + assertEquals(1, servingRegions.size()); + HRegion region = servingRegions.get(0); + + FileSystem fs = region.getFilesystem(); + + // make sure there are some files in the regiondir + Path rootDir = FSUtils.getRootDir(fs.getConf()); + Path regionDir = HRegion.getRegionDir(rootDir, region.getRegionInfo()); + FileStatus[] regionFiles = FSUtils.listStatus(fs, regionDir, null); + Assert.assertNotNull("No files in the region directory", regionFiles); + if (LOG.isDebugEnabled()) { + List files = new ArrayList(); + for (FileStatus file : regionFiles) { + files.add(file.getPath()); + } + LOG.debug("Current files:" + files); + } + // delete the visible folders so we just have hidden files/folders + final PathFilter dirFilter = new FSUtils.DirFilter(fs); + PathFilter nonHidden = new PathFilter() { + @Override + public boolean accept(Path file) { + return dirFilter.accept(file) && !file.getName().toString().startsWith("."); + } + }; + FileStatus[] storeDirs = FSUtils.listStatus(fs, regionDir, nonHidden); + for (FileStatus store : storeDirs) { + LOG.debug("Deleting store for test"); + fs.delete(store.getPath(), true); + } + + // then archive the region + HFileArchiver.archiveRegion(UTIL.getConfiguration(), fs, region.getRegionInfo()); + + // and check to make sure the region directoy got deleted + assertFalse("Region directory (" + regionDir + "), still exists.", fs.exists(regionDir)); + + UTIL.deleteTable(TABLE_NAME); + } + + @Test + public void testArchiveOnTableDelete() throws Exception { + byte[] TABLE_NAME = Bytes.toBytes("testArchiveOnTableDelete"); + UTIL.createTable(TABLE_NAME, TEST_FAM); + + List servingRegions = UTIL.getHBaseCluster().getRegions(TABLE_NAME); + // make sure we only have 1 region serving this table + assertEquals(1, servingRegions.size()); + HRegion region = servingRegions.get(0); + + // get the parent RS and monitor + HRegionServer hrs = UTIL.getRSForFirstRegionInTable(TABLE_NAME); + FileSystem fs = hrs.getFileSystem(); + + // put some data on the region + LOG.debug("-------Loading table"); + UTIL.loadRegion(region, TEST_FAM); + + // get the hfiles in the region + List regions = hrs.getOnlineRegions(TABLE_NAME); + assertEquals("More that 1 region for test table.", 1, regions.size()); + + region = regions.get(0); + // wait for all the compactions to complete + region.waitForFlushesAndCompactions(); + + // disable table to prevent new updates + UTIL.getHBaseAdmin().disableTable(TABLE_NAME); + LOG.debug("Disabled table"); + + // remove all the files from the archive to get a fair comparison + clearArchiveDirectory(); + + // then get the current store files + Path regionDir = region.getRegionDir(); + List storeFiles = getRegionStoreFiles(fs, regionDir); + + // then delete the table so the hfiles get archived + UTIL.deleteTable(TABLE_NAME); + + assertArchiveFiles(fs, storeFiles, 30000); + } + + private void assertArchiveFiles(FileSystem fs, List storeFiles, long timeout) throws IOException { + long end = System.currentTimeMillis() + timeout; + Path archiveDir = HFileArchiveUtil.getArchivePath(UTIL.getConfiguration()); + List archivedFiles = new ArrayList(); + + // We have to ensure that the DeleteTableHandler is finished. HBaseAdmin.deleteXXX() can return before all files + // are archived. We should fix HBASE-5487 and fix synchronous operations from admin. + while (System.currentTimeMillis() < end) { + archivedFiles = getAllFileNames(fs, archiveDir); + if (archivedFiles.size() >= storeFiles.size()) { + break; + } + } + + Collections.sort(storeFiles); + Collections.sort(archivedFiles); + + LOG.debug("Store files:"); + for (int i = 0; i < storeFiles.size(); i++) { + LOG.debug(i + " - " + storeFiles.get(i)); + } + LOG.debug("Archive files:"); + for (int i = 0; i < archivedFiles.size(); i++) { + LOG.debug(i + " - " + archivedFiles.get(i)); + } + + assertTrue("Archived files are missing some of the store files!", + archivedFiles.containsAll(storeFiles)); + } + + /** + * Test that the store files are archived when a column family is removed. + * @throws Exception + */ + @Test + public void testArchiveOnTableFamilyDelete() throws Exception { + byte[] TABLE_NAME = Bytes.toBytes("testArchiveOnTableFamilyDelete"); + UTIL.createTable(TABLE_NAME, TEST_FAM); + + List servingRegions = UTIL.getHBaseCluster().getRegions(TABLE_NAME); + // make sure we only have 1 region serving this table + assertEquals(1, servingRegions.size()); + HRegion region = servingRegions.get(0); + + // get the parent RS and monitor + HRegionServer hrs = UTIL.getRSForFirstRegionInTable(TABLE_NAME); + FileSystem fs = hrs.getFileSystem(); + + // put some data on the region + LOG.debug("-------Loading table"); + UTIL.loadRegion(region, TEST_FAM); + + // get the hfiles in the region + List regions = hrs.getOnlineRegions(TABLE_NAME); + assertEquals("More that 1 region for test table.", 1, regions.size()); + + region = regions.get(0); + // wait for all the compactions to complete + region.waitForFlushesAndCompactions(); + + // disable table to prevent new updates + UTIL.getHBaseAdmin().disableTable(TABLE_NAME); + LOG.debug("Disabled table"); + + // remove all the files from the archive to get a fair comparison + clearArchiveDirectory(); + + // then get the current store files + Path regionDir = region.getRegionDir(); + List storeFiles = getRegionStoreFiles(fs, regionDir); + + // then delete the table so the hfiles get archived + UTIL.getHBaseAdmin().deleteColumn(TABLE_NAME, TEST_FAM); + + assertArchiveFiles(fs, storeFiles, 30000); + + UTIL.deleteTable(TABLE_NAME); + } + + /** + * Test HFileArchiver.resolveAndArchive() race condition HBASE-7643 + */ + @Test + public void testCleaningRace() throws Exception { + final long TEST_TIME = 20 * 1000; + + Configuration conf = UTIL.getMiniHBaseCluster().getMaster().getConfiguration(); + Path rootDir = UTIL.getDataTestDir("testCleaningRace"); + FileSystem fs = UTIL.getTestFileSystem(); + + Path archiveDir = new Path(rootDir, HConstants.HFILE_ARCHIVE_DIRECTORY); + Path regionDir = new Path("table", "abcdef"); + Path familyDir = new Path(regionDir, "cf"); + + Path sourceRegionDir = new Path(rootDir, regionDir); + fs.mkdirs(sourceRegionDir); + + Stoppable stoppable = new StoppableImplementation(); + + // The cleaner should be looping without long pauses to reproduce the race condition. + HFileCleaner cleaner = new HFileCleaner(1, stoppable, conf, fs, archiveDir); + try { + cleaner.start(); + + // Keep creating/archiving new files while the cleaner is running in the other thread + long startTime = System.currentTimeMillis(); + for (long fid = 0; (System.currentTimeMillis() - startTime) < TEST_TIME; ++fid) { + Path file = new Path(familyDir, String.valueOf(fid)); + Path sourceFile = new Path(rootDir, file); + Path archiveFile = new Path(archiveDir, file); + + fs.createNewFile(sourceFile); + + try { + // Try to archive the file + HFileArchiver.archiveRegion(fs, rootDir, + sourceRegionDir.getParent(), sourceRegionDir); + + // The archiver succeded, the file is no longer in the original location + // but it's in the archive location. + LOG.debug("hfile=" + fid + " should be in the archive"); + assertTrue(fs.exists(archiveFile)); + assertFalse(fs.exists(sourceFile)); + } catch (IOException e) { + // The archiver is unable to archive the file. Probably HBASE-7643 race condition. + // in this case, the file should not be archived, and we should have the file + // in the original location. + LOG.debug("hfile=" + fid + " should be in the source location"); + assertFalse(fs.exists(archiveFile)); + assertTrue(fs.exists(sourceFile)); + + // Avoid to have this file in the next run + fs.delete(sourceFile, false); + } + } + } finally { + stoppable.stop("test end"); + cleaner.join(); + fs.delete(rootDir, true); + } + } + + private void clearArchiveDirectory() throws IOException { + UTIL.getTestFileSystem().delete(new Path(UTIL.getDefaultRootDirPath(), ".archive"), true); + } + + /** + * Get the names of all the files below the given directory + * @param fs + * @param archiveDir + * @return + * @throws IOException + */ + private List getAllFileNames(final FileSystem fs, Path archiveDir) throws IOException { + FileStatus[] files = FSUtils.listStatus(fs, archiveDir, null); + return recurseOnFiles(fs, files, new ArrayList()); + } + + /** Recursively lookup all the file names under the file[] array **/ + private List recurseOnFiles(FileSystem fs, FileStatus[] files, List fileNames) + throws IOException { + if (files == null || files.length == 0) return fileNames; + + for (FileStatus file : files) { + if (file.isDir()) { + recurseOnFiles(fs, FSUtils.listStatus(fs, file.getPath(), null), fileNames); + } else fileNames.add(file.getPath().getName()); + } + return fileNames; + } + + private List getRegionStoreFiles(final FileSystem fs, final Path regionDir) + throws IOException { + List storeFiles = getAllFileNames(fs, regionDir); + // remove all the non-storefile named files for the region + for (int i = 0; i < storeFiles.size(); i++) { + String file = storeFiles.get(i); + if (file.contains(HRegion.REGIONINFO_FILE) || file.contains("hlog")) { + storeFiles.remove(i--); + } + } + storeFiles.remove(HRegion.REGIONINFO_FILE); + return storeFiles; + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java b/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java index 90fa45ac68f4..a78a893c5047 100644 --- a/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java +++ b/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -25,6 +24,7 @@ import java.net.ConnectException; import java.util.ArrayList; import java.util.List; +import java.util.concurrent.CountDownLatch; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; @@ -48,6 +48,7 @@ import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.apache.hadoop.util.Progressable; import org.apache.zookeeper.KeeperException; +import org.apache.hadoop.hbase.ipc.ServerNotRunningYetException; import org.junit.After; import org.junit.AfterClass; import org.junit.Before; @@ -96,13 +97,24 @@ public boolean isAborted() { } @After public void after() { + try { + // Clean out root location or later tests will be confused... they presume + // start fresh in zk. + RootLocationEditor.deleteRootLocation(this.watcher); + } catch (KeeperException e) { + LOG.warn("Unable to delete root location", e); + } + + // Clear out our doctored connection or could mess up subsequent tests. + HConnectionManager.deleteConnection(UTIL.getConfiguration()); + this.watcher.close(); } private CatalogTracker constructAndStartCatalogTracker(final HConnection c) throws IOException, InterruptedException { CatalogTracker ct = new CatalogTracker(this.watcher, UTIL.getConfiguration(), - c, this.abortable, 0); + c, this.abortable); ct.start(); return ct; } @@ -117,14 +129,9 @@ private CatalogTracker constructAndStartCatalogTracker(final HConnection c) throws IOException, InterruptedException, KeeperException { HConnection connection = Mockito.mock(HConnection.class); constructAndStartCatalogTracker(connection); - try { - RootLocationEditor.setRootLocation(this.watcher, - new ServerName("example.com", 1234, System.currentTimeMillis())); - } finally { - // Clean out root location or later tests will be confused... they presume - // start fresh in zk. - RootLocationEditor.deleteRootLocation(this.watcher); - } + + RootLocationEditor.setRootLocation(this.watcher, + new ServerName("example.com", 1234, System.currentTimeMillis())); } /** @@ -136,33 +143,30 @@ private CatalogTracker constructAndStartCatalogTracker(final HConnection c) throws IOException, InterruptedException { HRegionInterface implementation = Mockito.mock(HRegionInterface.class); HConnection connection = mockConnection(implementation); - try { - final CatalogTracker ct = constructAndStartCatalogTracker(connection); - ServerName hsa = ct.getRootLocation(); - Assert.assertNull(hsa); - ServerName meta = ct.getMetaLocation(); - Assert.assertNull(meta); - Thread t = new Thread() { - @Override - public void run() { - try { - ct.waitForMeta(); - } catch (InterruptedException e) { - throw new RuntimeException("Interrupted", e); - } + + final CatalogTracker ct = constructAndStartCatalogTracker(connection); + ServerName hsa = ct.getRootLocation(); + Assert.assertNull(hsa); + ServerName meta = ct.getMetaLocation(); + Assert.assertNull(meta); + Thread t = new Thread() { + @Override + public void run() { + try { + ct.waitForMeta(); + } catch (InterruptedException e) { + throw new RuntimeException("Interrupted", e); } - }; - t.start(); - while (!t.isAlive()) - Threads.sleep(1); + } + }; + t.start(); + while (!t.isAlive()) Threads.sleep(1); - assertTrue(t.isAlive()); - ct.stop(); - // Join the thread... should exit shortly. - t.join(); - } finally { - HConnectionManager.deleteConnection(UTIL.getConfiguration(), true); - } + Threads.sleep(1); + assertTrue(t.isAlive()); + ct.stop(); + // Join the thread... should exit shortly. + t.join(); } /** @@ -179,66 +183,78 @@ public void testServerNotRunningIOException() // Mock an HRegionInterface. final HRegionInterface implementation = Mockito.mock(HRegionInterface.class); HConnection connection = mockConnection(implementation); + + // If a 'getRegionInfo' is called on mocked HRegionInterface, throw IOE + // the first time. 'Succeed' the second time we are called. + Mockito.when(implementation.getRegionInfo((byte[]) Mockito.any())). + thenThrow(new IOException("Server not running, aborting")). + thenReturn(new HRegionInfo()); + + // After we encounter the above 'Server not running', we should catch the + // IOE and go into retrying for the meta mode. We'll do gets on -ROOT- to + // get new meta location. Return something so this 'get' succeeds + // (here we mock up getRegionServerWithRetries, the wrapper around + // the actual get). + + // TODO: Refactor. This method has been moved out of HConnection. + // It works for now but has been deprecated. + Mockito.when(connection.getRegionServerWithRetries((ServerCallable)Mockito.any())). + thenReturn(getMetaTableRowResult()); + + // Now start up the catalogtracker with our doctored Connection. + final CatalogTracker ct = constructAndStartCatalogTracker(connection); try { - // If a 'getRegionInfo' is called on mocked HRegionInterface, throw IOE - // the first time. 'Succeed' the second time we are called. - Mockito.when(implementation.getRegionInfo((byte[]) Mockito.any())). - thenThrow(new IOException("Server not running, aborting")). - thenReturn(new HRegionInfo()); - - // After we encounter the above 'Server not running', we should catch the - // IOE and go into retrying for the meta mode. We'll do gets on -ROOT- to - // get new meta location. Return something so this 'get' succeeds - // (here we mock up getRegionServerWithRetries, the wrapper around - // the actual get). - - // TODO: Refactor. This method has been moved out of HConnection. - // It works for now but has been deprecated. - Mockito.when(connection.getRegionServerWithRetries((ServerCallable)Mockito.any())). - thenReturn(getMetaTableRowResult()); - - // Now start up the catalogtracker with our doctored Connection. - final CatalogTracker ct = constructAndStartCatalogTracker(connection); - try { - // Set a location for root and meta. - RootLocationEditor.setRootLocation(this.watcher, SN); - ct.setMetaLocation(SN); - // Call the method that HBASE-4288 calls. It will try and verify the - // meta location and will fail on first attempt then go into a long wait. - // So, do this in a thread and then reset meta location to break it out - // of its wait after a bit of time. - final AtomicBoolean metaSet = new AtomicBoolean(false); - Thread t = new Thread() { - @Override - public void run() { - try { - metaSet.set(ct.waitForMetaServerConnectionDefault() != null); - } catch (NotAllMetaRegionsOnlineException e) { - throw new RuntimeException(e); - } catch (IOException e) { - throw new RuntimeException(e); - } + // Set a location for root and meta. + RootLocationEditor.setRootLocation(this.watcher, SN); + ct.setMetaLocation(SN); + // Call the method that HBASE-4288 calls. It will try and verify the + // meta location and will fail on first attempt then go into a long wait. + // So, do this in a thread and then reset meta location to break it out + // of its wait after a bit of time. + final AtomicBoolean metaSet = new AtomicBoolean(false); + final CountDownLatch latch = new CountDownLatch(1); + Thread t = new Thread() { + @Override + public void run() { + try { + latch.countDown(); + metaSet.set(ct.waitForMeta(100000) != null); + } catch (Exception e) { + throw new RuntimeException(e); } - }; - t.start(); - while(!t.isAlive()) Threads.sleep(1); - Threads.sleep(1); - // Now reset the meta as though it were redeployed. - ct.setMetaLocation(SN); - t.join(); - Assert.assertTrue(metaSet.get()); - } finally { - // Clean out root and meta locations or later tests will be confused... - // they presume start fresh in zk. - ct.resetMetaLocation(); - RootLocationEditor.deleteRootLocation(this.watcher); - } + } + }; + t.start(); + latch.await(); + Threads.sleep(1); + // Now reset the meta as though it were redeployed. + ct.setMetaLocation(SN); + t.join(); + Assert.assertTrue(metaSet.get()); } finally { - // Clear out our doctored connection or could mess up subsequent tests. - HConnectionManager.deleteConnection(UTIL.getConfiguration(), true); + // Clean out root and meta locations or later tests will be confused... + // they presume start fresh in zk. + ct.resetMetaLocation(); } } + private void testVerifyMetaRegionLocationWithException(Exception ex) + throws IOException, InterruptedException, KeeperException { + // Mock an HRegionInterface. + final HRegionInterface implementation = Mockito.mock(HRegionInterface.class); + HConnection connection = mockConnection(implementation); + + // If a 'get' is called on mocked interface, throw connection refused. + Mockito.when(implementation.get((byte[]) Mockito.any(), (Get) Mockito.any())). + thenThrow(ex); + // Now start up the catalogtracker with our doctored Connection. + final CatalogTracker ct = constructAndStartCatalogTracker(connection); + RootLocationEditor.setRootLocation(this.watcher, SN); + long timeout = UTIL.getConfiguration(). + getLong("hbase.catalog.verification.timeout", 1000); + Assert.assertFalse(ct.verifyMetaRegionLocation(timeout)); + } + /** * Test we survive a connection refused {@link ConnectException} * @throws IOException @@ -248,29 +264,26 @@ public void run() { @Test public void testGetMetaServerConnectionFails() throws IOException, InterruptedException, KeeperException { - // Mock an HRegionInterface. - final HRegionInterface implementation = Mockito.mock(HRegionInterface.class); - HConnection connection = mockConnection(implementation); - try { - // If a 'get' is called on mocked interface, throw connection refused. - Mockito.when(implementation.get((byte[]) Mockito.any(), (Get) Mockito.any())). - thenThrow(new ConnectException("Connection refused")); - // Now start up the catalogtracker with our doctored Connection. - final CatalogTracker ct = constructAndStartCatalogTracker(connection); - try { - RootLocationEditor.setRootLocation(this.watcher, SN); - long timeout = UTIL.getConfiguration(). - getLong("hbase.catalog.verification.timeout", 1000); - Assert.assertFalse(ct.verifyMetaRegionLocation(timeout)); - } finally { - // Clean out root location or later tests will be confused... they - // presume start fresh in zk. - RootLocationEditor.deleteRootLocation(this.watcher); - } - } finally { - // Clear out our doctored connection or could mess up subsequent tests. - HConnectionManager.deleteConnection(UTIL.getConfiguration(), true); - } + testVerifyMetaRegionLocationWithException(new ConnectException("Connection refused")); + } + + /** + * Test that verifyMetaRegionLocation properly handles getting a + * ServerNotRunningException. See HBASE-4470. + * Note this doesn't check the exact exception thrown in the + * HBASE-4470 as there it is thrown from getHConnection() and + * here it is thrown from get() -- but those are both called + * from the same function anyway, and this way is less invasive than + * throwing from getHConnection would be. + * + * @throws IOException + * @throws InterruptedException + * @throws KeeperException + */ + @Test + public void testVerifyMetaRegionServerNotRunning() + throws IOException, InterruptedException, KeeperException { + testVerifyMetaRegionLocationWithException(new ServerNotRunningYetException("mock")); } /** @@ -293,15 +306,9 @@ public void testVerifyRootRegionLocationFails() Mockito.anyInt(), Mockito.anyBoolean())). thenReturn(implementation); final CatalogTracker ct = constructAndStartCatalogTracker(connection); - try { - RootLocationEditor.setRootLocation(this.watcher, - new ServerName("example.com", 1234, System.currentTimeMillis())); - Assert.assertFalse(ct.verifyRootRegionLocation(100)); - } finally { - // Clean out root location or later tests will be confused... they presume - // start fresh in zk. - RootLocationEditor.deleteRootLocation(this.watcher); - } + RootLocationEditor.setRootLocation(this.watcher, + new ServerName("example.com", 1234, System.currentTimeMillis())); + Assert.assertFalse(ct.verifyRootRegionLocation(100)); } @Test (expected = NotAllMetaRegionsOnlineException.class) @@ -317,12 +324,8 @@ public void testTimeoutWaitForMeta() throws IOException, InterruptedException { HConnection connection = HConnectionTestingUtility.getMockedConnection(UTIL.getConfiguration()); - try { - final CatalogTracker ct = constructAndStartCatalogTracker(connection); - ct.waitForMeta(100); - } finally { - HConnectionManager.deleteConnection(UTIL.getConfiguration(), true); - } + final CatalogTracker ct = constructAndStartCatalogTracker(connection); + ct.waitForMeta(100); } /** @@ -371,60 +374,57 @@ private ServerName setRootLocation() throws KeeperException { // Mock an HRegionInterface. final HRegionInterface implementation = Mockito.mock(HRegionInterface.class); HConnection connection = mockConnection(implementation); - try { - // Now the ct is up... set into the mocks some answers that make it look - // like things have been getting assigned. Make it so we'll return a - // location (no matter what the Get is). Same for getHRegionInfo -- always - // just return the meta region. - final Result result = getMetaTableRowResult(); - - // TODO: Refactor. This method has been moved out of HConnection. - // It works for now but has been deprecated. - Mockito.when(connection.getRegionServerWithRetries((ServerCallable)Mockito.any())). - thenReturn(result); - Mockito.when(implementation.getRegionInfo((byte[]) Mockito.any())). - thenReturn(HRegionInfo.FIRST_META_REGIONINFO); - final CatalogTracker ct = constructAndStartCatalogTracker(connection); - ServerName hsa = ct.getMetaLocation(); - Assert.assertNull(hsa); - - // Now test waiting on meta location getting set. - Thread t = new WaitOnMetaThread(ct) { - @Override - void doWaiting() throws InterruptedException { - this.ct.waitForMeta(); - } - }; - startWaitAliveThenWaitItLives(t, 1000); - - // This should trigger wake up of meta wait (Its the removal of the meta - // region unassigned node that triggers catalogtrackers that a meta has - // been assigned). - String node = ct.getMetaNodeTracker().getNode(); - ZKUtil.createAndFailSilent(this.watcher, node); - MetaEditor.updateMetaLocation(ct, HRegionInfo.FIRST_META_REGIONINFO, SN); - ZKUtil.deleteNode(this.watcher, node); - // Go get the new meta location. waitForMeta gets and verifies meta. - Assert.assertTrue(ct.waitForMeta(10000).equals(SN)); - // Join the thread... should exit shortly. - t.join(); - // Now meta is available. - Assert.assertTrue(ct.waitForMeta(10000).equals(SN)); - } finally { - HConnectionManager.deleteConnection(UTIL.getConfiguration(), true); - } + + // Now the ct is up... set into the mocks some answers that make it look + // like things have been getting assigned. Make it so we'll return a + // location (no matter what the Get is). Same for getHRegionInfo -- always + // just return the meta region. + final Result result = getMetaTableRowResult(); + + // TODO: Refactor. This method has been moved out of HConnection. + // It works for now but has been deprecated. + Mockito.when(connection.getRegionServerWithRetries((ServerCallable)Mockito.any())). + thenReturn(result); + Mockito.when(implementation.getRegionInfo((byte[]) Mockito.any())). + thenReturn(HRegionInfo.FIRST_META_REGIONINFO); + final CatalogTracker ct = constructAndStartCatalogTracker(connection); + ServerName hsa = ct.getMetaLocation(); + Assert.assertNull(hsa); + + // Now test waiting on meta location getting set. + Thread t = new WaitOnMetaThread(ct) { + @Override + void doWaiting() throws InterruptedException { + this.ct.waitForMeta(); + } + }; + startWaitAliveThenWaitItLives(t, 1000); + + // This should trigger wake up of meta wait (Its the removal of the meta + // region unassigned node that triggers catalogtrackers that a meta has + // been assigned). + String node = ct.getMetaNodeTracker().getNode(); + ZKUtil.createAndFailSilent(this.watcher, node); + MetaEditor.updateMetaLocation(ct, HRegionInfo.FIRST_META_REGIONINFO, SN); + ZKUtil.deleteNode(this.watcher, node); + // Go get the new meta location. waitForMeta gets and verifies meta. + Assert.assertTrue(ct.waitForMeta(10000).equals(SN)); + // Join the thread... should exit shortly. + t.join(); + // Now meta is available. + Assert.assertTrue(ct.waitForMeta(10000).equals(SN)); } /** * @param implementation An {@link HRegionInterface} instance; you'll likely * want to pass a mocked HRS; can be null. - * @return Mock up a connection that returns a {@link Configuration} when + * @return Mock up a connection that returns a {@link org.apache.hadoop.conf.Configuration} when * {@link HConnection#getConfiguration()} is called, a 'location' when * {@link HConnection#getRegionLocation(byte[], byte[], boolean)} is called, * and that returns the passed {@link HRegionInterface} instance when * {@link HConnection#getHRegionConnection(String, int)} * is called (Be sure call - * {@link HConnectionManager#deleteConnection(org.apache.hadoop.conf.Configuration, boolean)} + * {@link HConnectionManager#deleteConnection(org.apache.hadoop.conf.Configuration)} * when done with this mocked Connection. * @throws IOException */ diff --git a/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java b/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java deleted file mode 100644 index fe371565a0f8..000000000000 --- a/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java +++ /dev/null @@ -1,81 +0,0 @@ -/** - * Copyright 2011 The Apache Software Foundation - * - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.hbase.catalog; - -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; -import org.apache.zookeeper.KeeperException; -import org.junit.Test; -import org.junit.experimental.categories.Category; - -/** - * Do {@link CatalogTracker} tests on running cluster. - */ -@Category(LargeTests.class) -public class TestCatalogTrackerOnCluster { - private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); - private static final Log LOG = - LogFactory.getLog(TestCatalogTrackerOnCluster.class); - - /** - * @throws Exception - * @see {https://issues.apache.org/jira/browse/HBASE-3445} - */ - @Test public void testBadOriginalRootLocation() throws Exception { - UTIL.getConfiguration().setInt("ipc.socket.timeout", 3000); - // Launch cluster so it does bootstrapping. - UTIL.startMiniCluster(); - // Shutdown hbase. - UTIL.shutdownMiniHBaseCluster(); - // Mess with the root location in the running zk. Set it to be nonsense. - ZooKeeperWatcher zookeeper = new ZooKeeperWatcher(UTIL.getConfiguration(), - "Bad Root Location Writer", new Abortable() { - @Override - public void abort(String why, Throwable e) { - LOG.error("Abort was called on 'bad root location writer'", e); - } - - @Override - public boolean isAborted() { - return false; - } - }); - ServerName nonsense = - new ServerName("example.org", 1234, System.currentTimeMillis()); - RootLocationEditor.setRootLocation(zookeeper, nonsense); - - // Bring back up the hbase cluster. See if it can deal with nonsense root - // location. The cluster should start and be fully available. - UTIL.startMiniHBaseCluster(1, 1); - - // if we can create a table, it's a good sign that it's working - UTIL.createTable( - getClass().getSimpleName().getBytes(), "family".getBytes()); - - UTIL.shutdownMiniCluster(); - } - - @org.junit.Rule - public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = - new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); -} - diff --git a/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java b/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java index 1105ec94c903..07af255cbb23 100644 --- a/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java +++ b/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java b/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java index 00424683f932..5e78c247128e 100644 --- a/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java +++ b/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java @@ -125,7 +125,7 @@ public void testRideOverServerNotRunning() throws IOException, InterruptedExcept HConstants.CATALOG_FAMILY, HConstants.STARTCODE_QUALIFIER, Bytes.toBytes(sn.getStartcode()))); final Result [] result = new Result [] {new Result(kvs)}; - Mockito.when(implementation.next(Mockito.anyLong(), Mockito.anyInt())). + Mockito.when(implementation.next(Mockito.anyLong(), Mockito.anyInt(), Mockito.anyInt())). thenReturn(result). thenReturn(null); @@ -153,7 +153,7 @@ public void testRideOverServerNotRunning() throws IOException, InterruptedExcept when(connection).getHRegionConnection(Mockito.anyString(), Mockito.anyInt()); // Now start up the catalogtracker with our doctored Connection. - ct = new CatalogTracker(zkw, null, connection, ABORTABLE, 0); + ct = new CatalogTracker(zkw, null, connection, ABORTABLE); ct.start(); // Scan meta for user tables and verify we got back expected answer. NavigableMap hris = MetaReader.getServerUserRegions(ct, sn); @@ -166,7 +166,7 @@ public void testRideOverServerNotRunning() throws IOException, InterruptedExcept openScanner((byte [])Mockito.any(), (Scan)Mockito.any()); } finally { if (ct != null) ct.stop(); - HConnectionManager.deleteConnection(UTIL.getConfiguration(), true); + HConnectionManager.deleteConnection(UTIL.getConfiguration()); zkw.close(); } } diff --git a/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java b/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java index e34d8bcdd4fa..fdac02d575a6 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java +++ b/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java @@ -129,7 +129,7 @@ public static HConnection getSpiedConnection(final Configuration conf) HConnectionImplementation connection = HConnectionManager.HBASE_INSTANCES.get(connectionKey); if (connection == null) { - connection = Mockito.spy(new HConnectionImplementation(conf, true)); + connection = Mockito.spy(new HConnectionImplementation(conf, true, null)); HConnectionManager.HBASE_INSTANCES.put(connectionKey, connection); } return connection; diff --git a/src/test/java/org/apache/hadoop/hbase/client/InstantSchemaChangeTestBase.java b/src/test/java/org/apache/hadoop/hbase/client/InstantSchemaChangeTestBase.java deleted file mode 100644 index 378c2b47dab2..000000000000 --- a/src/test/java/org/apache/hadoop/hbase/client/InstantSchemaChangeTestBase.java +++ /dev/null @@ -1,169 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.hbase.client; - -import static org.junit.Assert.assertEquals; - -import java.io.IOException; -import java.util.List; - -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.HBaseTestingUtility; -import org.apache.hadoop.hbase.HConstants; -import org.apache.hadoop.hbase.HTableDescriptor; -import org.apache.hadoop.hbase.LargeTests; -import org.apache.hadoop.hbase.MiniHBaseCluster; -import org.apache.hadoop.hbase.regionserver.HRegion; -import org.apache.hadoop.hbase.regionserver.HRegionServer; -import org.apache.hadoop.hbase.util.Bytes; -import org.apache.hadoop.hbase.util.JVMClusterUtil; -import org.apache.hadoop.hbase.zookeeper.MasterSchemaChangeTracker; -import org.apache.zookeeper.KeeperException; -import org.junit.After; -import org.junit.Before; -import org.junit.experimental.categories.Category; - -@Category(LargeTests.class) -public class InstantSchemaChangeTestBase { - - final Log LOG = LogFactory.getLog(getClass()); - protected final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); - protected HBaseAdmin admin; - protected static MiniHBaseCluster miniHBaseCluster = null; - protected Configuration conf; - protected static MasterSchemaChangeTracker msct = null; - - protected final byte [] row = Bytes.toBytes("row"); - protected final byte [] qualifier = Bytes.toBytes("qualifier"); - final byte [] value = Bytes.toBytes("value"); - - @Before - public void setUpBeforeClass() throws Exception { - TEST_UTIL.getConfiguration().setInt("hbase.regionserver.msginterval", 100); - TEST_UTIL.getConfiguration().setInt("hbase.client.pause", 250); - TEST_UTIL.getConfiguration().setInt("hbase.client.retries.number", 6); - TEST_UTIL.getConfiguration().setBoolean("hbase.instant.schema.alter.enabled", true); - TEST_UTIL.getConfiguration().setBoolean("hbase.online.schema.update.enable", true); - TEST_UTIL.getConfiguration().setInt("hbase.instant.schema.janitor.period", 10000); - TEST_UTIL.getConfiguration().setInt("hbase.instant.schema.alter.timeout", 30000); - // - miniHBaseCluster = TEST_UTIL.startMiniCluster(2,5); - msct = TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - this.admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); - - } - - @After - public void tearDownAfterClass() throws Exception { - TEST_UTIL.shutdownMiniCluster(); - } - - /** - * Find the RS that is currently holding our online region. - * @param tableName - * @return - */ - protected HRegionServer findRSWithOnlineRegionFor(String tableName) { - List rsThreads = - miniHBaseCluster.getLiveRegionServerThreads(); - for (JVMClusterUtil.RegionServerThread rsT : rsThreads) { - HRegionServer rs = rsT.getRegionServer(); - List regions = rs.getOnlineRegions(Bytes.toBytes(tableName)); - if (regions != null && !regions.isEmpty()) { - return rs; - } - } - return null; - } - - protected void waitForSchemaChangeProcess(final String tableName) - throws KeeperException, InterruptedException { - waitForSchemaChangeProcess(tableName, 10000); - } - - /** - * This a pretty low cost signalling mechanism. It is quite possible that we will - * miss out the ZK node creation signal as in some cases the schema change process - * happens rather quickly and our thread waiting for ZK node creation might wait forver. - * The fool-proof strategy would be to directly listen for ZK events. - * @param tableName - * @throws KeeperException - * @throws InterruptedException - */ - protected void waitForSchemaChangeProcess(final String tableName, final long waitTimeMills) - throws KeeperException, InterruptedException { - LOG.info("Waiting for ZK node creation for table = " + tableName); - final MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - final Runnable r = new Runnable() { - public void run() { - try { - while(!msct.doesSchemaChangeNodeExists(tableName)) { - try { - Thread.sleep(50); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - } - } catch (KeeperException ke) { - ke.printStackTrace(); - } - LOG.info("Waiting for ZK node deletion for table = " + tableName); - try { - while(msct.doesSchemaChangeNodeExists(tableName)) { - try { - Thread.sleep(50); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - } - } catch (KeeperException ke) { - ke.printStackTrace(); - } - } - }; - Thread t = new Thread(r); - t.start(); - if (waitTimeMills > 0) { - t.join(waitTimeMills); - } else { - t.join(10000); - } - } - - protected HTable createTableAndValidate(String tableName) throws IOException { - conf = TEST_UTIL.getConfiguration(); - LOG.info("Start createTableAndValidate()"); - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - HTableDescriptor[] tables = admin.listTables(); - int numTables = 0; - if (tables != null) { - numTables = tables.length; - } - HTable ht = TEST_UTIL.createTable(Bytes.toBytes(tableName), - HConstants.CATALOG_FAMILY); - tables = this.admin.listTables(); - assertEquals(numTables + 1, tables.length); - LOG.info("created table = " + tableName); - return ht; - } - -} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java b/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java index 327950c118a4..a3f3d1657e3a 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -24,22 +23,37 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; +import org.apache.zookeeper.KeeperException; import java.io.IOException; import java.util.ArrayList; -import java.util.Collections; -import java.util.Comparator; import java.util.HashMap; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.DoNotRetryIOException; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HRegionLocation; +import org.apache.hadoop.hbase.HServerAddress; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.InvalidFamilyOperationException; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.TableExistsException; +import org.apache.hadoop.hbase.TableNotDisabledException; +import org.apache.hadoop.hbase.TableNotEnabledException; +import org.apache.hadoop.hbase.TableNotFoundException; +import org.apache.hadoop.hbase.catalog.CatalogTracker; import org.apache.hadoop.hbase.executor.EventHandler; import org.apache.hadoop.hbase.executor.EventHandler.EventType; import org.apache.hadoop.hbase.executor.ExecutorService; @@ -47,9 +61,16 @@ import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.HRegionServer; import org.apache.hadoop.hbase.regionserver.wal.HLogUtilsForTests; -import org.apache.hadoop.hbase.InvalidFamilyOperationException; import org.apache.hadoop.hbase.util.Bytes; -import org.junit.*; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.hadoop.hbase.zookeeper.ZKTableReadOnly; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Assert; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; import org.junit.experimental.categories.Category; @@ -89,6 +110,47 @@ public void setUp() throws Exception { public void tearDown() throws Exception { } + @Test (timeout=300000) + public void testFailedCatalogTrackerGetCleansUpProperly() + throws IOException { + // An HBaseAdmin that we can make fail when it goes to get catalogtracker. + final AtomicBoolean fail = new AtomicBoolean(false); + final AtomicReference internalCt = new AtomicReference(); + HBaseAdmin doctoredAdmin = new HBaseAdmin(this.admin.getConfiguration()) { + @Override + protected CatalogTracker startCatalogTracker(CatalogTracker ct) + throws IOException, InterruptedException { + internalCt.set(ct); + super.startCatalogTracker(ct); + if (fail.get()) { + throw new IOException("Intentional test fail", + new KeeperException.ConnectionLossException()); + } + return ct; + } + }; + try { + CatalogTracker ct = doctoredAdmin.getCatalogTracker(); + assertFalse(ct.isStopped()); + doctoredAdmin.cleanupCatalogTracker(ct); + assertTrue(ct.isStopped()); + // Now have mess with our doctored admin and make the start of catalog tracker 'fail'. + fail.set(true); + boolean expectedException = false; + try { + doctoredAdmin.getCatalogTracker(); + } catch (IOException ioe) { + assertTrue(ioe.getCause() instanceof KeeperException.ConnectionLossException); + expectedException = true; + } + if (!expectedException) fail("Didn't get expected exception!"); + // Assert that the internally made ct was properly shutdown. + assertTrue("Internal CatalogTracker not closed down", internalCt.get().isStopped()); + } finally { + doctoredAdmin.close(); + } + } + @Test public void testSplitFlushCompactUnknownTable() throws InterruptedException { final String unknowntable = "fubar"; @@ -222,6 +284,9 @@ public void testDisableAndEnableTable() throws IOException { ht.get(get); this.admin.disableTable(table); + assertTrue("Table must be disabled.", TEST_UTIL.getHBaseCluster() + .getMaster().getAssignmentManager().getZKTable().isDisabledTable( + Bytes.toString(table))); // Test that table is disabled get = new Get(row); @@ -229,13 +294,14 @@ public void testDisableAndEnableTable() throws IOException { boolean ok = false; try { ht.get(get); - } catch (NotServingRegionException e) { - ok = true; - } catch (RetriesExhaustedException e) { + } catch (DoNotRetryIOException e) { ok = true; } assertTrue(ok); this.admin.enableTable(table); + assertTrue("Table must be enabled.", TEST_UTIL.getHBaseCluster() + .getMaster().getAssignmentManager().getZKTable().isEnabledTable( + Bytes.toString(table))); // Test that table is enabled try { @@ -247,6 +313,15 @@ public void testDisableAndEnableTable() throws IOException { ht.close(); } + @Test + public void testIsEnabledOnUnknownTable() throws Exception { + try { + admin.isTableEnabled(Bytes.toBytes("unkownTable")); + fail("Test should fail if isTableEnabled called on unknown table."); + } catch (IOException e) { + } + } + @Test public void testDisableAndEnableTables() throws IOException { final byte [] row = Bytes.toBytes("row"); @@ -274,23 +349,22 @@ public void testDisableAndEnableTables() throws IOException { try { ht1.get(get); ht2.get(get); - } catch (NotServingRegionException e) { - ok = true; - } catch (RetriesExhaustedException e) { + } catch (DoNotRetryIOException e) { ok = true; } + assertTrue(ok); this.admin.enableTables("testDisableAndEnableTable.*"); // Test that tables are enabled try { ht1.get(get); - } catch (RetriesExhaustedException e) { + } catch (IOException e) { ok = false; } try { ht2.get(get); - } catch (RetriesExhaustedException e) { + } catch (IOException e) { ok = false; } assertTrue(ok); @@ -307,6 +381,9 @@ public void testCreateTable() throws IOException { HConstants.CATALOG_FAMILY).close(); tables = this.admin.listTables(); assertEquals(numTables + 1, tables.length); + assertTrue("Table must be enabled.", TEST_UTIL.getHBaseCluster() + .getMaster().getAssignmentManager().getZKTable().isEnabledTable( + "testCreateTable")); } @Test @@ -343,8 +420,6 @@ public void testHColumnValidName() { @Test public void testOnlineChangeTableSchema() throws IOException, InterruptedException { final byte [] tableName = Bytes.toBytes("changeTableSchemaOnline"); - TEST_UTIL.getMiniHBaseCluster().getMaster().getConfiguration().setBoolean( - "hbase.online.schema.update.enable", true); HTableDescriptor [] tables = admin.listTables(); int numTables = tables.length; TEST_UTIL.createTable(tableName, HConstants.CATALOG_FAMILY).close(); @@ -358,6 +433,9 @@ public void testOnlineChangeTableSchema() throws IOException, InterruptedExcepti assertTrue(htd.equals(copy)); // Now amend the copy. Introduce differences. long newFlushSize = htd.getMemStoreFlushSize() / 2; + if (newFlushSize <=0) { + newFlushSize = HTableDescriptor.DEFAULT_MEMSTORE_FLUSH_SIZE / 2; + } copy.setMemStoreFlushSize(newFlushSize); final String key = "anyoldkey"; assertTrue(htd.getValue(key) == null); @@ -446,6 +524,9 @@ public void testShouldFailOnlineSchemaUpdateIfOnlineSchemaIsNotEnabled() assertTrue(htd.equals(copy)); // Now amend the copy. Introduce differences. long newFlushSize = htd.getMemStoreFlushSize() / 2; + if (newFlushSize <=0) { + newFlushSize = HTableDescriptor.DEFAULT_MEMSTORE_FLUSH_SIZE / 2; + } copy.setMemStoreFlushSize(newFlushSize); final String key = "anyoldkey"; assertTrue(htd.getValue(key) == null); @@ -457,6 +538,8 @@ public void testShouldFailOnlineSchemaUpdateIfOnlineSchemaIsNotEnabled() expectedException = true; } assertTrue("Online schema update should not happen.", expectedException); + TEST_UTIL.getMiniHBaseCluster().getMaster().getConfiguration().setBoolean( + "hbase.online.schema.update.enable", true); } /** @@ -531,6 +614,55 @@ protected void verifyRoundRobinDistribution(HTable ht, int expectedRegions) thro } } + @Test + public void testCreateTableNumberOfRegions() throws IOException, InterruptedException { + byte[] tableName = Bytes.toBytes("testCreateTableNumberOfRegions"); + HTableDescriptor desc = new HTableDescriptor(tableName); + desc.addFamily(new HColumnDescriptor(HConstants.CATALOG_FAMILY)); + admin.createTable(desc); + HTable ht = new HTable(TEST_UTIL.getConfiguration(), tableName); + Map regions = ht.getRegionLocations(); + assertEquals("Table should have only 1 region", 1, regions.size()); + ht.close(); + + byte[] TABLE_2 = Bytes.add(tableName, Bytes.toBytes("_2")); + desc = new HTableDescriptor(TABLE_2); + desc.addFamily(new HColumnDescriptor(HConstants.CATALOG_FAMILY)); + admin.createTable(desc, new byte[][] { new byte[] { 42 } }); + HTable ht2 = new HTable(TEST_UTIL.getConfiguration(), TABLE_2); + regions = ht2.getRegionLocations(); + assertEquals("Table should have only 2 region", 2, regions.size()); + ht2.close(); + + byte[] TABLE_3 = Bytes.add(tableName, Bytes.toBytes("_3")); + desc = new HTableDescriptor(TABLE_3); + desc.addFamily(new HColumnDescriptor(HConstants.CATALOG_FAMILY)); + admin.createTable(desc, "a".getBytes(), "z".getBytes(), 3); + HTable ht3 = new HTable(TEST_UTIL.getConfiguration(), TABLE_3); + regions = ht3.getRegionLocations(); + assertEquals("Table should have only 3 region", 3, regions.size()); + ht3.close(); + + byte[] TABLE_4 = Bytes.add(tableName, Bytes.toBytes("_4")); + desc = new HTableDescriptor(TABLE_4); + desc.addFamily(new HColumnDescriptor(HConstants.CATALOG_FAMILY)); + try { + admin.createTable(desc, "a".getBytes(), "z".getBytes(), 2); + fail("Should not be able to create a table with only 2 regions using this API."); + } catch (IllegalArgumentException eae) { + // Expected + } + + byte[] TABLE_5 = Bytes.add(tableName, Bytes.toBytes("_5")); + desc = new HTableDescriptor(TABLE_5); + desc.addFamily(new HColumnDescriptor(HConstants.CATALOG_FAMILY)); + admin.createTable(desc, new byte[] { 1 }, new byte[] { 127 }, 16); + HTable ht5 = new HTable(TEST_UTIL.getConfiguration(), TABLE_5); + regions = ht5.getRegionLocations(); + assertEquals("Table should have 16 region", 16, regions.size()); + ht5.close(); + } + @Test public void testCreateTableWithRegions() throws IOException, InterruptedException { @@ -703,6 +835,38 @@ public void testCreateTableWithRegions() throws IOException, InterruptedExceptio ladmin.close(); } + + @Test + public void testCreateTableWithOnlyEmptyStartRow() throws IOException { + byte[] tableName = Bytes.toBytes("testCreateTableWithOnlyEmptyStartRow"); + byte[][] splitKeys = new byte[1][]; + splitKeys[0] = HConstants.EMPTY_BYTE_ARRAY; + HTableDescriptor desc = new HTableDescriptor(tableName); + desc.addFamily(new HColumnDescriptor("col")); + try { + admin.createTable(desc, splitKeys); + fail("Test case should fail as empty split key is passed."); + } catch (IllegalArgumentException e) { + } + } + + @Test + public void testCreateTableWithEmptyRowInTheSplitKeys() throws IOException { + byte[] tableName = Bytes + .toBytes("testCreateTableWithEmptyRowInTheSplitKeys"); + byte[][] splitKeys = new byte[3][]; + splitKeys[0] = "region1".getBytes(); + splitKeys[1] = HConstants.EMPTY_BYTE_ARRAY; + splitKeys[2] = "region2".getBytes(); + HTableDescriptor desc = new HTableDescriptor(tableName); + desc.addFamily(new HColumnDescriptor("col")); + try { + admin.createTable(desc, splitKeys); + fail("Test case should fail as empty split key is passed."); + } catch (IllegalArgumentException e) { + } + } + @Test public void testTableExist() throws IOException { final byte [] table = Bytes.toBytes("testTableExist"); @@ -732,12 +896,12 @@ public void testForceSplit() throws Exception { } /** - * Test round-robin assignment on enableTable. + * Test retain assignment on enableTable. * * @throws IOException */ @Test - public void testEnableTableRoundRobinAssignment() throws IOException { + public void testEnableTableRetainAssignment() throws IOException, InterruptedException { byte[] tableName = Bytes.toBytes("testEnableTableAssignment"); byte[][] splitKeys = { new byte[] { 1, 1, 1 }, new byte[] { 2, 2, 2 }, new byte[] { 3, 3, 3 }, new byte[] { 4, 4, 4 }, new byte[] { 5, 5, 5 }, @@ -748,42 +912,22 @@ public void testEnableTableRoundRobinAssignment() throws IOException { desc.addFamily(new HColumnDescriptor(HConstants.CATALOG_FAMILY)); admin.createTable(desc, splitKeys); HTable ht = new HTable(TEST_UTIL.getConfiguration(), tableName); - Map regions = ht.getRegionsInfo(); + Map regions = ht.getRegionLocations(); assertEquals("Tried to create " + expectedRegions + " regions " + "but only found " + regions.size(), expectedRegions, regions.size()); // Disable table. admin.disableTable(tableName); - // Enable table, use round-robin assignment to assign regions. + // Enable table, use retain assignment to assign regions. admin.enableTable(tableName); + Map regions2 = ht.getRegionLocations(); // Check the assignment. - HTable metaTable = new HTable(TEST_UTIL.getConfiguration(), - HConstants.META_TABLE_NAME); - List regionInfos = admin.getTableRegions(tableName); - Map serverMap = new HashMap(); - for (int i = 0, j = regionInfos.size(); i < j; i++) { - HRegionInfo hri = regionInfos.get(i); - Get get = new Get(hri.getRegionName()); - Result result = metaTable.get(get); - String server = Bytes.toString(result.getValue(HConstants.CATALOG_FAMILY, - HConstants.SERVER_QUALIFIER)); - Integer regioncount = serverMap.get(server); - if (regioncount == null) { - regioncount = 0; - } - regioncount++; - serverMap.put(server, regioncount); - } - List> entryList = new ArrayList>( - serverMap.entrySet()); - Collections.sort(entryList, new Comparator>() { - public int compare(Map.Entry oa, - Map.Entry ob) { - return (oa.getValue() - ob.getValue()); - } - }); - assertTrue(entryList.size() == 3); - assertTrue((entryList.get(2).getValue() - entryList.get(0).getValue()) < 2); + assertEquals(regions.size(), regions2.size()); + for (Map.Entry entry : regions.entrySet()) { + ServerName sn1 = regions2.get(entry.getKey()); + ServerName sn2 = entry.getValue(); + assertEquals(sn1, sn2); + } } /** @@ -826,116 +970,116 @@ void splitTest(byte[] splitPoint, byte[][] familyNames, int[] rowCounts, assertFalse(admin.tableExists(tableName)); final HTable table = TEST_UTIL.createTable(tableName, familyNames, numVersions, blockSize); - try { - int rowCount = 0; - - // insert rows into column families. The number of rows that have values - // in a specific column family is decided by rowCounts[familyIndex] - for (int index = 0; index < familyNames.length; index++) { - for (int i = 0; i < rowCounts[index]; i++) { - byte[] k = Bytes.toBytes(i); - Put put = new Put(k); - put.add(familyNames[index], new byte[0], k); - table.put(put); - } - - if ( rowCount < rowCounts[index] ) { - rowCount = rowCounts[index]; - } + int rowCount = 0; + byte[] q = new byte[0]; + + // insert rows into column families. The number of rows that have values + // in a specific column family is decided by rowCounts[familyIndex] + for (int index = 0; index < familyNames.length; index++) { + ArrayList puts = new ArrayList(rowCounts[index]); + for (int i = 0; i < rowCounts[index]; i++) { + byte[] k = Bytes.toBytes(i); + Put put = new Put(k); + put.add(familyNames[index], q, k); + puts.add(put); } + table.put(puts); - // get the initial layout (should just be one region) - Map m = table.getRegionsInfo(); - System.out.println("Initial regions (" + m.size() + "): " + m); - assertTrue(m.size() == 1); - - // Verify row count - Scan scan = new Scan(); - ResultScanner scanner = table.getScanner(scan); - int rows = 0; - for(@SuppressWarnings("unused") Result result : scanner) { - rows++; + if ( rowCount < rowCounts[index] ) { + rowCount = rowCounts[index]; } - scanner.close(); - assertEquals(rowCount, rows); - - // Have an outstanding scan going on to make sure we can scan over splits. - scan = new Scan(); - scanner = table.getScanner(scan); - // Scan first row so we are into first region before split happens. - scanner.next(); + } - final AtomicInteger count = new AtomicInteger(0); - Thread t = new Thread("CheckForSplit") { - public void run() { - for (int i = 0; i < 20; i++) { - try { - sleep(1000); - } catch (InterruptedException e) { - continue; - } - // check again table = new HTable(conf, tableName); - Map regions = null; - try { - regions = table.getRegionsInfo(); - } catch (IOException e) { - e.printStackTrace(); - } - if (regions == null) continue; - count.set(regions.size()); - if (count.get() >= 2) break; - LOG.debug("Cycle waiting on split"); + // get the initial layout (should just be one region) + Map m = table.getRegionsInfo(); + System.out.println("Initial regions (" + m.size() + "): " + m); + assertTrue(m.size() == 1); + + // Verify row count + Scan scan = new Scan(); + ResultScanner scanner = table.getScanner(scan); + int rows = 0; + for(@SuppressWarnings("unused") Result result : scanner) { + rows++; + } + scanner.close(); + assertEquals(rowCount, rows); + + // Have an outstanding scan going on to make sure we can scan over splits. + scan = new Scan(); + scanner = table.getScanner(scan); + // Scan first row so we are into first region before split happens. + scanner.next(); + + final AtomicInteger count = new AtomicInteger(0); + Thread t = new Thread("CheckForSplit") { + public void run() { + for (int i = 0; i < 20; i++) { + try { + sleep(1000); + } catch (InterruptedException e) { + continue; } - } - }; - t.start(); - // Split the table - this.admin.split(tableName, splitPoint); - t.join(); - - // Verify row count - rows = 1; // We counted one row above. - for (@SuppressWarnings("unused") Result result : scanner) { - rows++; - if (rows > rowCount) { - scanner.close(); - assertTrue("Scanned more than expected (" + rowCount + ")", false); + // check again table = new HTable(conf, tableName); + Map regions = null; + try { + regions = table.getRegionsInfo(); + } catch (IOException e) { + e.printStackTrace(); + } + if (regions == null) continue; + count.set(regions.size()); + if (count.get() >= 2) break; + LOG.debug("Cycle waiting on split"); } } - scanner.close(); - assertEquals(rowCount, rows); - - Map regions = null; - try { - regions = table.getRegionsInfo(); - } catch (IOException e) { - e.printStackTrace(); + }; + t.start(); + // Split the table + this.admin.split(tableName, splitPoint); + t.join(); + + // Verify row count + rows = 1; // We counted one row above. + for (@SuppressWarnings("unused") Result result : scanner) { + rows++; + if (rows > rowCount) { + scanner.close(); + assertTrue("Scanned more than expected (" + rowCount + ")", false); } - assertEquals(2, regions.size()); - HRegionInfo[] r = regions.keySet().toArray(new HRegionInfo[0]); - if (splitPoint != null) { - // make sure the split point matches our explicit configuration - assertEquals(Bytes.toString(splitPoint), - Bytes.toString(r[0].getEndKey())); - assertEquals(Bytes.toString(splitPoint), - Bytes.toString(r[1].getStartKey())); - LOG.debug("Properly split on " + Bytes.toString(splitPoint)); - } else { - if (familyNames.length > 1) { - int splitKey = Bytes.toInt(r[0].getEndKey()); - // check if splitKey is based on the largest column family - // in terms of it store size - int deltaForLargestFamily = Math.abs(rowCount/2 - splitKey); - for (int index = 0; index < familyNames.length; index++) { - int delta = Math.abs(rowCounts[index]/2 - splitKey); - assertTrue(delta >= deltaForLargestFamily); - } + } + scanner.close(); + assertEquals(rowCount, rows); + + Map regions = null; + try { + regions = table.getRegionsInfo(); + } catch (IOException e) { + e.printStackTrace(); + } + assertEquals(2, regions.size()); + HRegionInfo[] r = regions.keySet().toArray(new HRegionInfo[0]); + if (splitPoint != null) { + // make sure the split point matches our explicit configuration + assertEquals(Bytes.toString(splitPoint), + Bytes.toString(r[0].getEndKey())); + assertEquals(Bytes.toString(splitPoint), + Bytes.toString(r[1].getStartKey())); + LOG.debug("Properly split on " + Bytes.toString(splitPoint)); + } else { + if (familyNames.length > 1) { + int splitKey = Bytes.toInt(r[0].getEndKey()); + // check if splitKey is based on the largest column family + // in terms of it store size + int deltaForLargestFamily = Math.abs(rowCount/2 - splitKey); + for (int index = 0; index < familyNames.length; index++) { + int delta = Math.abs(rowCounts[index]/2 - splitKey); + assertTrue(delta >= deltaForLargestFamily); } } - } finally { - TEST_UTIL.deleteTable(tableName); - table.close(); } + TEST_UTIL.deleteTable(tableName); + table.close(); } /** @@ -952,16 +1096,21 @@ public void testInvalidHColumnDescriptor() throws IOException { new HColumnDescriptor("/cfamily/name"); } - @Test + @Test(timeout=300000) public void testEnableDisableAddColumnDeleteColumn() throws Exception { + ZooKeeperWatcher zkw = HBaseTestingUtility.getZooKeeperWatcher(TEST_UTIL); byte [] tableName = Bytes.toBytes("testMasterAdmin"); TEST_UTIL.createTable(tableName, HConstants.CATALOG_FAMILY).close(); + while (!ZKTableReadOnly.isEnabledTable(zkw, "testMasterAdmin")) { + Thread.sleep(10); + } this.admin.disableTable(tableName); try { new HTable(TEST_UTIL.getConfiguration(), tableName); - } catch (org.apache.hadoop.hbase.client.RegionOfflineException e) { - // Expected + } catch (DoNotRetryIOException e) { + //expected } + this.admin.addColumn(tableName, new HColumnDescriptor("col2")); this.admin.enableTable(tableName); try { @@ -1059,16 +1208,21 @@ public void testTableNameClash() throws Exception { @Test public void testCreateTableRPCTimeOut() throws Exception { String name = "testCreateTableRPCTimeOut"; + int oldTimeout = TEST_UTIL.getConfiguration(). + getInt(HConstants.HBASE_RPC_TIMEOUT_KEY, HConstants.DEFAULT_HBASE_RPC_TIMEOUT); TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_RPC_TIMEOUT_KEY, 1500); - - int expectedRegions = 100; - // Use 80 bit numbers to make sure we aren't limited - byte [] startKey = { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 }; - byte [] endKey = { 9, 9, 9, 9, 9, 9, 9, 9, 9, 9 }; - HBaseAdmin hbaseadmin = new HBaseAdmin(TEST_UTIL.getConfiguration()); - hbaseadmin.createTable(new HTableDescriptor(name), startKey, endKey, - expectedRegions); - hbaseadmin.close(); + try { + int expectedRegions = 100; + // Use 80 bit numbers to make sure we aren't limited + byte [] startKey = { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 }; + byte [] endKey = { 9, 9, 9, 9, 9, 9, 9, 9, 9, 9 }; + HBaseAdmin hbaseadmin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + hbaseadmin.createTable(new HTableDescriptor(name), startKey, endKey, + expectedRegions); + hbaseadmin.close(); + } finally { + TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_RPC_TIMEOUT_KEY, oldTimeout); + } } /** @@ -1184,10 +1338,14 @@ public void testShouldCloseTheRegionBasedOnTheEncodedRegionName() .getServerName().getServerName()); } } - Thread.sleep(1000); - onlineRegions = rs.getOnlineRegions(); + boolean isInList = rs.getOnlineRegions().contains(info); + long timeout = System.currentTimeMillis() + 10000; + while ((System.currentTimeMillis() < timeout) && (isInList)) { + Thread.sleep(100); + isInList = rs.getOnlineRegions().contains(info); + } assertFalse("The region should not be present in online regions list.", - onlineRegions.contains(info)); + isInList); } @Test @@ -1232,7 +1390,7 @@ public void testCloseRegionThatFetchesTheHRIFromMeta() throws Exception { } boolean isInList = rs.getOnlineRegions().contains(info); - long timeout = System.currentTimeMillis() + 2000; + long timeout = System.currentTimeMillis() + 10000; while ((System.currentTimeMillis() < timeout) && (isInList)) { Thread.sleep(100); isInList = rs.getOnlineRegions().contains(info); @@ -1385,7 +1543,7 @@ public void testHLogRollWriting() throws Exception { int count = HLogUtilsForTests.getNumLogFiles(regionServer.getWAL()); LOG.info("after flushing all regions and rolling logs there are " + count + " log files"); - assertTrue(("actual count: " + count), count <= 2); + assertTrue(("actual count: " + count), count <= 3); } private void setUpforLogRolling() { @@ -1482,6 +1640,70 @@ public void testCheckHBaseAvailableClosesConnection() throws Exception { Assert.assertEquals(initialCount, finalCount) ; } + @Test + public void testDisableCatalogTable() throws Exception { + try { + this.admin.disableTable(".META."); + fail("Expected to throw IllegalArgumentException"); + } catch (IllegalArgumentException e) { + } + // Before the fix for HBASE-6146, the below table creation was failing as the META table + // actually getting disabled by the disableTable() call. + HTableDescriptor htd = new HTableDescriptor("testDisableCatalogTable".getBytes()); + HColumnDescriptor hcd = new HColumnDescriptor("cf1".getBytes()); + htd.addFamily(hcd); + TEST_UTIL.getHBaseAdmin().createTable(htd); + } + + @Test + public void testGetRegion() throws Exception { + final String name = "testGetRegion"; + LOG.info("Started " + name); + final byte [] nameBytes = Bytes.toBytes(name); + HTable t = TEST_UTIL.createTable(nameBytes, HConstants.CATALOG_FAMILY); + TEST_UTIL.createMultiRegions(t, HConstants.CATALOG_FAMILY); + CatalogTracker ct = new CatalogTracker(TEST_UTIL.getConfiguration()); + ct.start(); + try { + HRegionLocation regionLocation = t.getRegionLocation("mmm"); + HRegionInfo region = regionLocation.getRegionInfo(); + byte[] regionName = region.getRegionName(); + Pair pair = admin.getRegion(regionName, ct); + assertTrue(Bytes.equals(regionName, pair.getFirst().getRegionName())); + pair = admin.getRegion(region.getEncodedNameAsBytes(), ct); + assertTrue(Bytes.equals(regionName, pair.getFirst().getRegionName())); + } finally { + ct.stop(); + } + } + + @Test + public void testRootTableSplit() throws Exception { + Scan s = new Scan(); + HTable rootTable = new HTable(TEST_UTIL.getConfiguration(), HConstants.ROOT_TABLE_NAME); + ResultScanner scanner = rootTable.getScanner(s); + Result metaEntry = scanner.next(); + this.admin.split(HConstants.ROOT_TABLE_NAME, metaEntry.getRow()); + Thread.sleep(1000); + List regions = TEST_UTIL.getMiniHBaseCluster().getRegions(HConstants.ROOT_TABLE_NAME); + assertEquals("ROOT region should not be splitted.",1, regions.size()); + } + + @Test + public void testIsEnabledOrDisabledOnUnknownTable() throws Exception { + try { + admin.isTableEnabled(Bytes.toBytes("unkownTable")); + fail("Test should fail if isTableEnabled called on unknown table."); + } catch (IOException e) { + } + + try { + admin.isTableDisabled(Bytes.toBytes("unkownTable")); + fail("Test should fail if isTableDisabled called on unknown table."); + } catch (IOException e) { + } + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestAttributes.java b/src/test/java/org/apache/hadoop/hbase/client/TestAttributes.java index 4821ceaf06ac..a65724369d6b 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestAttributes.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestAttributes.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -154,6 +153,46 @@ public void testDeleteAttributes() { Assert.assertNull(del.getAttributesMap().get("attribute1")); } + @Test + public void testGetId() { + Get get = new Get(); + Assert.assertNull("Make sure id is null if unset", get.toMap().get("id")); + get.setId("myId"); + Assert.assertEquals("myId", get.toMap().get("id")); + } + + @Test + public void testAppendId() { + Append append = new Append(); + Assert.assertNull("Make sure id is null if unset", append.toMap().get("id")); + append.setId("myId"); + Assert.assertEquals("myId", append.toMap().get("id")); + } + + @Test + public void testDeleteId() { + Delete delete = new Delete(); + Assert.assertNull("Make sure id is null if unset", delete.toMap().get("id")); + delete.setId("myId"); + Assert.assertEquals("myId", delete.toMap().get("id")); + } + + @Test + public void testPutId() { + Put put = new Put(); + Assert.assertNull("Make sure id is null if unset", put.toMap().get("id")); + put.setId("myId"); + Assert.assertEquals("myId", put.toMap().get("id")); + } + + @Test + public void testScanId() { + Scan scan = new Scan(); + Assert.assertNull("Make sure id is null if unset", scan.toMap().get("id")); + scan.setId("myId"); + Assert.assertEquals("myId", scan.toMap().get("id")); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestClientScannerRPCTimeout.java b/src/test/java/org/apache/hadoop/hbase/client/TestClientScannerRPCTimeout.java new file mode 100644 index 000000000000..38b558cd848c --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestClientScannerRPCTimeout.java @@ -0,0 +1,115 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import static org.junit.Assert.assertNotNull; + +import java.io.IOException; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.MiniHBaseCluster.MiniHBaseClusterRegionServer; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test the scenario where a next() call, while scanning, timeout at client side and getting retried. + * This scenario should not result in some data being skipped at RS side. + */ +@Category(MediumTests.class) +public class TestClientScannerRPCTimeout { + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final byte[] FAMILY = Bytes.toBytes("testFamily"); + private static final byte[] QUALIFIER = Bytes.toBytes("testQualifier"); + private static final byte[] VALUE = Bytes.toBytes("testValue"); + private static final int SLAVES = 1; + private static final int rpcTimeout = 5 * 1000; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + Configuration conf = TEST_UTIL.getConfiguration(); + conf.setInt(HConstants.HBASE_RPC_TIMEOUT_KEY, rpcTimeout); + conf.setStrings(HConstants.REGION_SERVER_IMPL, RegionServerWithScanTimeout.class.getName()); + TEST_UTIL.startMiniCluster(SLAVES); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + @Test + public void testScannerNextRPCTimesout() throws Exception { + byte[] TABLE = Bytes.toBytes("testScannerNextRPCTimesout"); + HTable ht = TEST_UTIL.createTable(TABLE, FAMILY); + putToTable(ht, "row-1"); + putToTable(ht, "row-2"); + RegionServerWithScanTimeout.seqNoToSleepOn = 1; + Scan scan = new Scan(); + scan.setCaching(1); + ResultScanner scanner = ht.getScanner(scan); + Result result = scanner.next(); + assertNotNull("Expected not null result", result); + result = scanner.next(); + assertNotNull("Expected not null result", result); + scanner.close(); + } + + private void putToTable(HTable ht, String rowkey) throws IOException { + Put put = new Put(rowkey.getBytes()); + put.add(FAMILY, QUALIFIER, VALUE); + ht.put(put); + } + + private static class RegionServerWithScanTimeout extends MiniHBaseClusterRegionServer { + private long tableScannerId; + private boolean slept; + private static long seqNoToSleepOn = -1; + + public RegionServerWithScanTimeout(Configuration conf) throws IOException, + InterruptedException { + super(conf); + } + + @Override + public long openScanner(byte[] regionName, Scan scan) throws IOException { + long scannerId = super.openScanner(regionName, scan); + if (!getRegionInfo(regionName).isMetaTable()) { + tableScannerId = scannerId; + } + return scannerId; + } + + @Override + public Result[] next(long scannerId, int nbRows, long callSeq) throws IOException { + if (!slept && this.tableScannerId == scannerId && seqNoToSleepOn == callSeq) { + try { + Thread.sleep(rpcTimeout + 500); + } catch (InterruptedException e) { + } + slept = true; + } + return super.next(scannerId, nbRows, callSeq); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java b/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java new file mode 100644 index 000000000000..841ab7400f26 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java @@ -0,0 +1,223 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import static org.junit.Assert.assertEquals; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.master.MasterFileSystem; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException; +import org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test clone snapshots from the client + */ +@Category(LargeTests.class) +public class TestCloneSnapshotFromClient { + final Log LOG = LogFactory.getLog(getClass()); + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + private final byte[] FAMILY = Bytes.toBytes("cf"); + + private byte[] emptySnapshot; + private byte[] snapshotName0; + private byte[] snapshotName1; + private byte[] snapshotName2; + private int snapshot0Rows; + private int snapshot1Rows; + private byte[] tableName; + private HBaseAdmin admin; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.getConfiguration().setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + TEST_UTIL.getConfiguration().setBoolean("hbase.online.schema.update.enable", true); + TEST_UTIL.getConfiguration().setInt("hbase.hstore.compactionThreshold", 10); + TEST_UTIL.getConfiguration().setInt("hbase.regionserver.msginterval", 100); + TEST_UTIL.getConfiguration().setInt("hbase.client.pause", 250); + TEST_UTIL.getConfiguration().setInt("hbase.client.retries.number", 6); + TEST_UTIL.getConfiguration().setBoolean( + "hbase.master.enabletable.roundrobin", true); + TEST_UTIL.startMiniCluster(3); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * Initialize the tests with a table filled with some data + * and two snapshots (snapshotName0, snapshotName1) of different states. + * The tableName, snapshotNames and the number of rows in the snapshot are initialized. + */ + @Before + public void setup() throws Exception { + this.admin = TEST_UTIL.getHBaseAdmin(); + + long tid = System.currentTimeMillis(); + tableName = Bytes.toBytes("testtb-" + tid); + emptySnapshot = Bytes.toBytes("emptySnaptb-" + tid); + snapshotName0 = Bytes.toBytes("snaptb0-" + tid); + snapshotName1 = Bytes.toBytes("snaptb1-" + tid); + snapshotName2 = Bytes.toBytes("snaptb2-" + tid); + + // create Table and disable it + SnapshotTestingUtils.createTable(TEST_UTIL, tableName, FAMILY); + admin.disableTable(tableName); + + // take an empty snapshot + admin.snapshot(emptySnapshot, tableName); + + HTable table = new HTable(TEST_UTIL.getConfiguration(), tableName); + try { + // enable table and insert data + admin.enableTable(tableName); + SnapshotTestingUtils.loadData(TEST_UTIL, table, 500, FAMILY); + snapshot0Rows = TEST_UTIL.countRows(table); + admin.disableTable(tableName); + + // take a snapshot + admin.snapshot(snapshotName0, tableName); + + // enable table and insert more data + admin.enableTable(tableName); + SnapshotTestingUtils.loadData(TEST_UTIL, table, 500, FAMILY); + snapshot1Rows = TEST_UTIL.countRows(table); + admin.disableTable(tableName); + + // take a snapshot of the updated table + admin.snapshot(snapshotName1, tableName); + + // re-enable table + admin.enableTable(tableName); + } finally { + table.close(); + } + } + + @After + public void tearDown() throws Exception { + if (admin.tableExists(tableName)) { + TEST_UTIL.deleteTable(tableName); + } + SnapshotTestingUtils.deleteAllSnapshots(admin); + SnapshotTestingUtils.deleteArchiveDirectory(TEST_UTIL); + } + + @Test(expected=SnapshotDoesNotExistException.class) + public void testCloneNonExistentSnapshot() throws IOException, InterruptedException { + String snapshotName = "random-snapshot-" + System.currentTimeMillis(); + String tableName = "random-table-" + System.currentTimeMillis(); + admin.cloneSnapshot(snapshotName, tableName); + } + + @Test + public void testCloneSnapshot() throws IOException, InterruptedException { + byte[] clonedTableName = Bytes.toBytes("clonedtb-" + System.currentTimeMillis()); + testCloneSnapshot(clonedTableName, snapshotName0, snapshot0Rows); + testCloneSnapshot(clonedTableName, snapshotName1, snapshot1Rows); + testCloneSnapshot(clonedTableName, emptySnapshot, 0); + } + + private void testCloneSnapshot(final byte[] tableName, final byte[] snapshotName, + int snapshotRows) throws IOException, InterruptedException { + // create a new table from snapshot + admin.cloneSnapshot(snapshotName, tableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, tableName, snapshotRows); + + TEST_UTIL.deleteTable(tableName); + } + + /** + * Verify that tables created from the snapshot are still alive after source table deletion. + */ + @Test + public void testCloneLinksAfterDelete() throws IOException, InterruptedException { + // Clone a table from the first snapshot + byte[] clonedTableName = Bytes.toBytes("clonedtb1-" + System.currentTimeMillis()); + admin.cloneSnapshot(snapshotName0, clonedTableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, clonedTableName, snapshot0Rows); + + // Take a snapshot of this cloned table. + admin.disableTable(clonedTableName); + admin.snapshot(snapshotName2, clonedTableName); + + // Clone the snapshot of the cloned table + byte[] clonedTableName2 = Bytes.toBytes("clonedtb2-" + System.currentTimeMillis()); + admin.cloneSnapshot(snapshotName2, clonedTableName2); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, clonedTableName2, snapshot0Rows); + admin.disableTable(clonedTableName2); + + // Remove the original table + TEST_UTIL.deleteTable(tableName); + waitCleanerRun(); + + // Verify the first cloned table + admin.enableTable(clonedTableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, clonedTableName, snapshot0Rows); + + // Verify the second cloned table + admin.enableTable(clonedTableName2); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, clonedTableName2, snapshot0Rows); + admin.disableTable(clonedTableName2); + + // Delete the first cloned table + TEST_UTIL.deleteTable(clonedTableName); + waitCleanerRun(); + + // Verify the second cloned table + admin.enableTable(clonedTableName2); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, clonedTableName2, snapshot0Rows); + + // Clone a new table from cloned + byte[] clonedTableName3 = Bytes.toBytes("clonedtb3-" + System.currentTimeMillis()); + admin.cloneSnapshot(snapshotName2, clonedTableName3); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, clonedTableName3, snapshot0Rows); + + // Delete the cloned tables + TEST_UTIL.deleteTable(clonedTableName2); + TEST_UTIL.deleteTable(clonedTableName3); + admin.deleteSnapshot(snapshotName2); + } + + // ========================================================================== + // Helpers + // ========================================================================== + private void waitCleanerRun() throws InterruptedException { + TEST_UTIL.getMiniHBaseCluster().getMaster().getHFileCleaner().choreForTesting(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestConnectionUtils.java b/src/test/java/org/apache/hadoop/hbase/client/TestConnectionUtils.java new file mode 100644 index 000000000000..4120ec332029 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestConnectionUtils.java @@ -0,0 +1,56 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import java.util.Set; +import java.util.TreeSet; + +import static org.junit.Assert.assertTrue; + +@Category(SmallTests.class) +public class TestConnectionUtils { + + @Test + public void testRetryTimeJitter() { + long[] retries = new long[200]; + long baseTime = 1000000; //Larger number than reality to help test randomness. + long maxTimeExpected = (long) (baseTime * 1.01f); + for (int i = 0; i < retries.length; i++) { + retries[i] = ConnectionUtils.getPauseTime(baseTime, 0); + } + + Set retyTimeSet = new TreeSet(); + for (long l : retries) { + /*make sure that there is some jitter but only 1%*/ + assertTrue(l >= baseTime); + assertTrue(l <= maxTimeExpected); + // Add the long to the set + retyTimeSet.add(l); + } + + //Make sure that most are unique. some overlap will happen + assertTrue(retyTimeSet.size() > (retries.length * 0.80)); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestCoprocessorHConnection.java b/src/test/java/org/apache/hadoop/hbase/client/TestCoprocessorHConnection.java new file mode 100644 index 000000000000..535c5b268bf2 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestCoprocessorHConnection.java @@ -0,0 +1,115 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import static org.junit.Assert.assertEquals; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +@Category(MediumTests.class) +public class TestCoprocessorHConnection { + + private static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @BeforeClass + public static void setupCluster() throws Exception { + UTIL.startMiniCluster(); + } + + @AfterClass + public static void shutdownCluster() throws Exception { + UTIL.shutdownMiniCluster(); + } + + /** + * Ensure that if the HRegion we are looking up isn't on this server (and not in the cache), that + * we still correctly look it up. + * @throws Exception on failure + */ + @Test + public void testNonServerLocalLookup() throws Exception { + Configuration conf = UTIL.getConfiguration(); + // make a fake server that should never be called + HRegionServer server = Mockito.mock(HRegionServer.class); + ServerName name = new ServerName("not.a.server.hostname", 12345, -1L); + Mockito.when(server.getServerName()).thenReturn(name); + CoprocessorHConnection connection = new CoprocessorHConnection(conf, server); + + // make sure we get the mock server when doing a direct lookup + assertEquals("Didn't get the mock server from the connection", server, + connection.getHRegionConnection(name.getHostname(), name.getPort())); + + // create a table that exists + byte[] tableName = Bytes.toBytes("testNonServerLocalLookup"); + byte[] family = Bytes.toBytes("family"); + UTIL.createTable(tableName, family); + + // if we can write to the table correctly, then our connection is doing the right thing + HTable table = new HTable(tableName, connection); + Put p = new Put(Bytes.toBytes("row")); + p.add(family, null, null); + table.put(p); + table.flushCommits(); + + + // cleanup + table.close(); + connection.close(); + } + + @Test + public void testLocalServerLookup() throws Exception { + Configuration conf = UTIL.getConfiguration(); + // get a real rs + HRegionServer server = + UTIL.getMiniHBaseCluster().getLiveRegionServerThreads().get(0).getRegionServer(); + // fake the connection to look like we are living on that server + CoprocessorHConnection connection = new CoprocessorHConnection(conf, server); + + // create a table that exists + byte[] tableName = Bytes.toBytes("testLocalServerLookup"); + byte[] family = Bytes.toBytes("family"); + UTIL.createTable(tableName, family); + + // if we can write to the table correctly, then our connection is doing the right thing + HTable table = new HTable(tableName, connection); + Put p = new Put(Bytes.toBytes("row")); + p.add(family, null, null); + table.put(p); + table.flushCommits(); + + //make sure we get the actual server when doing a direct lookup + ServerName name = server.getServerName(); + assertEquals("Didn't get the expected server from the connection", server, + connection.getHRegionConnection(name.getHostname(), name.getPort())); + + // cleanup + table.close(); + connection.close(); + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java index bdeaefefb943..0c8829132659 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -47,22 +46,25 @@ import java.util.concurrent.Executors; import java.util.concurrent.SynchronousQueue; import java.util.concurrent.ThreadPoolExecutor; -import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; +import org.apache.commons.lang.ArrayUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.DoNotRetryIOException; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HConstants; import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HRegionLocation; import org.apache.hadoop.hbase.HServerAddress; import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; import org.apache.hadoop.hbase.ServerName; -import org.apache.hadoop.hbase.client.HTable.DaemonThreadFactory; import org.apache.hadoop.hbase.client.metrics.ScanMetrics; import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; import org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint; @@ -81,10 +83,14 @@ import org.apache.hadoop.hbase.filter.WhileMatchFilter; import org.apache.hadoop.hbase.io.hfile.BlockCache; import org.apache.hadoop.hbase.io.hfile.CacheConfig; +import org.apache.hadoop.hbase.ipc.SecureRpcEngine; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException; import org.apache.hadoop.hbase.regionserver.Store; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; +import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.io.DataInputBuffer; import org.junit.After; import org.junit.AfterClass; @@ -100,14 +106,15 @@ * Each creates a table named for the method and does its stuff against that. */ @Category(LargeTests.class) +@SuppressWarnings ("deprecation") public class TestFromClientSide { final Log LOG = LogFactory.getLog(getClass()); - private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + protected final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); private static byte [] ROW = Bytes.toBytes("testRow"); private static byte [] FAMILY = Bytes.toBytes("testFamily"); private static byte [] QUALIFIER = Bytes.toBytes("testQualifier"); private static byte [] VALUE = Bytes.toBytes("testValue"); - private static int SLAVES = 3; + protected static int SLAVES = 3; /** * @throws java.lang.Exception @@ -115,6 +122,8 @@ public class TestFromClientSide { @BeforeClass public static void setUpBeforeClass() throws Exception { Configuration conf = TEST_UTIL.getConfiguration(); + // force the rpc engine to the non-secure one in order to get coverage + conf.set("hbase.rpc.engine", "org.apache.hadoop.hbase.ipc.WritableRpcEngine"); conf.setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, MultiRowMutationEndpoint.class.getName()); // We need more than one region server in this test @@ -177,7 +186,7 @@ public void testKeepDeletedCells() throws Exception { p.add(FAMILY, C0, T3); h.put(p); - Delete d = new Delete(T1, ts+2, null); + Delete d = new Delete(T1, ts+3, null); h.delete(d); d = new Delete(T1, ts+3, null); @@ -213,6 +222,72 @@ public void testKeepDeletedCells() throws Exception { h.close(); } + /** + * Basic client side validation of HBASE-10118 + */ + @Test + public void testPurgeFutureDeletes() throws Exception { + final byte[] TABLENAME = Bytes.toBytes("testPurgeFutureDeletes"); + final byte[] ROW = Bytes.toBytes("row"); + final byte[] FAMILY = Bytes.toBytes("family"); + final byte[] COLUMN = Bytes.toBytes("column"); + final byte[] VALUE = Bytes.toBytes("value"); + + HTable table = TEST_UTIL.createTable(TABLENAME, FAMILY); + + // future timestamp + long ts = System.currentTimeMillis() * 2; + Put put = new Put(ROW, ts); + put.add(FAMILY, COLUMN, VALUE); + table.put(put); + + Get get = new Get(ROW); + Result result = table.get(get); + assertArrayEquals(VALUE, result.getValue(FAMILY, COLUMN)); + + Delete del = new Delete(ROW); + del.deleteColumn(FAMILY, COLUMN, ts); + table.delete(del); + + get = new Get(ROW); + result = table.get(get); + assertNull(result.getValue(FAMILY, COLUMN)); + + // major compaction, purged future deletes + TEST_UTIL.getHBaseAdmin().flush(TABLENAME); + TEST_UTIL.getHBaseAdmin().majorCompact(TABLENAME); + + // waiting for the major compaction to complete + int sleepTime = 0; + while (true) { + Thread.sleep(200); + sleepTime += 200; + + Scan s = new Scan(); + s.setRaw(true); + ResultScanner scanner = table.getScanner(s); + if (scanner.next() == null) { + scanner.close(); + break; + } + scanner.close(); + + if (sleepTime > 6000) { + throw new IOException("Major compaction time is larger than 6000s"); + } + } + + put = new Put(ROW, ts); + put.add(FAMILY, COLUMN, VALUE); + table.put(put); + + get = new Get(ROW); + result = table.get(get); + assertArrayEquals(VALUE, result.getValue(FAMILY, COLUMN)); + + table.close(); + } + /** * HBASE-2468 use case 1 and 2: region info de/serialization */ @@ -3610,6 +3685,25 @@ public void testPut() throws IOException { } } + @Test + public void testPutNoCF() throws IOException { + final byte[] BAD_FAM = Bytes.toBytes("BAD_CF"); + final byte[] VAL = Bytes.toBytes(100); + HTable table = TEST_UTIL.createTable(Bytes.toBytes("testPutNoCF"), new byte[][]{FAMILY}); + + boolean caughtNSCFE = false; + + try { + Put p = new Put(ROW); + p.add(BAD_FAM, QUALIFIER, VAL); + table.put(p); + } catch (RetriesExhaustedWithDetailsException e) { + caughtNSCFE = e.getCause(0) instanceof NoSuchColumnFamilyException; + } + assertTrue("Should throw NoSuchColumnFamilyException", caughtNSCFE); + + } + @Test public void testRowsPut() throws IOException { final byte[] CONTENTS_FAMILY = Bytes.toBytes("contents"); @@ -3864,6 +3958,19 @@ public void testListTables() throws IOException, InterruptedException { } } + /** + * creates an HTable for tableName using an unmanaged HConnection. + * + * @param tableName - table to create + * @return the created HTable object + * @throws IOException + */ + HTable createUnmangedHConnectionHTable(final byte [] tableName) throws IOException { + TEST_UTIL.createTable(tableName, HConstants.CATALOG_FAMILY); + HConnection conn = HConnectionManager.createConnection(TEST_UTIL.getConfiguration()); + return (HTable)conn.getTable(tableName); + } + /** * simple test that just executes parts of the client * API that accept a pre-created HConnction instance @@ -3873,18 +3980,41 @@ public void testListTables() throws IOException, InterruptedException { @Test public void testUnmanagedHConnection() throws IOException { final byte[] tableName = Bytes.toBytes("testUnmanagedHConnection"); - TEST_UTIL.createTable(tableName, HConstants.CATALOG_FAMILY); - HConnection conn = HConnectionManager.createConnection(TEST_UTIL - .getConfiguration()); - ExecutorService pool = new ThreadPoolExecutor(1, Integer.MAX_VALUE, - 60, TimeUnit.SECONDS, - new SynchronousQueue(), - new DaemonThreadFactory()); - ((ThreadPoolExecutor)pool).allowCoreThreadTimeOut(true); - HTable t = new HTable(tableName, conn, pool); + HTable t = createUnmangedHConnectionHTable(tableName); + HBaseAdmin ha = new HBaseAdmin(t.getConnection()); + assertTrue(ha.tableExists(tableName)); + assertTrue(t.get(new Get(ROW)).isEmpty()); + } + + /** + * test of that unmanaged HConnections are able to reconnect + * properly (see HBASE-5058) + * + * @throws Exception + */ + @Test + public void testUnmanagedHConnectionReconnect() throws Exception { + final byte[] tableName = Bytes.toBytes("testUnmanagedHConnectionReconnect"); + HTable t = createUnmangedHConnectionHTable(tableName); + HConnection conn = t.getConnection(); HBaseAdmin ha = new HBaseAdmin(conn); assertTrue(ha.tableExists(tableName)); assertTrue(t.get(new Get(ROW)).isEmpty()); + + // stop the master + MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + cluster.stopMaster(0, false); + cluster.waitOnMaster(0); + + // start up a new master + cluster.startMaster(); + assertTrue(cluster.waitForActiveAndReadyMaster()); + + // test that the same unmanaged connection works with a new + // HBaseAdmin and can connect to the new master; + HBaseAdmin newAdmin = new HBaseAdmin(conn); + assertTrue(newAdmin.tableExists(tableName)); + assert(newAdmin.getClusterStatus().getServersSize() == SLAVES); } @Test @@ -4098,7 +4228,7 @@ public void testAppend() throws Exception { byte[] v1 = Bytes.toBytes("42"); byte[] v2 = Bytes.toBytes("23"); byte [][] QUALIFIERS = new byte [][] { - Bytes.toBytes("a"), Bytes.toBytes("b") + Bytes.toBytes("b"), Bytes.toBytes("a") }; Append a = new Append(ROW); a.add(FAMILY, QUALIFIERS[0], v1); @@ -4113,7 +4243,56 @@ public void testAppend() throws Exception { assertEquals(0, Bytes.compareTo(Bytes.add(v1,v2), r.getValue(FAMILY, QUALIFIERS[0]))); assertEquals(0, Bytes.compareTo(Bytes.add(v2,v1), r.getValue(FAMILY, QUALIFIERS[1]))); } - + + @Test + public void testIncrementWithDeletes() throws Exception { + LOG.info("Starting testIncrementWithDeletes"); + final byte [] TABLENAME = Bytes.toBytes("testIncrementWithDeletes"); + HTable ht = TEST_UTIL.createTable(TABLENAME, FAMILY); + final byte[] COLUMN = Bytes.toBytes("column"); + + ht.incrementColumnValue(ROW, FAMILY, COLUMN, 5); + TEST_UTIL.flush(TABLENAME); + + Delete del = new Delete(ROW); + ht.delete(del); + + ht.incrementColumnValue(ROW, FAMILY, COLUMN, 5); + + Get get = new Get(ROW); + Result r = ht.get(get); + assertEquals(1, r.size()); + assertEquals(5, Bytes.toLong(r.getValue(FAMILY, COLUMN))); + } + + @Test + public void testIncrementingInvalidValue() throws Exception { + LOG.info("Starting testIncrementingInvalidValue"); + final byte [] TABLENAME = Bytes.toBytes("testIncrementingInvalidValue"); + HTable ht = TEST_UTIL.createTable(TABLENAME, FAMILY); + final byte[] COLUMN = Bytes.toBytes("column"); + Put p = new Put(ROW); + // write an integer here (not a Long) + p.add(FAMILY, COLUMN, Bytes.toBytes(5)); + ht.put(p); + try { + ht.incrementColumnValue(ROW, FAMILY, COLUMN, 5); + fail("Should have thrown DoNotRetryIOException"); + } catch (DoNotRetryIOException iox) { + // success + } + Increment inc = new Increment(ROW); + inc.addColumn(FAMILY, COLUMN, 5); + try { + ht.increment(inc); + fail("Should have thrown DoNotRetryIOException"); + } catch (DoNotRetryIOException iox) { + // success + } + } + + + @Test public void testIncrement() throws Exception { LOG.info("Starting testIncrement"); @@ -4207,9 +4386,9 @@ public void testPoolBehavior() throws IOException, InterruptedException { // Build a SynchronousQueue that we use for thread coordination final SynchronousQueue queue = new SynchronousQueue(); - List threads = new ArrayList(5); + List tasks = new ArrayList(5); for (int i = 0; i < 5; i++) { - threads.add(new Thread() { + tasks.add(new Runnable() { public void run() { try { // The thread blocks here until we decide to let it go @@ -4218,25 +4397,32 @@ public void run() { } }); } - // First, add two threads and make sure the pool size follows - pool.submit(threads.get(0)); + // First, add two tasks and make sure the pool size follows + pool.submit(tasks.get(0)); assertEquals(1, pool.getPoolSize()); - pool.submit(threads.get(1)); + pool.submit(tasks.get(1)); assertEquals(2, pool.getPoolSize()); - // Next, terminate those threads and then make sure the pool is still the + // Next, terminate those tasks and then make sure the pool is still the // same size queue.put(new Object()); - threads.get(0).join(); queue.put(new Object()); - threads.get(1).join(); assertEquals(2, pool.getPoolSize()); + //ensure that ThreadPoolExecutor knows that tasks are finished. + while (pool.getCompletedTaskCount() < 2) { + Threads.sleep(1); + } + + //ensure that ThreadPoolExecutor knows that threads are finished. + while (pool.getCompletedTaskCount() < 2) { + Threads.sleep(1); + } // Now let's simulate adding a RS meaning that we'll go up to three // concurrent threads. The pool should not grow larger than three. - pool.submit(threads.get(2)); - pool.submit(threads.get(3)); - pool.submit(threads.get(4)); + pool.submit(tasks.get(2)); + pool.submit(tasks.get(3)); + pool.submit(tasks.get(4)); assertEquals(3, pool.getPoolSize()); queue.put(new Object()); queue.put(new Object()); @@ -4256,14 +4442,15 @@ public void testClientPoolRoundRobin() throws IOException { HTable table = TEST_UTIL.createTable(tableName, new byte[][] { FAMILY }, conf, Integer.MAX_VALUE); table.setAutoFlush(true); - Put put = new Put(ROW); - put.add(FAMILY, QUALIFIER, VALUE); + final long ts = EnvironmentEdgeManager.currentTimeMillis(); Get get = new Get(ROW); get.addColumn(FAMILY, QUALIFIER); get.setMaxVersions(); for (int versions = 1; versions <= numVersions; versions++) { + Put put = new Put(ROW); + put.add(FAMILY, QUALIFIER, ts + versions, VALUE); table.put(put); Result result = table.get(get); @@ -4293,14 +4480,15 @@ public void testClientPoolThreadLocal() throws IOException { final HTable table = TEST_UTIL.createTable(tableName, new byte[][] { FAMILY }, conf); table.setAutoFlush(true); - final Put put = new Put(ROW); - put.add(FAMILY, QUALIFIER, VALUE); + final long ts = EnvironmentEdgeManager.currentTimeMillis(); final Get get = new Get(ROW); get.addColumn(FAMILY, QUALIFIER); get.setMaxVersions(); for (int versions = 1; versions <= numVersions; versions++) { + Put put = new Put(ROW); + put.add(FAMILY, QUALIFIER, ts + versions, VALUE); table.put(put); Result result = table.get(get); @@ -4317,14 +4505,16 @@ public void testClientPoolThreadLocal() throws IOException { } final Object waitLock = new Object(); - ExecutorService executorService = Executors.newFixedThreadPool(numVersions); + final AtomicReference error = new AtomicReference(null); for (int versions = numVersions; versions < numVersions * 2; versions++) { final int versionsCopy = versions; executorService.submit(new Callable() { @Override public Void call() { try { + Put put = new Put(ROW); + put.add(FAMILY, QUALIFIER, ts + versionsCopy, VALUE); table.put(put); Result result = table.get(get); @@ -4343,6 +4533,11 @@ public Void call() { waitLock.wait(); } } catch (Exception e) { + } catch (AssertionError e) { + // the error happens in a thread, it won't fail the test, + // need to pass it to the caller for proper handling. + error.set(e); + LOG.error(e); } return null; @@ -4353,9 +4548,9 @@ public Void call() { waitLock.notifyAll(); } executorService.shutdownNow(); + assertNull(error.get()); } - @Test public void testCheckAndPut() throws IOException { final byte [] anotherrow = Bytes.toBytes("anotherrow"); @@ -4401,6 +4596,7 @@ public void testCheckAndPut() throws IOException { * @throws Exception */ @Test + @SuppressWarnings ("unused") public void testScanMetrics() throws Exception { byte [] TABLENAME = Bytes.toBytes("testScanMetrics"); @@ -4413,30 +4609,86 @@ public void testScanMetrics() throws Exception { // Create multiple regions for this table int numOfRegions = TEST_UTIL.createMultiRegions(ht, FAMILY); + // Create 3 rows in the table, with rowkeys starting with "z*" so that + // scan are forced to hit all the regions. + Put put1 = new Put(Bytes.toBytes("z1")); + put1.add(FAMILY, QUALIFIER, VALUE); + Put put2 = new Put(Bytes.toBytes("z2")); + put2.add(FAMILY, QUALIFIER, VALUE); + Put put3 = new Put(Bytes.toBytes("z3")); + put3.add(FAMILY, QUALIFIER, VALUE); + ht.put(Arrays.asList(put1, put2, put3)); Scan scan1 = new Scan(); + int numRecords = 0; for(Result result : ht.getScanner(scan1)) { + numRecords++; } + LOG.info("test data has " + numRecords + " records."); // by default, scan metrics collection is turned off - assertEquals(null, scan1.getAttribute( - Scan.SCAN_ATTRIBUTES_METRICS_DATA)); + assertEquals(null, scan1.getAttribute(Scan.SCAN_ATTRIBUTES_METRICS_DATA)); // turn on scan metrics Scan scan = new Scan(); - scan.setAttribute(Scan.SCAN_ATTRIBUTES_METRICS_ENABLE, - Bytes.toBytes(Boolean.TRUE)); - for(Result result : ht.getScanner(scan)) { + scan.setAttribute(Scan.SCAN_ATTRIBUTES_METRICS_ENABLE, Bytes.toBytes(Boolean.TRUE)); + ResultScanner scanner = ht.getScanner(scan); + // per HBASE-5717, this should still collect even if you don't run all the way to + // the end of the scanner. So this is asking for 2 of the 3 rows we inserted. + for (Result result : scanner.next(numRecords - 1)) { } + scanner.close(); - byte[] serializedMetrics = scan.getAttribute( - Scan.SCAN_ATTRIBUTES_METRICS_DATA); + ScanMetrics scanMetrics = getScanMetrics(scan); + assertEquals("Did not access all the regions in the table", numOfRegions, + scanMetrics.countOfRegions.getCurrentIntervalValue()); + + // set caching to 100 + scan = new Scan(); + scan.setCaching(100); + scan.setAttribute(Scan.SCAN_ATTRIBUTES_METRICS_ENABLE, Bytes.toBytes(Boolean.TRUE)); + scanner = ht.getScanner(scan); + for (Result result : scanner.next(numRecords - 1)) { + } + scanner.close(); + + scanMetrics = getScanMetrics(scan); + assertEquals("Did not access all the regions in the table", numOfRegions, + scanMetrics.countOfRegions.getCurrentIntervalValue()); + + // now, test that the metrics are still collected even if you don't call close, but do + // run past the end of all the records + Scan scanWithoutClose = new Scan(); + scanWithoutClose.setAttribute(Scan.SCAN_ATTRIBUTES_METRICS_ENABLE, Bytes.toBytes(Boolean.TRUE)); + ResultScanner scannerWithoutClose = ht.getScanner(scanWithoutClose); + for (Result result : scannerWithoutClose.next(numRecords + 1)) { + } + ScanMetrics scanMetricsWithoutClose = getScanMetrics(scanWithoutClose); + assertEquals("Did not access all the regions in the table", numOfRegions, + scanMetricsWithoutClose.countOfRegions.getCurrentIntervalValue()); + + // finally, test that the metrics are collected correctly if you both run past all the records, + // AND close the scanner + Scan scanWithClose = new Scan(); + scanWithClose.setAttribute(Scan.SCAN_ATTRIBUTES_METRICS_ENABLE, Bytes.toBytes(Boolean.TRUE)); + ResultScanner scannerWithClose = ht.getScanner(scanWithClose); + for (Result result : scannerWithClose.next(numRecords + 1)) { + } + scannerWithClose.close(); + ScanMetrics scanMetricsWithClose = getScanMetrics(scanWithClose); + assertEquals("Did not access all the regions in the table", numOfRegions, + scanMetricsWithClose.countOfRegions.getCurrentIntervalValue()); + } + + private ScanMetrics getScanMetrics(Scan scan) throws Exception { + byte[] serializedMetrics = scan.getAttribute(Scan.SCAN_ATTRIBUTES_METRICS_DATA); + assertTrue("Serialized metrics were not found.", serializedMetrics != null); DataInputBuffer in = new DataInputBuffer(); in.reset(serializedMetrics, 0, serializedMetrics.length); ScanMetrics scanMetrics = new ScanMetrics(); scanMetrics.readFields(in); - assertEquals(numOfRegions, scanMetrics.countOfRegions.getCurrentIntervalValue()); + return scanMetrics; } /** @@ -4466,6 +4718,20 @@ public void testCacheOnWriteEvictOnClose() throws Exception { long startBlockCount = cache.getBlockCount(); long startBlockHits = cache.getStats().getHitCount(); long startBlockMiss = cache.getStats().getMissCount(); + + // wait till baseline is stable, (minimal 500 ms) + for (int i = 0; i < 5; i++) { + Thread.sleep(100); + if (startBlockCount != cache.getBlockCount() + || startBlockHits != cache.getStats().getHitCount() + || startBlockMiss != cache.getStats().getMissCount()) { + startBlockCount = cache.getBlockCount(); + startBlockHits = cache.getStats().getHitCount(); + startBlockMiss = cache.getStats().getMissCount(); + i = -1; + } + } + // insert data Put put = new Put(ROW); put.add(FAMILY, QUALIFIER, data); @@ -4593,7 +4859,204 @@ public void testNonCachedGetRegionLocation() throws Exception { assertNotNull(addrAfter); assertTrue(addrAfter.getPort() != addrCache.getPort()); assertEquals(addrAfter.getPort(), addrNoCache.getPort()); - } + } + + @Test + /** + * Tests getRegionsInRange by creating some regions over which a range of + * keys spans; then changing the key range. + */ + public void testGetRegionsInRange() throws Exception { + // Test Initialization. + byte [] startKey = Bytes.toBytes("ddc"); + byte [] endKey = Bytes.toBytes("mmm"); + byte [] TABLE = Bytes.toBytes("testGetRegionsInRange"); + HTable table = TEST_UTIL.createTable(TABLE, new byte[][] {FAMILY}, 10); + int numOfRegions = TEST_UTIL.createMultiRegions(table, FAMILY); + assertEquals(25, numOfRegions); + + // Get the regions in this range + List regionsList = table.getRegionsInRange(startKey, + endKey); + assertEquals(10, regionsList.size()); + + // Change the start key + startKey = Bytes.toBytes("fff"); + regionsList = table.getRegionsInRange(startKey, endKey); + assertEquals(7, regionsList.size()); + + // Change the end key + endKey = Bytes.toBytes("nnn"); + regionsList = table.getRegionsInRange(startKey, endKey); + assertEquals(8, regionsList.size()); + + // Empty start key + regionsList = table.getRegionsInRange(HConstants.EMPTY_START_ROW, endKey); + assertEquals(13, regionsList.size()); + + // Empty end key + regionsList = table.getRegionsInRange(startKey, HConstants.EMPTY_END_ROW); + assertEquals(20, regionsList.size()); + + // Both start and end keys empty + regionsList = table.getRegionsInRange(HConstants.EMPTY_START_ROW, + HConstants.EMPTY_END_ROW); + assertEquals(25, regionsList.size()); + + // Change the end key to somewhere in the last block + endKey = Bytes.toBytes("yyz"); + regionsList = table.getRegionsInRange(startKey, endKey); + assertEquals(20, regionsList.size()); + + // Change the start key to somewhere in the first block + startKey = Bytes.toBytes("aac"); + regionsList = table.getRegionsInRange(startKey, endKey); + assertEquals(25, regionsList.size()); + + // Make start and end key the same + startKey = endKey = Bytes.toBytes("ccc"); + regionsList = table.getRegionsInRange(startKey, endKey); + assertEquals(1, regionsList.size()); + } + + @Test + public void testJira6912() throws Exception { + byte [] TABLE = Bytes.toBytes("testJira6912"); + HTable foo = TEST_UTIL.createTable(TABLE, new byte[][] {FAMILY}, 10); + + List puts = new ArrayList(); + for (int i=0;i !=100; i++){ + Put put = new Put(Bytes.toBytes(i)); + put.add(FAMILY, FAMILY, Bytes.toBytes(i)); + puts.add(put); + } + foo.put(puts); + // If i comment this out it works + TEST_UTIL.flush(); + + Scan scan = new Scan(); + scan.setStartRow(Bytes.toBytes(1)); + scan.setStopRow(Bytes.toBytes(3)); + scan.addColumn(FAMILY, FAMILY); + scan.setFilter(new RowFilter(CompareFilter.CompareOp.NOT_EQUAL, new BinaryComparator(Bytes.toBytes(1)))); + + ResultScanner scanner = foo.getScanner(scan); + Result[] bar = scanner.next(100); + assertEquals(1, bar.length); + } + + @Test + public void testRawScanRespectsVersions() throws Exception { + byte[] TABLE = Bytes.toBytes("testRawScan"); + HTable table = TEST_UTIL.createTable(TABLE, new byte[][] { FAMILY }); + byte[] row = Bytes.toBytes("row"); + + // put the same row 4 times, with different values + Put p = new Put(row); + p.add(FAMILY, QUALIFIER, 10, VALUE); + table.put(p); + table.flushCommits(); + + p = new Put(row); + p.add(FAMILY, QUALIFIER, 11, ArrayUtils.add(VALUE, (byte) 2)); + table.put(p); + table.flushCommits(); + + p = new Put(row); + p.add(FAMILY, QUALIFIER, 12, ArrayUtils.add(VALUE, (byte) 3)); + table.put(p); + table.flushCommits(); + + p = new Put(row); + p.add(FAMILY, QUALIFIER, 13, ArrayUtils.add(VALUE, (byte) 4)); + table.put(p); + table.flushCommits(); + + int versions = 4; + Scan s = new Scan(row); + // get all the possible versions + s.setMaxVersions(); + s.setRaw(true); + + ResultScanner scanner = table.getScanner(s); + int count = 0; + for (Result r : scanner) { + assertEquals("Found an unexpected number of results for the row!", versions, r.list().size()); + count++; + } + assertEquals("Found more than a single row when raw scanning the table with a single row!", 1, + count); + scanner.close(); + + // then if we decrease the number of versions, but keep the scan raw, we should see exactly that + // number of versions + versions = 2; + s.setMaxVersions(versions); + scanner = table.getScanner(s); + count = 0; + for (Result r : scanner) { + assertEquals("Found an unexpected number of results for the row!", versions, r.list().size()); + count++; + } + assertEquals("Found more than a single row when raw scanning the table with a single row!", 1, + count); + scanner.close(); + + // finally, if we turn off raw scanning, but max out the number of versions, we should go back + // to seeing just three + versions = 3; + s.setMaxVersions(versions); + scanner = table.getScanner(s); + count = 0; + for (Result r : scanner) { + assertEquals("Found an unexpected number of results for the row!", versions, r.list().size()); + count++; + } + assertEquals("Found more than a single row when raw scanning the table with a single row!", 1, + count); + scanner.close(); + + table.close(); + TEST_UTIL.deleteTable(TABLE); + } + + @Test + public void testSmallScan() throws Exception { + // Test Initialization. + byte[] TABLE = Bytes.toBytes("testSmallScan"); + HTable table = TEST_UTIL.createTable(TABLE, FAMILY); + + // Insert one row each region + int insertNum = 10; + for (int i = 0; i < 10; i++) { + Put put = new Put(Bytes.toBytes("row" + String.format("%03d", i))); + put.add(FAMILY, QUALIFIER, VALUE); + table.put(put); + } + + // nomal scan + ResultScanner scanner = table.getScanner(new Scan()); + int count = 0; + for (Result r : scanner) { + assertTrue(!r.isEmpty()); + count++; + } + assertEquals(insertNum, count); + + // small scan + Scan scan = new Scan(HConstants.EMPTY_START_ROW, HConstants.EMPTY_END_ROW); + scan.setSmall(true); + scan.setCaching(2); + scanner = table.getScanner(scan); + count = 0; + for (Result r : scanner) { + assertTrue(!r.isEmpty()); + count++; + } + assertEquals(insertNum, count); + + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java new file mode 100644 index 000000000000..7dd60de90915 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java @@ -0,0 +1,260 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.util.ArrayList; +import java.util.Random; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HRegionLocation; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.ipc.HRegionInterface; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import com.google.common.collect.Lists; + +@Category(LargeTests.class) +public class TestFromClientSide3 { + final Log LOG = LogFactory.getLog(getClass()); + private final static HBaseTestingUtility TEST_UTIL + = new HBaseTestingUtility(); + private static byte[] ROW = Bytes.toBytes("testRow"); + private static byte[] FAMILY = Bytes.toBytes("testFamily"); + private static byte[] QUALIFIER = Bytes.toBytes("testQualifier"); + private static byte[] VALUE = Bytes.toBytes("testValue"); + private static Random random = new Random(); + private static int SLAVES = 3; + + /** + * @throws java.lang.Exception + */ + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.getConfiguration().setBoolean( + "hbase.online.schema.update.enable", true); + TEST_UTIL.startMiniCluster(SLAVES); + } + + /** + * @throws java.lang.Exception + */ + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * @throws java.lang.Exception + */ + @Before + public void setUp() throws Exception { + // Nothing to do. + } + + /** + * @throws java.lang.Exception + */ + @After + public void tearDown() throws Exception { + // Nothing to do. + } + + private void randomCFPuts(HTable table, byte[] row, byte[] family, int nPuts) + throws Exception { + Put put = new Put(row); + for (int i = 0; i < nPuts; i++) { + byte[] qualifier = Bytes.toBytes(random.nextInt()); + byte[] value = Bytes.toBytes(random.nextInt()); + put.add(family, qualifier, value); + } + table.put(put); + } + + private void performMultiplePutAndFlush(HBaseAdmin admin, HTable table, + byte[] row, byte[] family, int nFlushes, int nPuts) throws Exception { + + // connection needed for poll-wait + HConnection conn = HConnectionManager.getConnection(TEST_UTIL + .getConfiguration()); + HRegionLocation loc = table.getRegionLocation(row, true); + HRegionInterface server = conn.getHRegionConnection(loc.getHostname(), loc + .getPort()); + byte[] regName = loc.getRegionInfo().getRegionName(); + + for (int i = 0; i < nFlushes; i++) { + randomCFPuts(table, row, family, nPuts); + int sfCount = server.getStoreFileList(regName, FAMILY).size(); + + // TODO: replace this api with a synchronous flush after HBASE-2949 + admin.flush(table.getTableName()); + + // synchronously poll wait for a new storefile to appear (flush happened) + while (server.getStoreFileList(regName, FAMILY).size() == sfCount) { + Thread.sleep(40); + } + } + } + + // override the config settings at the CF level and ensure priority + @Test(timeout = 60000) + public void testAdvancedConfigOverride() throws Exception { + /* + * Overall idea: (1) create 3 store files and issue a compaction. config's + * compaction.min == 3, so should work. (2) Increase the compaction.min + * toggle in the HTD to 5 and modify table. If we use the HTD value instead + * of the default config value, adding 3 files and issuing a compaction + * SHOULD NOT work (3) Decrease the compaction.min toggle in the HCD to 2 + * and modify table. The CF schema should override the Table schema and now + * cause a minor compaction. + */ + TEST_UTIL.getConfiguration().setInt("hbase.hstore.compaction.min", 3); + + String tableName = "testAdvancedConfigOverride"; + byte[] TABLE = Bytes.toBytes(tableName); + HTable hTable = TEST_UTIL.createTable(TABLE, FAMILY, 10); + HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + HConnection connection = HConnectionManager.getConnection(TEST_UTIL + .getConfiguration()); + + // Create 3 store files. + byte[] row = Bytes.toBytes(random.nextInt()); + performMultiplePutAndFlush(admin, hTable, row, FAMILY, 3, 100); + + // Verify we have multiple store files. + HRegionLocation loc = hTable.getRegionLocation(row, true); + byte[] regionName = loc.getRegionInfo().getRegionName(); + HRegionInterface server = connection.getHRegionConnection( + loc.getHostname(), loc.getPort()); + assertTrue(server.getStoreFileList(regionName, FAMILY).size() > 1); + + // Issue a compaction request + admin.compact(TABLE); + + // poll wait for the compactions to happen + for (int i = 0; i < 10 * 1000 / 40; ++i) { + // The number of store files after compaction should be lesser. + loc = hTable.getRegionLocation(row, true); + if (!loc.getRegionInfo().isOffline()) { + regionName = loc.getRegionInfo().getRegionName(); + server = connection.getHRegionConnection(loc.getHostname(), loc + .getPort()); + if (server.getStoreFileList(regionName, FAMILY).size() <= 1) { + break; + } + } + Thread.sleep(40); + } + // verify the compactions took place and that we didn't just time out + assertTrue(server.getStoreFileList(regionName, FAMILY).size() <= 1); + + // change the compaction.min config option for this table to 5 + LOG.info("hbase.hstore.compaction.min should now be 5"); + HTableDescriptor htd = new HTableDescriptor(hTable.getTableDescriptor()); + htd.setValue("hbase.hstore.compaction.min", String.valueOf(5)); + admin.modifyTable(TABLE, htd); + Pair st; + while (null != (st = admin.getAlterStatus(TABLE)) && st.getFirst() > 0) { + LOG.debug(st.getFirst() + " regions left to update"); + Thread.sleep(40); + } + LOG.info("alter status finished"); + + // Create 3 more store files. + performMultiplePutAndFlush(admin, hTable, row, FAMILY, 3, 10); + + // Issue a compaction request + admin.compact(TABLE); + + // This time, the compaction request should not happen + Thread.sleep(10 * 1000); + int sfCount = 0; + loc = hTable.getRegionLocation(row, true); + regionName = loc.getRegionInfo().getRegionName(); + server = connection.getHRegionConnection(loc.getHostname(), loc.getPort()); + sfCount = server.getStoreFileList(regionName, FAMILY).size(); + assertTrue(sfCount > 1); + + // change an individual CF's config option to 2 & online schema update + LOG.info("hbase.hstore.compaction.min should now be 2"); + HColumnDescriptor hcd = new HColumnDescriptor(htd.getFamily(FAMILY)); + hcd.setValue("hbase.hstore.compaction.min", String.valueOf(2)); + htd.addFamily(hcd); + admin.modifyTable(TABLE, htd); + while (null != (st = admin.getAlterStatus(TABLE)) && st.getFirst() > 0) { + LOG.debug(st.getFirst() + " regions left to update"); + Thread.sleep(40); + } + LOG.info("alter status finished"); + + // Issue a compaction request + admin.compact(TABLE); + + // poll wait for the compactions to happen + for (int i = 0; i < 10 * 1000 / 40; ++i) { + loc = hTable.getRegionLocation(row, true); + regionName = loc.getRegionInfo().getRegionName(); + try { + server = connection.getHRegionConnection(loc.getHostname(), loc + .getPort()); + if (server.getStoreFileList(regionName, FAMILY).size() < sfCount) { + break; + } + } catch (Exception e) { + LOG.debug("Waiting for region to come online: " + regionName); + } + Thread.sleep(40); + } + // verify the compaction took place and that we didn't just time out + assertTrue(server.getStoreFileList(regionName, FAMILY).size() < sfCount); + + // Finally, ensure that we can remove a custom config value after we made it + LOG.info("Removing CF config value"); + LOG.info("hbase.hstore.compaction.min should now be 5"); + hcd = new HColumnDescriptor(htd.getFamily(FAMILY)); + hcd.setValue("hbase.hstore.compaction.min", null); + htd.addFamily(hcd); + admin.modifyTable(TABLE, htd); + while (null != (st = admin.getAlterStatus(TABLE)) && st.getFirst() > 0) { + LOG.debug(st.getFirst() + " regions left to update"); + Thread.sleep(40); + } + LOG.info("alter status finished"); + assertNull(hTable.getTableDescriptor().getFamily(FAMILY).getValue( + "hbase.hstore.compaction.min")); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideWithCoprocessor.java b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideWithCoprocessor.java new file mode 100644 index 000000000000..7b313dc67fac --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideWithCoprocessor.java @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; +import org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint; +import org.apache.hadoop.hbase.regionserver.NoOpScanPolicyObserver; +import org.junit.BeforeClass; +import org.junit.experimental.categories.Category; + +/** + * Test all client operations with a coprocessor that + * just implements the default flush/compact/scan policy + */ +@Category(LargeTests.class) +public class TestFromClientSideWithCoprocessor extends TestFromClientSide { + @BeforeClass + public static void setUpBeforeClass() throws Exception { + Configuration conf = TEST_UTIL.getConfiguration(); + conf.setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, + MultiRowMutationEndpoint.class.getName(), NoOpScanPolicyObserver.class.getName()); + // We need more than one region server in this test + TEST_UTIL.startMiniCluster(SLAVES); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideWithSecureRpcEngine.java b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideWithSecureRpcEngine.java new file mode 100644 index 000000000000..7fd8f9a8fb9e --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideWithSecureRpcEngine.java @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; +import org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint; +import org.apache.hadoop.hbase.ipc.SecureRpcEngine; +import org.junit.BeforeClass; +import org.junit.experimental.categories.Category; + +/** + * Test all client operations with {@link SecureRpcEngine} + */ +@Category(LargeTests.class) +public class TestFromClientSideWithSecureRpcEngine extends TestFromClientSide { + @BeforeClass + public static void setUpBeforeClass() throws Exception { + Configuration conf = TEST_UTIL.getConfiguration(); + conf.set("hbase.rpc.engine", SecureRpcEngine.class.getName()); + conf.setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, + MultiRowMutationEndpoint.class.getName()); + // We need more than one region server in this test + TEST_UTIL.startMiniCluster(SLAVES); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestGet.java b/src/test/java/org/apache/hadoop/hbase/client/TestGet.java index 93c59ef48997..3af07dbd3d62 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestGet.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestGet.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,23 +19,74 @@ package org.apache.hadoop.hbase.client; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; +import java.io.DataInput; import java.io.DataInputStream; import java.io.DataOutput; import java.io.DataOutputStream; +import java.io.File; +import java.io.FileOutputStream; import java.io.IOException; import java.util.Arrays; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterList; +import org.apache.hadoop.hbase.filter.KeyOnlyFilter; +import org.apache.hadoop.hbase.util.Base64; import org.apache.hadoop.hbase.util.Bytes; import org.junit.Assert; import org.junit.Test; import org.junit.experimental.categories.Category; +import com.google.common.io.ByteStreams; + // TODO: cover more test cases @Category(SmallTests.class) public class TestGet { + + private static final String WRITABLE_GET = + "AgD//////////wAAAAEBD3Rlc3QuTW9ja0ZpbHRlcgEAAAAAAAAAAH//////////AQAAAAAAAAAA"; + + private static final String WRITABLE_GET_WITH_FILTER_LIST = + "AgD//////////wAAAAEBKW9yZy5hcGFjaGUuaGFkb29wLmhiYXNlLmZpbHRlci5GaWx0ZXJMaXN0" + + "AAAAAAMOAA90ZXN0Lk1vY2tGaWx0ZXIOAA1teS5Nb2NrRmlsdGVyDkYAAQAAAAAAAAAAf///////" + + "//8BAAAAAAAAAAA="; + + private static final String MOCK_FILTER_JAR = + "UEsDBBQACAgIADRQI0QAAAAAAAAAAAAAAAAJAAQATUVUQS1JTkYv/soAAAMAUEsHCAAAAAACAAAA" + + "AAAAAFBLAwQUAAgICAA0UCNEAAAAAAAAAAAAAAAAFAAAAE1FVEEtSU5GL01BTklGRVNULk1G803M" + + "y0xLLS7RDUstKs7Mz7NSMNQz4OVyLkpNLElN0XWqBAmY6xnEG1gqaPgXJSbnpCo45xcV5BcllgCV" + + "a/Jy8XIBAFBLBwgxyqRbQwAAAEQAAABQSwMEFAAICAgAcIsiRAAAAAAAAAAAAAAAABMAAABteS9N" + + "b2NrRmlsdGVyLmNsYXNzjVDLTsJAFL1DC5UKIg9B3bkDF4wxcYVx4YOEBGWhYT+0Ix0tnWaY+vgs" + + "VyYu/AA/yng7gAaDibO4r3PuOTfz8fn2DgCHsOtAhkBx8kwvpXffFaHmygGbQEuqMWUx8wJOA+ZL" + + "GdNgxKac3hoOnVFPcUIgdywioU8IWM3WkIB9Jn3uggX5AmQhR6DUFxG/SiYjrm7YKMSNSl96LBwy" + + "JdJ+PrR1IKYpe+maDgFXceZ3BQ99hOvN/h17YFRIes4060VxojuprXvx5PFYCxlNHagQqC5ovcE3" + + "giZMjQ8QXCFCIPuohMZLGsseg0QvTGqrAPS+lonyOF6M26Wf49spG/YAvwbSZ2GFX4LRwY5iJpiz" + + "+6+w9oJFBlyMOTPMwzrGwoyAuYiZwAaUkLWJtY1d2cgcmU1Ef0sUjUR9Bs4l0qoKNeO8ZbB/ipX/" + + "FGsYsW3D3/kCUEsHCEYmW6RQAQAAWgIAAFBLAwQUAAgICABuiyJEAAAAAAAAAAAAAAAAFQAAAHRl" + + "c3QvTW9ja0ZpbHRlci5jbGFzc41Qy07CQBS9A4VKBZGHoO7cgQvHmLjCuPBBQlJloWE/tCMdLZ1m" + + "OlV/y5WJCz/AjzLeDqCRYOIs7uuce87NfHy+vQPAEezakCNQ1TzR9Ep6D30Raq5ssAh0pZpQFjMv" + + "4DRgvpQxDcYs4fTOcOiMeoYTAsUTEQl9SiDf6Y4IWOfS5w7koVSGAhTRwBURv06nY65u2TjEjbor" + + "PRaOmBJZPx9aOhAJgZq7dE+PgKM48/uChz4SWh33nj0yKiS9YJoNojjVvczYuXz2eKyFjBIb6gQa" + + "C9pg+I2gDVOTQwRXiBAoPCmh8Zb2b49hqhcmzVUAet/IVHkcL8bt6s/xBxkb9gA/B7KXxwo/BaON" + + "HcVMMBf2X2HtBYscOBiLZliCdYzlGQFzBTOBDagiaxNrC7uakTk2m4guS1SMRGsGziWyqgFN47xl" + + "sH+K1f4UaxuxbcPf+QJQSwcI8UIYqlEBAABeAgAAUEsBAhQAFAAICAgANFAjRAAAAAACAAAAAAAA" + + "AAkABAAAAAAAAAAAAAAAAAAAAE1FVEEtSU5GL/7KAABQSwECFAAUAAgICAA0UCNEMcqkW0MAAABE" + + "AAAAFAAAAAAAAAAAAAAAAAA9AAAATUVUQS1JTkYvTUFOSUZFU1QuTUZQSwECFAAUAAgICABwiyJE" + + "RiZbpFABAABaAgAAEwAAAAAAAAAAAAAAAADCAAAAbXkvTW9ja0ZpbHRlci5jbGFzc1BLAQIUABQA" + + "CAgIAG6LIkTxQhiqUQEAAF4CAAAVAAAAAAAAAAAAAAAAAFMCAAB0ZXN0L01vY2tGaWx0ZXIuY2xh" + + "c3NQSwUGAAAAAAQABAABAQAA5wMAAAAA"; + @Test public void testAttributesSerialization() throws IOException { Get get = new Get(); @@ -107,6 +157,52 @@ public void testGetAttributes() { Assert.assertNull(get.getAttributesMap().get("attribute1")); } + @Test + public void testDynamicFilter() throws Exception { + Configuration conf = HBaseConfiguration.create(); + String localPath = conf.get("hbase.local.dir") + + File.separator + "jars" + File.separator; + File jarFile = new File(localPath, "MockFilter.jar"); + jarFile.delete(); + assertFalse("Should be deleted: " + jarFile.getPath(), jarFile.exists()); + + DataInput dis = ByteStreams.newDataInput(Base64.decode(WRITABLE_GET)); + Get get = new Get(); + try { + get.readFields(dis); + fail("Should not be able to load the filter class"); + } catch (RuntimeException re) { + String msg = re.getMessage(); + assertTrue(msg != null && msg.contains("Can't find class test.MockFilter")); + } + + dis = ByteStreams.newDataInput(Base64.decode(WRITABLE_GET_WITH_FILTER_LIST)); + try { + get.readFields(dis); + fail("Should not be able to load the filter class"); + } catch (IOException ioe) { + assertTrue(ioe.getCause() instanceof ClassNotFoundException); + } + + FileOutputStream fos = new FileOutputStream(jarFile); + fos.write(Base64.decode(MOCK_FILTER_JAR)); + fos.close(); + + dis = ByteStreams.newDataInput(Base64.decode(WRITABLE_GET)); + get.readFields(dis); + assertEquals("test.MockFilter", get.getFilter().getClass().getName()); + + get = new Get(); + dis = ByteStreams.newDataInput(Base64.decode(WRITABLE_GET_WITH_FILTER_LIST)); + get.readFields(dis); + assertTrue(get.getFilter() instanceof FilterList); + List filters = ((FilterList)get.getFilter()).getFilters(); + assertEquals(3, filters.size()); + assertEquals("test.MockFilter", filters.get(0).getClass().getName()); + assertEquals("my.MockFilter", filters.get(1).getClass().getName()); + assertTrue(filters.get(2) instanceof KeyOnlyFilter); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java b/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java index 9ac4751bd398..8c9b74f4de11 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,27 +18,38 @@ */ package org.apache.hadoop.hbase.client; +import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertEquals; +import static org.junit.Assert.fail; import java.io.IOException; import java.lang.reflect.Field; import java.util.ArrayList; -import java.util.HashSet; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Random; -import java.util.Set; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.SynchronousQueue; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; -import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionLocation; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.ZooKeeperConnectionException; +import org.apache.hadoop.hbase.client.HConnectionManager.HConnectionImplementation; +import org.apache.hadoop.hbase.client.HConnectionManager.HConnectionKey; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Threads; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.junit.AfterClass; import org.junit.Assert; import org.junit.BeforeClass; @@ -54,6 +64,8 @@ public class TestHCM { private static final Log LOG = LogFactory.getLog(TestHCM.class); private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); private static final byte[] TABLE_NAME = Bytes.toBytes("test"); + private static final byte[] TABLE_NAME1 = Bytes.toBytes("test1"); + private static final byte[] TABLE_NAME2 = Bytes.toBytes("test2"); private static final byte[] FAM_NAM = Bytes.toBytes("f"); private static final byte[] ROW = Bytes.toBytes("bbb"); @@ -117,6 +129,83 @@ public static void createNewConfigurations() throws SecurityException, private static int getHConnectionManagerCacheSize(){ return HConnectionTestingUtility.getConnectionCount(); } + + @Test + public void testClusterConnection() throws IOException { + ThreadPoolExecutor otherPool = new ThreadPoolExecutor(1, 1, 5, TimeUnit.SECONDS, + new SynchronousQueue(), Threads.newDaemonThreadFactory("test-hcm")); + + HConnection con1 = HConnectionManager.createConnection(TEST_UTIL.getConfiguration()); + HConnection con2 = HConnectionManager.createConnection(TEST_UTIL.getConfiguration(), otherPool); + // make sure the internally created ExecutorService is the one passed + assertTrue(otherPool == ((HConnectionImplementation) con2).getCurrentBatchPool()); + + String tableName = "testClusterConnection"; + TEST_UTIL.createTable(tableName.getBytes(), FAM_NAM).close(); + HTable t = (HTable) con1.getTable(tableName, otherPool); + // make sure passing a pool to the getTable does not trigger creation of an + // internal pool + assertNull("Internal Thread pool should be null", + ((HConnectionImplementation) con1).getCurrentBatchPool()); + // table should use the pool passed + assertTrue(otherPool == t.getPool()); + t.close(); + + t = (HTable) con2.getTable(tableName); + // table should use the connectin's internal pool + assertTrue(otherPool == t.getPool()); + t.close(); + + t = (HTable) con2.getTable(Bytes.toBytes(tableName)); + // try other API too + assertTrue(otherPool == t.getPool()); + t.close(); + + t = (HTable) con1.getTable(tableName); + ExecutorService pool = ((HConnectionImplementation) con1).getCurrentBatchPool(); + // make sure an internal pool was created + assertNotNull("An internal Thread pool should have been created", pool); + // and that the table is using it + assertTrue(t.getPool() == pool); + t.close(); + + t = (HTable) con1.getTable(tableName); + // still using the *same* internal pool + assertTrue(t.getPool() == pool); + t.close(); + + con1.close(); + // if the pool was created on demand it should be closed upon connectin + // close + assertTrue(pool.isShutdown()); + + con2.close(); + // if the pool is passed, it is not closed + assertFalse(otherPool.isShutdown()); + otherPool.shutdownNow(); + } + + @Test + public void abortingHConnectionRemovesItselfFromHCM() throws Exception { + // Save off current HConnections + Map oldHBaseInstances = + new HashMap(); + oldHBaseInstances.putAll(HConnectionManager.HBASE_INSTANCES); + + HConnectionManager.HBASE_INSTANCES.clear(); + + try { + HConnection connection = HConnectionManager.getConnection(TEST_UTIL.getConfiguration()); + connection.abort("test abortingHConnectionRemovesItselfFromHCM", new Exception( + "test abortingHConnectionRemovesItselfFromHCM")); + Assert.assertNotSame(connection, + HConnectionManager.getConnection(TEST_UTIL.getConfiguration())); + } finally { + // Put original HConnections back + HConnectionManager.HBASE_INSTANCES.clear(); + HConnectionManager.HBASE_INSTANCES.putAll(oldHBaseInstances); + } + } /** * Test that when we delete a location using the first row of a region @@ -136,12 +225,31 @@ public void testRegionCaching() throws Exception{ conn.deleteCachedLocation(TABLE_NAME, ROW); HRegionLocation rl = conn.getCachedLocation(TABLE_NAME, ROW); assertNull("What is this location?? " + rl, rl); - conn.close(); table.close(); } /** - * Make sure that {@link HConfiguration} instances that are essentially the + * Test that Connection or Pool are not closed when managed externally + * @throws Exception + */ + @Test + public void testConnectionManagement() throws Exception{ + TEST_UTIL.createTable(TABLE_NAME1, FAM_NAM); + HConnection conn = HConnectionManager.createConnection(TEST_UTIL.getConfiguration()); + + HTableInterface table = conn.getTable(TABLE_NAME1); + table.close(); + assertFalse(conn.isClosed()); + assertFalse(((HTable)table).getPool().isShutdown()); + table = conn.getTable(TABLE_NAME1); + table.close(); + assertFalse(((HTable)table).getPool().isShutdown()); + conn.close(); + assertTrue(((HTable)table).getPool().isShutdown()); + } + + /** + * Make sure that {@link Configuration} instances that are essentially the * same map to the same {@link HConnection} instance. */ @Test @@ -215,7 +323,7 @@ public void testConnectionUniqueness() throws Exception { } finally { for (HConnection c: connections) { // Clean up connections made so we don't interfere w/ subsequent tests. - HConnectionManager.deleteConnection(c.getConfiguration(), true); + HConnectionManager.deleteConnection(c.getConfiguration()); } } } @@ -274,6 +382,85 @@ public void testCreateConnection() throws Exception { assertTrue(c2 != c3); } + /** + * Tests that a destroyed connection does not have a live zookeeper. + * Below is timing based. We put up a connection to a table and then close the connection while + * having a background thread running that is forcing close of the connection to try and + * provoke a close catastrophe; we are hoping for a car crash so we can see if we are leaking + * zk connections. + * @throws Exception + */ + @Test + public void testDeleteForZKConnLeak() throws Exception { + TEST_UTIL.createTable(TABLE_NAME2, FAM_NAM); + final Configuration config = HBaseConfiguration.create(TEST_UTIL.getConfiguration()); + config.setInt("zookeeper.recovery.retry", 1); + config.setInt("zookeeper.recovery.retry.intervalmill", 1000); + config.setInt("hbase.rpc.timeout", 2000); + config.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 1); + + ThreadPoolExecutor pool = new ThreadPoolExecutor(1, 10, + 5, TimeUnit.SECONDS, + new SynchronousQueue(), + Threads.newDaemonThreadFactory("test-hcm-delete")); + + pool.submit(new Runnable() { + @Override + public void run() { + while (!Thread.interrupted()) { + try { + HConnection conn = HConnectionManager.getConnection(config); + LOG.info("Connection " + conn); + HConnectionManager.deleteStaleConnection(conn); + LOG.info("Connection closed " + conn); + // TODO: This sleep time should be less than the time that it takes to open and close + // a table. Ideally we would do a few runs first to measure. For now this is + // timing based; hopefully we hit the bad condition. + Threads.sleep(10); + } catch (Exception e) { + } + } + } + }); + + // Use connection multiple times. + for (int i = 0; i < 30; i++) { + HConnection c1 = null; + try { + c1 = HConnectionManager.getConnection(config); + LOG.info("HTable connection " + i + " " + c1); + HTable table = new HTable(TABLE_NAME2, c1, pool); + table.close(); + LOG.info("HTable connection " + i + " closed " + c1); + } catch (Exception e) { + LOG.info("We actually want this to happen!!!! So we can see if we are leaking zk", e); + } finally { + if (c1 != null) { + if (c1.isClosed()) { + // cannot use getZooKeeper as method instantiates watcher if null + Field zkwField = c1.getClass().getDeclaredField("zooKeeper"); + zkwField.setAccessible(true); + Object watcher = zkwField.get(c1); + + if (watcher != null) { + if (((ZooKeeperWatcher)watcher).getRecoverableZooKeeper().getState().isAlive()) { + // non-synchronized access to watcher; sleep and check again in case zk connection + // hasn't been cleaned up yet. + Thread.sleep(1000); + if (((ZooKeeperWatcher) watcher).getRecoverableZooKeeper().getState().isAlive()) { + pool.shutdownNow(); + fail("Live zookeeper in closed connection"); + } + } + } + } + c1.close(); + } + } + } + pool.shutdownNow(); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestHConnection.java b/src/test/java/org/apache/hadoop/hbase/client/TestHConnection.java new file mode 100644 index 000000000000..ab0cc09c3013 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestHConnection.java @@ -0,0 +1,163 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.client; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.MultithreadedTestUtil; +import org.apache.hadoop.hbase.MultithreadedTestUtil.RepeatingTestThread; +import org.apache.hadoop.hbase.ZooKeeperConnectionException; +import org.apache.hadoop.hbase.client.HConnectionManager.HConnectionImplementation; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.apache.zookeeper.KeeperException; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Various tests of using HConnection + */ +@Category(MediumTests.class) +public class TestHConnection { + final Log LOG = LogFactory.getLog(getClass()); + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private final long MILLIS_TO_WAIT_FOR_RACE = 1000; + + /** + * @throws java.lang.Exception + */ + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.startMiniCluster(); + } + + /** + * @throws java.lang.Exception + */ + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * Thread that periodically aborts the connection + */ + class AbortThread extends RepeatingTestThread { + final HConnection connection; + + public AbortThread(MultithreadedTestUtil.TestContext ctx, HConnection connection) { + super(ctx); + this.connection = connection; + } + + @Override + public void doAnAction() throws IOException { + connection.abort("test session expired", new KeeperException.SessionExpiredException()); + } + }; + + class HConnectionRaceTester extends HConnectionImplementation { + public HConnectionRaceTester(Configuration configuration, boolean managed) throws ZooKeeperConnectionException { + super(configuration, managed, null); + } + + /** + * Sleep after calling getZookeeperWatcher to attempt to trigger a race condition. + * If the HConnection retrieves the ZooKeeperWatcher but does not cache the value, + * by the time the new watcher is used, it could be null if the connection was aborted. + * This sleep will increase the time between when the watcher is retrieved and when it is used. + */ + public ZooKeeperWatcher getZooKeeperWatcher() throws ZooKeeperConnectionException { + ZooKeeperWatcher zkw = super.getZooKeeperWatcher(); + try { + Thread.sleep(10); + } catch (InterruptedException ie) { + // Ignore + } + return zkw; + } + } + + /** + * Test that a connection that is aborted while calling isTableDisabled doesn't NPE + */ + @Test + public void testTableDisabledRace() throws Exception { + final HConnection connection = new HConnectionRaceTester(TEST_UTIL.getConfiguration(), true); + MultithreadedTestUtil.TestContext ctx = + new MultithreadedTestUtil.TestContext(TEST_UTIL.getConfiguration()); + RepeatingTestThread disabledChecker = new RepeatingTestThread(ctx) { + @Override + public void doAnAction() throws IOException { + try { + connection.isTableDisabled(Bytes.toBytes("tableToCheck")); + } catch (IOException ioe) { + // Ignore. ZK can legitimately fail, only care if we get a NullPointerException + } + } + }; + AbortThread abortThread = new AbortThread(ctx, connection); + + ctx.addThread(disabledChecker); + ctx.addThread(abortThread); + ctx.startThreads(); + ctx.waitFor(MILLIS_TO_WAIT_FOR_RACE); + ctx.stop(); + } + + /** + * Test that a connection that is aborted while calling getCurrentNrNRS doesn't NPE + */ + @Test + public void testGetCurrentNrHRSRace() throws Exception { + final HConnection connection = new HConnectionRaceTester(TEST_UTIL.getConfiguration(), true); + MultithreadedTestUtil.TestContext ctx = + new MultithreadedTestUtil.TestContext(TEST_UTIL.getConfiguration()); + RepeatingTestThread getCurrentNrHRSCaller = new RepeatingTestThread(ctx) { + @Override + public void doAnAction() throws IOException { + try { + connection.getCurrentNrHRS(); + } catch (IOException ioe) { + // Ignore. ZK can legitimately fail, only care if we get a NullPointerException + } + } + }; + AbortThread abortThread = new AbortThread(ctx, connection); + + ctx.addThread(getCurrentNrHRSCaller); + ctx.addThread(abortThread); + ctx.startThreads(); + ctx.waitFor(MILLIS_TO_WAIT_FOR_RACE); + ctx.stop(); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java b/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java index 5ab918ea44fb..0c5a0759d84e 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -21,9 +20,6 @@ import java.io.IOException; -import junit.framework.Assert; -import junit.framework.TestCase; - import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.PoolMap.PoolType; @@ -39,22 +35,23 @@ @Suite.SuiteClasses({TestHTablePool.TestHTableReusablePool.class, TestHTablePool.TestHTableThreadLocalPool.class}) @Category(MediumTests.class) public class TestHTablePool { - private static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); - private final static byte[] TABLENAME = Bytes.toBytes("TestHTablePool"); + private static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private final static byte[] TABLENAME = Bytes.toBytes("TestHTablePool"); + + public abstract static class TestHTablePoolType { @BeforeClass public static void setUpBeforeClass() throws Exception { - TEST_UTIL.startMiniCluster(1); - TEST_UTIL.createTable(TABLENAME, HConstants.CATALOG_FAMILY); - } + TEST_UTIL.startMiniCluster(1); + TEST_UTIL.createTable(TABLENAME, HConstants.CATALOG_FAMILY); + } @AfterClass - public static void tearDownAfterClass() throws Exception { - TEST_UTIL.shutdownMiniCluster(); - } + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } - public abstract static class TestHTablePoolType extends TestCase { - protected abstract PoolType getPoolType(); + protected abstract PoolType getPoolType(); @Test public void testTableWithStringName() throws Exception { @@ -183,22 +180,6 @@ public void testReturnDifferentTable() throws IOException { Assert.assertTrue("alien table rejected", true); } } - - @Test - public void testClassCastException() { - //this test makes sure that client code that - //casts the table it got from pool to HTable won't break - HTablePool pool = new HTablePool(TEST_UTIL.getConfiguration(), - Integer.MAX_VALUE); - String tableName = Bytes.toString(TABLENAME); - try { - // get table and check if type is HTable - HTable table = (HTable) pool.getTable(tableName); - Assert.assertTrue("return type is HTable as expected", true); - } catch (ClassCastException e) { - Assert.fail("return type is not HTable"); - } - } } @Category(MediumTests.class) diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestHTableUtil.java b/src/test/java/org/apache/hadoop/hbase/client/TestHTableUtil.java index 24f878e7d74a..8da61a5cd655 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestHTableUtil.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestHTableUtil.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java b/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java deleted file mode 100644 index 4ac28474239d..000000000000 --- a/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java +++ /dev/null @@ -1,473 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.hbase.client; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.io.IOException; -import java.util.List; - -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.hbase.HColumnDescriptor; -import org.apache.hadoop.hbase.HConstants; -import org.apache.hadoop.hbase.HTableDescriptor; -import org.apache.hadoop.hbase.LargeTests; -import org.apache.hadoop.hbase.regionserver.HRegion; -import org.apache.hadoop.hbase.util.Bytes; -import org.apache.hadoop.hbase.zookeeper.MasterSchemaChangeTracker; -import org.apache.zookeeper.KeeperException; -import org.junit.Test; -import org.junit.experimental.categories.Category; - -@Category(LargeTests.class) -public class TestInstantSchemaChange extends InstantSchemaChangeTestBase { - - final Log LOG = LogFactory.getLog(getClass()); - - @Test - public void testInstantSchemaChangeForModifyTable() throws IOException, - KeeperException, InterruptedException { - - String tableName = "testInstantSchemaChangeForModifyTable"; - conf = TEST_UTIL.getConfiguration(); - LOG.info("Start testInstantSchemaChangeForModifyTable()"); - HTable ht = createTableAndValidate(tableName); - - String newFamily = "newFamily"; - HTableDescriptor htd = new HTableDescriptor(tableName); - htd.addFamily(new HColumnDescriptor(newFamily)); - - admin.modifyTable(Bytes.toBytes(tableName), htd); - waitForSchemaChangeProcess(tableName); - assertFalse(msct.doesSchemaChangeNodeExists(tableName)); - - Put put1 = new Put(row); - put1.add(Bytes.toBytes(newFamily), qualifier, value); - ht.put(put1); - - Get get1 = new Get(row); - get1.addColumn(Bytes.toBytes(newFamily), qualifier); - Result r = ht.get(get1); - byte[] tvalue = r.getValue(Bytes.toBytes(newFamily), qualifier); - int result = Bytes.compareTo(value, tvalue); - assertEquals(result, 0); - LOG.info("END testInstantSchemaChangeForModifyTable()"); - ht.close(); - } - - @Test - public void testInstantSchemaChangeForAddColumn() throws IOException, - KeeperException, InterruptedException { - LOG.info("Start testInstantSchemaChangeForAddColumn() "); - String tableName = "testSchemachangeForAddColumn"; - HTable ht = createTableAndValidate(tableName); - String newFamily = "newFamily"; - HColumnDescriptor hcd = new HColumnDescriptor("newFamily"); - - admin.addColumn(Bytes.toBytes(tableName), hcd); - waitForSchemaChangeProcess(tableName); - assertFalse(msct.doesSchemaChangeNodeExists(tableName)); - - Put put1 = new Put(row); - put1.add(Bytes.toBytes(newFamily), qualifier, value); - LOG.info("******** Put into new column family "); - ht.put(put1); - - Get get1 = new Get(row); - get1.addColumn(Bytes.toBytes(newFamily), qualifier); - Result r = ht.get(get1); - byte[] tvalue = r.getValue(Bytes.toBytes(newFamily), qualifier); - LOG.info(" Value put = " + value + " value from table = " + tvalue); - int result = Bytes.compareTo(value, tvalue); - assertEquals(result, 0); - LOG.info("End testInstantSchemaChangeForAddColumn() "); - ht.close(); - } - - @Test - public void testInstantSchemaChangeForModifyColumn() throws IOException, - KeeperException, InterruptedException { - LOG.info("Start testInstantSchemaChangeForModifyColumn() "); - String tableName = "testSchemachangeForModifyColumn"; - createTableAndValidate(tableName); - - HColumnDescriptor hcd = new HColumnDescriptor(HConstants.CATALOG_FAMILY); - hcd.setMaxVersions(99); - hcd.setBlockCacheEnabled(false); - - admin.modifyColumn(Bytes.toBytes(tableName), hcd); - waitForSchemaChangeProcess(tableName); - assertFalse(msct.doesSchemaChangeNodeExists(tableName)); - - List onlineRegions - = miniHBaseCluster.getRegions(Bytes.toBytes("testSchemachangeForModifyColumn")); - for (HRegion onlineRegion : onlineRegions) { - HTableDescriptor htd = onlineRegion.getTableDesc(); - HColumnDescriptor tableHcd = htd.getFamily(HConstants.CATALOG_FAMILY); - assertTrue(tableHcd.isBlockCacheEnabled() == false); - assertEquals(tableHcd.getMaxVersions(), 99); - } - LOG.info("End testInstantSchemaChangeForModifyColumn() "); - - } - - @Test - public void testInstantSchemaChangeForDeleteColumn() throws IOException, - KeeperException, InterruptedException { - LOG.info("Start testInstantSchemaChangeForDeleteColumn() "); - String tableName = "testSchemachangeForDeleteColumn"; - int numTables = 0; - HTableDescriptor[] tables = admin.listTables(); - if (tables != null) { - numTables = tables.length; - } - - byte[][] FAMILIES = new byte[][] { - Bytes.toBytes("A"), Bytes.toBytes("B"), Bytes.toBytes("C") }; - - HTable ht = TEST_UTIL.createTable(Bytes.toBytes(tableName), - FAMILIES); - tables = this.admin.listTables(); - assertEquals(numTables + 1, tables.length); - LOG.info("Table testSchemachangeForDeleteColumn created"); - - admin.deleteColumn(tableName, "C"); - - waitForSchemaChangeProcess(tableName); - assertFalse(msct.doesSchemaChangeNodeExists(tableName)); - HTableDescriptor modifiedHtd = this.admin.getTableDescriptor(Bytes.toBytes(tableName)); - HColumnDescriptor hcd = modifiedHtd.getFamily(Bytes.toBytes("C")); - assertTrue(hcd == null); - LOG.info("End testInstantSchemaChangeForDeleteColumn() "); - ht.close(); - } - - @Test - public void testInstantSchemaChangeWhenTableIsNotEnabled() throws IOException, - KeeperException { - final String tableName = "testInstantSchemaChangeWhenTableIsDisabled"; - conf = TEST_UTIL.getConfiguration(); - LOG.info("Start testInstantSchemaChangeWhenTableIsDisabled()"); - HTable ht = createTableAndValidate(tableName); - // Disable table - admin.disableTable("testInstantSchemaChangeWhenTableIsDisabled"); - // perform schema changes - HColumnDescriptor hcd = new HColumnDescriptor("newFamily"); - admin.addColumn(Bytes.toBytes(tableName), hcd); - MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - assertTrue(msct.doesSchemaChangeNodeExists(tableName) == false); - ht.close(); - } - - /** - * Test that when concurrent alter requests are received for a table we don't miss any. - * @throws IOException - * @throws KeeperException - * @throws InterruptedException - */ - @Test - public void testConcurrentInstantSchemaChangeForAddColumn() throws IOException, - KeeperException, InterruptedException { - final String tableName = "testConcurrentInstantSchemaChangeForModifyTable"; - conf = TEST_UTIL.getConfiguration(); - LOG.info("Start testConcurrentInstantSchemaChangeForModifyTable()"); - HTable ht = createTableAndValidate(tableName); - - Runnable run1 = new Runnable() { - public void run() { - HColumnDescriptor hcd = new HColumnDescriptor("family1"); - try { - admin.addColumn(Bytes.toBytes(tableName), hcd); - } catch (IOException ioe) { - ioe.printStackTrace(); - - } - } - }; - Runnable run2 = new Runnable() { - public void run() { - HColumnDescriptor hcd = new HColumnDescriptor("family2"); - try { - admin.addColumn(Bytes.toBytes(tableName), hcd); - } catch (IOException ioe) { - ioe.printStackTrace(); - - } - } - }; - - run1.run(); - // We have to add a sleep here as in concurrent scenarios the HTD update - // in HDFS fails and returns with null HTD. This needs to be investigated, - // but it doesn't impact the instant alter functionality in any way. - Thread.sleep(100); - run2.run(); - - waitForSchemaChangeProcess(tableName); - - Put put1 = new Put(row); - put1.add(Bytes.toBytes("family1"), qualifier, value); - ht.put(put1); - - Get get1 = new Get(row); - get1.addColumn(Bytes.toBytes("family1"), qualifier); - Result r = ht.get(get1); - byte[] tvalue = r.getValue(Bytes.toBytes("family1"), qualifier); - int result = Bytes.compareTo(value, tvalue); - assertEquals(result, 0); - Thread.sleep(10000); - - Put put2 = new Put(row); - put2.add(Bytes.toBytes("family2"), qualifier, value); - ht.put(put2); - - Get get2 = new Get(row); - get2.addColumn(Bytes.toBytes("family2"), qualifier); - Result r2 = ht.get(get2); - byte[] tvalue2 = r2.getValue(Bytes.toBytes("family2"), qualifier); - int result2 = Bytes.compareTo(value, tvalue2); - assertEquals(result2, 0); - LOG.info("END testConcurrentInstantSchemaChangeForModifyTable()"); - ht.close(); - } - - /** - * The schema change request blocks while a LB run is in progress. This - * test validates this behavior. - * @throws IOException - * @throws InterruptedException - * @throws KeeperException - */ - @Test - public void testConcurrentInstantSchemaChangeAndLoadBalancerRun() throws IOException, - InterruptedException, KeeperException { - final String tableName = "testInstantSchemaChangeWithLoadBalancerRunning"; - conf = TEST_UTIL.getConfiguration(); - LOG.info("Start testInstantSchemaChangeWithLoadBalancerRunning()"); - final String newFamily = "newFamily"; - HTable ht = createTableAndValidate(tableName); - final MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - - - Runnable balancer = new Runnable() { - public void run() { - // run the balancer now. - miniHBaseCluster.getMaster().balance(); - } - }; - - Runnable schemaChanger = new Runnable() { - public void run() { - HColumnDescriptor hcd = new HColumnDescriptor(newFamily); - try { - admin.addColumn(Bytes.toBytes(tableName), hcd); - } catch (IOException ioe) { - ioe.printStackTrace(); - - } - } - }; - - balancer.run(); - schemaChanger.run(); - waitForSchemaChangeProcess(tableName, 40000); - assertFalse(msct.doesSchemaChangeNodeExists(tableName)); - - Put put1 = new Put(row); - put1.add(Bytes.toBytes(newFamily), qualifier, value); - LOG.info("******** Put into new column family "); - ht.put(put1); - ht.flushCommits(); - - LOG.info("******** Get from new column family "); - Get get1 = new Get(row); - get1.addColumn(Bytes.toBytes(newFamily), qualifier); - Result r = ht.get(get1); - byte[] tvalue = r.getValue(Bytes.toBytes(newFamily), qualifier); - LOG.info(" Value put = " + value + " value from table = " + tvalue); - int result = Bytes.compareTo(value, tvalue); - assertEquals(result, 0); - - LOG.info("End testInstantSchemaChangeWithLoadBalancerRunning() "); - ht.close(); - } - - - /** - * This test validates two things. One is that the LoadBalancer does not run when a schema - * change process is in progress. The second thing is that it also checks that failed/expired - * schema changes are expired to unblock the load balancer run. - * - */ - @Test (timeout=70000) - public void testLoadBalancerBlocksDuringSchemaChangeRequests() throws KeeperException, - IOException, InterruptedException { - LOG.info("Start testConcurrentLoadBalancerSchemaChangeRequests() "); - final MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - // Test that the load balancer does not run while an in-flight schema - // change operation is in progress. - // Simulate a new schema change request. - msct.createSchemaChangeNode("testLoadBalancerBlocks", 0); - // The schema change node is created. - assertTrue(msct.doesSchemaChangeNodeExists("testLoadBalancerBlocks")); - // Now, request an explicit LB run. - - Runnable balancer1 = new Runnable() { - public void run() { - // run the balancer now. - miniHBaseCluster.getMaster().balance(); - } - }; - balancer1.run(); - - // Load balancer should not run now. - assertTrue(miniHBaseCluster.getMaster().isLoadBalancerRunning() == false); - LOG.debug("testConcurrentLoadBalancerSchemaChangeRequests Asserted"); - LOG.info("End testConcurrentLoadBalancerSchemaChangeRequests() "); - } - - /** - * Test that instant schema change blocks while LB is running. - * @throws KeeperException - * @throws IOException - * @throws InterruptedException - */ - @Test (timeout=10000) - public void testInstantSchemaChangeBlocksDuringLoadBalancerRun() throws KeeperException, - IOException, InterruptedException { - final MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - - final String tableName = "testInstantSchemaChangeBlocksDuringLoadBalancerRun"; - conf = TEST_UTIL.getConfiguration(); - LOG.info("Start testInstantSchemaChangeBlocksDuringLoadBalancerRun()"); - final String newFamily = "newFamily"; - createTableAndValidate(tableName); - - // Test that the schema change request does not run while an in-flight LB run - // is in progress. - // First, request an explicit LB run. - - Runnable balancer1 = new Runnable() { - public void run() { - // run the balancer now. - miniHBaseCluster.getMaster().balance(); - } - }; - - Runnable schemaChanger = new Runnable() { - public void run() { - HColumnDescriptor hcd = new HColumnDescriptor(newFamily); - try { - admin.addColumn(Bytes.toBytes(tableName), hcd); - } catch (IOException ioe) { - ioe.printStackTrace(); - - } - } - }; - - Thread t1 = new Thread(balancer1); - Thread t2 = new Thread(schemaChanger); - t1.start(); - t2.start(); - - // check that they both happen concurrently - Runnable balancerCheck = new Runnable() { - public void run() { - // check whether balancer is running. - while(!miniHBaseCluster.getMaster().isLoadBalancerRunning()) { - try { - Thread.sleep(10); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - } - try { - assertFalse(msct.doesSchemaChangeNodeExists("testSchemaChangeBlocks")); - } catch (KeeperException ke) { - ke.printStackTrace(); - } - LOG.debug("Load Balancer is now running or skipped"); - while(miniHBaseCluster.getMaster().isLoadBalancerRunning()) { - try { - Thread.sleep(10); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - } - assertTrue(miniHBaseCluster.getMaster().isLoadBalancerRunning() == false); - try { - assertTrue(msct.doesSchemaChangeNodeExists("testSchemaChangeBlocks")); - } catch (KeeperException ke) { - - } - - } - }; - - Thread t = new Thread(balancerCheck); - t.start(); - t.join(1000); - // Load balancer should not run now. - //assertTrue(miniHBaseCluster.getMaster().isLoadBalancerRunning() == false); - // Schema change request node should now exist. - // assertTrue(msct.doesSchemaChangeNodeExists("testSchemaChangeBlocks")); - LOG.debug("testInstantSchemaChangeBlocksDuringLoadBalancerRun Asserted"); - LOG.info("End testInstantSchemaChangeBlocksDuringLoadBalancerRun() "); - } - - /** - * To test the schema janitor (that it cleans expired/failed schema alter attempts) we - * simply create a fake table (that doesn't exist, with fake number of online regions) in ZK. - * This schema alter request will time out (after 30 seconds) and our janitor will clean it up. - * regions - * @throws IOException - * @throws KeeperException - * @throws InterruptedException - */ - @Test - public void testInstantSchemaJanitor() throws IOException, - KeeperException, InterruptedException { - LOG.info("testInstantSchemaWithFailedExpiredOperations() "); - String fakeTableName = "testInstantSchemaWithFailedExpiredOperations"; - MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - msct.createSchemaChangeNode(fakeTableName, 10); - LOG.debug(msct.getSchemaChangeNodePathForTable(fakeTableName) - + " created"); - Thread.sleep(40000); - assertFalse(msct.doesSchemaChangeNodeExists(fakeTableName)); - LOG.debug(msct.getSchemaChangeNodePathForTable(fakeTableName) - + " deleted"); - LOG.info("END testInstantSchemaWithFailedExpiredOperations() "); - } - - - - @org.junit.Rule - public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = - new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); -} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java b/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java deleted file mode 100644 index c1490eb79ac6..000000000000 --- a/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java +++ /dev/null @@ -1,313 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.hbase.client; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.io.IOException; - -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.HBaseTestingUtility; -import org.apache.hadoop.hbase.HColumnDescriptor; -import org.apache.hadoop.hbase.HConstants; -import org.apache.hadoop.hbase.HTableDescriptor; -import org.apache.hadoop.hbase.LargeTests; -import org.apache.hadoop.hbase.MiniHBaseCluster; -import org.apache.hadoop.hbase.util.Bytes; -import org.apache.hadoop.hbase.zookeeper.MasterSchemaChangeTracker; -import org.apache.hadoop.hbase.zookeeper.ZKUtil; -import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; -import org.apache.zookeeper.KeeperException; -import org.junit.After; -import org.junit.Before; -import org.junit.Ignore; -import org.junit.Test; -import org.junit.experimental.categories.Category; - -@Category(LargeTests.class) -public class TestInstantSchemaChangeFailover { - - final Log LOG = LogFactory.getLog(getClass()); - private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); - private HBaseAdmin admin; - private static MiniHBaseCluster miniHBaseCluster = null; - private Configuration conf; - private ZooKeeperWatcher zkw; - private static MasterSchemaChangeTracker msct = null; - - private final byte [] row = Bytes.toBytes("row"); - private final byte [] qualifier = Bytes.toBytes("qualifier"); - final byte [] value = Bytes.toBytes("value"); - - @Before - public void setUpBeforeClass() throws Exception { - TEST_UTIL.getConfiguration().setInt("hbase.regionserver.msginterval", 100); - TEST_UTIL.getConfiguration().setInt("hbase.client.pause", 250); - TEST_UTIL.getConfiguration().setInt("hbase.client.retries.number", 6); - TEST_UTIL.getConfiguration().setBoolean("hbase.online.schema.update.enable", true); - TEST_UTIL.getConfiguration().setBoolean("hbase.instant.schema.alter.enabled", true); - TEST_UTIL.getConfiguration().setInt("hbase.instant.schema.janitor.period", 10000); - TEST_UTIL.getConfiguration().setInt("hbase.instant.schema.alter.timeout", 30000); - // - miniHBaseCluster = TEST_UTIL.startMiniCluster(2,5); - msct = TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - this.admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); - } - - @After - public void tearDownAfterClass() throws Exception { - TEST_UTIL.shutdownMiniCluster(); - } - - /** - * This a pretty low cost signalling mechanism. It is quite possible that we will - * miss out the ZK node creation signal as in some cases the schema change process - * happens rather quickly and our thread waiting for ZK node creation might wait forver. - * The fool-proof strategy would be to directly listen for ZK events. - * @param tableName - * @throws KeeperException - * @throws InterruptedException - */ - private void waitForSchemaChangeProcess(final String tableName) - throws KeeperException, InterruptedException { - LOG.info("Waiting for ZK node creation for table = " + tableName); - final MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - final Runnable r = new Runnable() { - public void run() { - try { - while(!msct.doesSchemaChangeNodeExists(tableName)) { - try { - Thread.sleep(20); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - } - } catch (KeeperException ke) { - ke.printStackTrace(); - } - - LOG.info("Waiting for ZK node deletion for table = " + tableName); - try { - while(msct.doesSchemaChangeNodeExists(tableName)) { - try { - Thread.sleep(20); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - } - } catch (KeeperException ke) { - ke.printStackTrace(); - } - } - }; - Thread t = new Thread(r); - t.start(); - t.join(10000); - } - - - /** - * Kill a random RS and see that the schema change can succeed. - * @throws IOException - * @throws KeeperException - * @throws InterruptedException - */ - @Test (timeout=50000) - public void testInstantSchemaChangeWhileRSCrash() throws IOException, - KeeperException, InterruptedException { - LOG.info("Start testInstantSchemaChangeWhileRSCrash()"); - zkw = miniHBaseCluster.getMaster().getZooKeeperWatcher(); - - final String tableName = "TestRSCrashDuringSchemaChange"; - HTable ht = createTableAndValidate(tableName); - HColumnDescriptor hcd = new HColumnDescriptor("family2"); - admin.addColumn(Bytes.toBytes(tableName), hcd); - - miniHBaseCluster.getRegionServer(0).abort("Killing while instant schema change"); - // Let the dust settle down - Thread.sleep(10000); - waitForSchemaChangeProcess(tableName); - Put put2 = new Put(row); - put2.add(Bytes.toBytes("family2"), qualifier, value); - ht.put(put2); - - Get get2 = new Get(row); - get2.addColumn(Bytes.toBytes("family2"), qualifier); - Result r2 = ht.get(get2); - byte[] tvalue2 = r2.getValue(Bytes.toBytes("family2"), qualifier); - int result2 = Bytes.compareTo(value, tvalue2); - assertEquals(result2, 0); - String nodePath = msct.getSchemaChangeNodePathForTable("TestRSCrashDuringSchemaChange"); - assertTrue(ZKUtil.checkExists(zkw, nodePath) == -1); - LOG.info("result2 = " + result2); - LOG.info("end testInstantSchemaChangeWhileRSCrash()"); - ht.close(); - } - - /** - * Randomly bring down/up RS servers while schema change is in progress. This test - * is same as the above one but the only difference is that we intent to kill and start - * new RS instances while a schema change is in progress. - * @throws IOException - * @throws KeeperException - * @throws InterruptedException - */ - @Test (timeout=70000) - public void testInstantSchemaChangeWhileRandomRSCrashAndStart() throws IOException, - KeeperException, InterruptedException { - LOG.info("Start testInstantSchemaChangeWhileRandomRSCrashAndStart()"); - miniHBaseCluster.getRegionServer(4).abort("Killing RS 4"); - // Start a new RS before schema change . - // Commenting the start RS as it is failing with DFS user permission NPE. - //miniHBaseCluster.startRegionServer(); - - // Let the dust settle - Thread.sleep(10000); - final String tableName = "testInstantSchemaChangeWhileRandomRSCrashAndStart"; - HTable ht = createTableAndValidate(tableName); - HColumnDescriptor hcd = new HColumnDescriptor("family2"); - admin.addColumn(Bytes.toBytes(tableName), hcd); - // Kill 2 RS now. - miniHBaseCluster.getRegionServer(2).abort("Killing RS 2"); - // Let the dust settle - Thread.sleep(10000); - // We will be left with only one RS. - waitForSchemaChangeProcess(tableName); - assertFalse(msct.doesSchemaChangeNodeExists(tableName)); - Put put2 = new Put(row); - put2.add(Bytes.toBytes("family2"), qualifier, value); - ht.put(put2); - - Get get2 = new Get(row); - get2.addColumn(Bytes.toBytes("family2"), qualifier); - Result r2 = ht.get(get2); - byte[] tvalue2 = r2.getValue(Bytes.toBytes("family2"), qualifier); - int result2 = Bytes.compareTo(value, tvalue2); - assertEquals(result2, 0); - LOG.info("result2 = " + result2); - LOG.info("end testInstantSchemaChangeWhileRandomRSCrashAndStart()"); - ht.close(); - } - - /** - * Test scenario where primary master is brought down while processing an - * alter request. This is harder one as it is very difficult the time this. - * @throws IOException - * @throws KeeperException - * @throws InterruptedException - */ - - @Test (timeout=50000) - public void testInstantSchemaChangeWhileMasterFailover() throws IOException, - KeeperException, InterruptedException { - LOG.info("Start testInstantSchemaChangeWhileMasterFailover()"); - //Thread.sleep(5000); - - final String tableName = "testInstantSchemaChangeWhileMasterFailover"; - HTable ht = createTableAndValidate(tableName); - HColumnDescriptor hcd = new HColumnDescriptor("family2"); - admin.addColumn(Bytes.toBytes(tableName), hcd); - // Kill primary master now. - Thread.sleep(50); - miniHBaseCluster.getMaster().abort("Aborting master now", new Exception("Schema exception")); - - // It may not be possible for us to check the schema change status - // using waitForSchemaChangeProcess as our ZK session in MasterSchemachangeTracker will be - // lost when master dies and hence may not be accurate. So relying on old-fashioned - // sleep here. - Thread.sleep(25000); - Put put2 = new Put(row); - put2.add(Bytes.toBytes("family2"), qualifier, value); - ht.put(put2); - - Get get2 = new Get(row); - get2.addColumn(Bytes.toBytes("family2"), qualifier); - Result r2 = ht.get(get2); - byte[] tvalue2 = r2.getValue(Bytes.toBytes("family2"), qualifier); - int result2 = Bytes.compareTo(value, tvalue2); - assertEquals(result2, 0); - LOG.info("result2 = " + result2); - LOG.info("end testInstantSchemaChangeWhileMasterFailover()"); - ht.close(); - } - - /** - * TEst the master fail over during a schema change request in ZK. - * We create a fake schema change request in ZK and abort the primary master - * mid-flight to simulate a master fail over scenario during a mid-flight - * schema change process. The new master's schema janitor will eventually - * cleanup this fake request after time out. - * @throws IOException - * @throws KeeperException - * @throws InterruptedException - */ - @Ignore - @Test - public void testInstantSchemaOperationsInZKForMasterFailover() throws IOException, - KeeperException, InterruptedException { - LOG.info("testInstantSchemaOperationsInZKForMasterFailover() "); - String tableName = "testInstantSchemaOperationsInZKForMasterFailover"; - - conf = TEST_UTIL.getConfiguration(); - MasterSchemaChangeTracker activesct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - activesct.createSchemaChangeNode(tableName, 10); - LOG.debug(activesct.getSchemaChangeNodePathForTable(tableName) - + " created"); - assertTrue(activesct.doesSchemaChangeNodeExists(tableName)); - // Kill primary master now. - miniHBaseCluster.getMaster().abort("Aborting master now", new Exception("Schema exception")); - // wait for 50 secs. This is so that our schema janitor from fail-over master will kick-in and - // cleanup this failed/expired schema change request. - Thread.sleep(50000); - MasterSchemaChangeTracker newmsct = miniHBaseCluster.getMaster().getSchemaChangeTracker(); - assertFalse(newmsct.doesSchemaChangeNodeExists(tableName)); - LOG.debug(newmsct.getSchemaChangeNodePathForTable(tableName) - + " deleted"); - LOG.info("END testInstantSchemaOperationsInZKForMasterFailover() "); - } - - private HTable createTableAndValidate(String tableName) throws IOException { - conf = TEST_UTIL.getConfiguration(); - LOG.info("Start createTableAndValidate()"); - HTableDescriptor[] tables = admin.listTables(); - int numTables = 0; - if (tables != null) { - numTables = tables.length; - } - HTable ht = TEST_UTIL.createTable(Bytes.toBytes(tableName), - HConstants.CATALOG_FAMILY); - tables = this.admin.listTables(); - assertEquals(numTables + 1, tables.length); - LOG.info("created table = " + tableName); - return ht; - } - - - @org.junit.Rule - public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = - new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); -} - - - diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeSplit.java b/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeSplit.java deleted file mode 100644 index 8f3124b8ea41..000000000000 --- a/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeSplit.java +++ /dev/null @@ -1,224 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.hbase.client; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; - -import java.io.IOException; -import java.util.List; - -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.hbase.HColumnDescriptor; -import org.apache.hadoop.hbase.HConstants; -import org.apache.hadoop.hbase.HTableDescriptor; -import org.apache.hadoop.hbase.LargeTests; -import org.apache.hadoop.hbase.io.hfile.Compression; -import org.apache.hadoop.hbase.regionserver.HRegion; -import org.apache.hadoop.hbase.regionserver.HRegionServer; -import org.apache.hadoop.hbase.util.Bytes; -import org.apache.hadoop.hbase.zookeeper.MasterSchemaChangeTracker; -import org.apache.zookeeper.KeeperException; -import org.junit.Test; -import org.junit.experimental.categories.Category; - -@Category(LargeTests.class) -public class TestInstantSchemaChangeSplit extends InstantSchemaChangeTestBase { - - final Log LOG = LogFactory.getLog(getClass()); - - /** - * The objective of the following test is to validate that schema exclusions happen properly. - * When a RS server dies or crashes(?) mid-flight during a schema refresh, we would exclude - * all online regions in that RS, as well as the RS itself from schema change process. - * - * @throws IOException - * @throws KeeperException - * @throws InterruptedException - */ - @Test - public void testInstantSchemaChangeExclusions() throws IOException, - KeeperException, InterruptedException { - MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - LOG.info("Start testInstantSchemaChangeExclusions() "); - String tableName = "testInstantSchemaChangeExclusions"; - HTable ht = createTableAndValidate(tableName); - - HColumnDescriptor hcd = new HColumnDescriptor(HConstants.CATALOG_FAMILY); - hcd.setMaxVersions(99); - hcd.setBlockCacheEnabled(false); - - HRegionServer hrs = findRSWithOnlineRegionFor(tableName); - //miniHBaseCluster.getRegionServer(0).abort("killed for test"); - admin.modifyColumn(Bytes.toBytes(tableName), hcd); - hrs.abort("Aborting for tests"); - hrs.getSchemaChangeTracker().setSleepTimeMillis(20000); - - //admin.modifyColumn(Bytes.toBytes(tableName), hcd); - LOG.debug("Waiting for Schema Change process to complete"); - waitForSchemaChangeProcess(tableName, 15000); - assertEquals(msct.doesSchemaChangeNodeExists(tableName), false); - // Sleep for some time so that our region is reassigned to some other RS - // by master. - Thread.sleep(10000); - List onlineRegions - = miniHBaseCluster.getRegions(Bytes.toBytes("testInstantSchemaChangeExclusions")); - assertTrue(!onlineRegions.isEmpty()); - for (HRegion onlineRegion : onlineRegions) { - HTableDescriptor htd = onlineRegion.getTableDesc(); - HColumnDescriptor tableHcd = htd.getFamily(HConstants.CATALOG_FAMILY); - assertTrue(tableHcd.isBlockCacheEnabled() == false); - assertEquals(tableHcd.getMaxVersions(), 99); - } - LOG.info("End testInstantSchemaChangeExclusions() "); - ht.close(); - } - - /** - * This test validates that when a schema change request fails on the - * RS side, we appropriately register the failure in the Master Schema change - * tracker's node as well as capture the error cause. - * - * Currently an alter request fails if RS fails with an IO exception say due to - * missing or incorrect codec. With instant schema change the same failure happens - * and we register the failure with associated cause and also update the - * monitor status appropriately. - * - * The region(s) will be orphaned in both the cases. - * - */ - @Test - public void testInstantSchemaChangeWhileRSOpenRegionFailure() throws IOException, - KeeperException, InterruptedException { - MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - - LOG.info("Start testInstantSchemaChangeWhileRSOpenRegionFailure() "); - String tableName = "testInstantSchemaChangeWhileRSOpenRegionFailure"; - HTable ht = createTableAndValidate(tableName); - - // create now 100 regions - TEST_UTIL.createMultiRegions(conf, ht, - HConstants.CATALOG_FAMILY, 10); - - // wait for all the regions to be assigned - Thread.sleep(10000); - List onlineRegions - = miniHBaseCluster.getRegions( - Bytes.toBytes("testInstantSchemaChangeWhileRSOpenRegionFailure")); - int size = onlineRegions.size(); - // we will not have any online regions - LOG.info("Size of online regions = " + onlineRegions.size()); - - HColumnDescriptor hcd = new HColumnDescriptor(HConstants.CATALOG_FAMILY); - hcd.setMaxVersions(99); - hcd.setBlockCacheEnabled(false); - hcd.setCompressionType(Compression.Algorithm.SNAPPY); - - admin.modifyColumn(Bytes.toBytes(tableName), hcd); - Thread.sleep(100); - - assertEquals(msct.doesSchemaChangeNodeExists(tableName), true); - Thread.sleep(10000); - // get the current alter status and validate that its failure with appropriate error msg. - MasterSchemaChangeTracker.MasterAlterStatus mas = msct.getMasterAlterStatus(tableName); - assertTrue(mas != null); - assertEquals(mas.getCurrentAlterStatus(), - MasterSchemaChangeTracker.MasterAlterStatus.AlterState.FAILURE); - assertTrue(mas.getErrorCause() != null); - LOG.info("End testInstantSchemaChangeWhileRSOpenRegionFailure() "); - ht.close(); - } - - @Test - public void testConcurrentInstantSchemaChangeAndSplit() throws IOException, - InterruptedException, KeeperException { - final String tableName = "testConcurrentInstantSchemaChangeAndSplit"; - conf = TEST_UTIL.getConfiguration(); - LOG.info("Start testConcurrentInstantSchemaChangeAndSplit()"); - final String newFamily = "newFamily"; - HTable ht = createTableAndValidate(tableName); - final MasterSchemaChangeTracker msct = - TEST_UTIL.getHBaseCluster().getMaster().getSchemaChangeTracker(); - - // create now 10 regions - TEST_UTIL.createMultiRegions(conf, ht, - HConstants.CATALOG_FAMILY, 4); - int rowCount = TEST_UTIL.loadTable(ht, HConstants.CATALOG_FAMILY); - //assertRowCount(t, rowCount); - - Runnable splitter = new Runnable() { - public void run() { - // run the splits now. - try { - LOG.info("Splitting table now "); - admin.split(Bytes.toBytes(tableName)); - } catch (IOException e) { - e.printStackTrace(); - } catch (InterruptedException e) { - e.printStackTrace(); - } - } - }; - - Runnable schemaChanger = new Runnable() { - public void run() { - HColumnDescriptor hcd = new HColumnDescriptor(newFamily); - try { - admin.addColumn(Bytes.toBytes(tableName), hcd); - } catch (IOException ioe) { - ioe.printStackTrace(); - - } - } - }; - schemaChanger.run(); - Thread.sleep(50); - splitter.run(); - waitForSchemaChangeProcess(tableName, 40000); - - Put put1 = new Put(row); - put1.add(Bytes.toBytes(newFamily), qualifier, value); - LOG.info("******** Put into new column family "); - ht.put(put1); - ht.flushCommits(); - - LOG.info("******** Get from new column family "); - Get get1 = new Get(row); - get1.addColumn(Bytes.toBytes(newFamily), qualifier); - Result r = ht.get(get1); - byte[] tvalue = r.getValue(Bytes.toBytes(newFamily), qualifier); - LOG.info(" Value put = " + value + " value from table = " + tvalue); - int result = Bytes.compareTo(value, tvalue); - assertEquals(result, 0); - LOG.info("End testConcurrentInstantSchemaChangeAndSplit() "); - ht.close(); - } - - - - @org.junit.Rule - public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = - new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); -} - - - diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java b/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java index d1c15afd5a1f..8dd6949ad540 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -77,6 +76,10 @@ public static void setUpBeforeClass() throws Exception { FileSystem fs = FileSystem.get(conf); // find where hbase will root itself, so we can copy filesystem there Path hbaseRootDir = TEST_UTIL.getDefaultRootDirPath(); + if (!fs.isDirectory(hbaseRootDir.getParent())) { + // mkdir at first + fs.mkdirs(hbaseRootDir.getParent()); + } doFsCommand(shell, new String [] {"-put", untar.toURI().toString(), hbaseRootDir.toString()}); // See whats in minihdfs. diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestMetaScanner.java b/src/test/java/org/apache/hadoop/hbase/client/TestMetaScanner.java index 55eac79beee8..f83148ce34ab 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestMetaScanner.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestMetaScanner.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,39 +18,55 @@ */ package org.apache.hadoop.hbase.client; +import static org.mockito.Matchers.anyObject; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.math.BigDecimal; +import java.util.List; +import java.util.NavigableMap; +import java.util.Random; + import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.catalog.MetaEditor; import org.apache.hadoop.hbase.util.Bytes; -import org.junit.AfterClass; -import org.junit.BeforeClass; +import org.apache.hadoop.hbase.util.StoppableImplementation; +import org.apache.hadoop.hbase.util.Threads; +import org.apache.hadoop.util.StringUtils; +import org.junit.After; +import org.junit.Assert; import org.junit.Test; import org.junit.experimental.categories.Category; - -import static org.mockito.Mockito.*; - @Category(MediumTests.class) public class TestMetaScanner { final Log LOG = LogFactory.getLog(getClass()); private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); - @BeforeClass - public static void setUpBeforeClass() throws Exception { + public void setUp() throws Exception { TEST_UTIL.startMiniCluster(1); } - /** - * @throws java.lang.Exception - */ - @AfterClass - public static void tearDownAfterClass() throws Exception { + @After + public void tearDown() throws Exception { TEST_UTIL.shutdownMiniCluster(); } @Test public void testMetaScanner() throws Exception { LOG.info("Starting testMetaScanner"); + setUp(); final byte[] TABLENAME = Bytes.toBytes("testMetaScanner"); final byte[] FAMILY = Bytes.toBytes("family"); TEST_UTIL.createTable(TABLENAME, FAMILY); @@ -64,29 +79,29 @@ public void testMetaScanner() throws Exception { Bytes.toBytes("region_b")}); // Make sure all the regions are deployed TEST_UTIL.countRows(table); - - MetaScanner.MetaScannerVisitor visitor = + + MetaScanner.MetaScannerVisitor visitor = mock(MetaScanner.MetaScannerVisitor.class); doReturn(true).when(visitor).processRow((Result)anyObject()); // Scanning the entire table should give us three rows - MetaScanner.metaScan(conf, visitor, TABLENAME); + MetaScanner.metaScan(conf, null, visitor, TABLENAME); verify(visitor, times(3)).processRow((Result)anyObject()); - + // Scanning the table with a specified empty start row should also // give us three META rows reset(visitor); doReturn(true).when(visitor).processRow((Result)anyObject()); MetaScanner.metaScan(conf, visitor, TABLENAME, HConstants.EMPTY_BYTE_ARRAY, 1000); verify(visitor, times(3)).processRow((Result)anyObject()); - + // Scanning the table starting in the middle should give us two rows: // region_a and region_b reset(visitor); doReturn(true).when(visitor).processRow((Result)anyObject()); MetaScanner.metaScan(conf, visitor, TABLENAME, Bytes.toBytes("region_ac"), 1000); verify(visitor, times(2)).processRow((Result)anyObject()); - + // Scanning with a limit of 1 should only give us one row reset(visitor); doReturn(true).when(visitor).processRow((Result)anyObject()); @@ -95,6 +110,139 @@ public void testMetaScanner() throws Exception { table.close(); } + @Test + public void testConcurrentMetaScannerAndCatalogJanitor() throws Throwable { + /* TEST PLAN: start with only one region in a table. Have a splitter + * thread and metascanner threads that continously scan the meta table for regions. + * CatalogJanitor from master will run frequently to clean things up + */ + TEST_UTIL.getConfiguration().setLong("hbase.catalogjanitor.interval", 500); + setUp(); + + final long runtime = 30 * 1000; //30 sec + LOG.info("Starting testConcurrentMetaScannerAndCatalogJanitor"); + final byte[] TABLENAME = Bytes.toBytes("testConcurrentMetaScannerAndCatalogJanitor"); + final byte[] FAMILY = Bytes.toBytes("family"); + TEST_UTIL.createTable(TABLENAME, FAMILY); + final CatalogTracker catalogTracker = mock(CatalogTracker.class); + when(catalogTracker.getConnection()).thenReturn(TEST_UTIL.getHBaseAdmin().getConnection()); + + class RegionMetaSplitter extends StoppableImplementation implements Runnable { + Random random = new Random(); + Throwable ex = null; + @Override + public void run() { + while (!isStopped()) { + try { + List regions = MetaScanner.listAllRegions( + TEST_UTIL.getConfiguration(), false); + + //select a random region + HRegionInfo parent = regions.get(random.nextInt(regions.size())); + if (parent == null || !Bytes.equals(TABLENAME, parent.getTableName())) { + continue; + } + + long startKey = 0, endKey = Long.MAX_VALUE; + byte[] start = parent.getStartKey(); + byte[] end = parent.getEndKey(); + if (!Bytes.equals(HConstants.EMPTY_START_ROW, parent.getStartKey())) { + startKey = Bytes.toLong(parent.getStartKey()); + } + if (!Bytes.equals(HConstants.EMPTY_END_ROW, parent.getEndKey())) { + endKey = Bytes.toLong(parent.getEndKey()); + } + if (startKey == endKey) { + continue; + } + + long midKey = BigDecimal.valueOf(startKey).add(BigDecimal.valueOf(endKey)) + .divideToIntegralValue(BigDecimal.valueOf(2)).longValue(); + + HRegionInfo splita = new HRegionInfo(TABLENAME, + start, + Bytes.toBytes(midKey)); + HRegionInfo splitb = new HRegionInfo(TABLENAME, + Bytes.toBytes(midKey), + end); + + MetaEditor.offlineParentInMeta(catalogTracker, parent, splita, splitb); + Threads.sleep(100); + MetaEditor.addDaughter(catalogTracker, splitb, null); + MetaEditor.addDaughter(catalogTracker, splita, null); + + Threads.sleep(random.nextInt(200)); + } catch (Throwable e) { + ex = e; + Assert.fail(StringUtils.stringifyException(e)); + } + } + } + void rethrowExceptionIfAny() throws Throwable { + if (ex != null) { throw ex; } + } + } + + class MetaScannerVerifier extends StoppableImplementation implements Runnable { + Random random = new Random(); + Throwable ex = null; + @Override + public void run() { + while(!isStopped()) { + try { + NavigableMap regions = + MetaScanner.allTableRegions(TEST_UTIL.getConfiguration(), null, TABLENAME, false); + + LOG.info("-------"); + byte[] lastEndKey = HConstants.EMPTY_START_ROW; + for (HRegionInfo hri: regions.navigableKeySet()) { + long startKey = 0, endKey = Long.MAX_VALUE; + if (!Bytes.equals(HConstants.EMPTY_START_ROW, hri.getStartKey())) { + startKey = Bytes.toLong(hri.getStartKey()); + } + if (!Bytes.equals(HConstants.EMPTY_END_ROW, hri.getEndKey())) { + endKey = Bytes.toLong(hri.getEndKey()); + } + LOG.info("start:" + startKey + " end:" + endKey + " hri:" + hri); + Assert.assertTrue("lastEndKey=" + Bytes.toString(lastEndKey) + ", startKey=" + + Bytes.toString(hri.getStartKey()), Bytes.equals(lastEndKey, hri.getStartKey())); + lastEndKey = hri.getEndKey(); + } + Assert.assertTrue(Bytes.equals(lastEndKey, HConstants.EMPTY_END_ROW)); + LOG.info("-------"); + Threads.sleep(10 + random.nextInt(50)); + } catch (Throwable e) { + ex = e; + Assert.fail(StringUtils.stringifyException(e)); + } + } + } + void rethrowExceptionIfAny() throws Throwable { + if (ex != null) { throw ex; } + } + } + + RegionMetaSplitter regionMetaSplitter = new RegionMetaSplitter(); + MetaScannerVerifier metaScannerVerifier = new MetaScannerVerifier(); + + Thread regionMetaSplitterThread = new Thread(regionMetaSplitter); + Thread metaScannerVerifierThread = new Thread(metaScannerVerifier); + + regionMetaSplitterThread.start(); + metaScannerVerifierThread.start(); + + Threads.sleep(runtime); + + regionMetaSplitter.stop("test finished"); + metaScannerVerifier.stop("test finished"); + + regionMetaSplitterThread.join(); + metaScannerVerifierThread.join(); + + regionMetaSplitter.rethrowExceptionIfAny(); + metaScannerVerifier.rethrowExceptionIfAny(); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java b/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java index c36272f4bcc4..a1e984676923 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -34,11 +33,14 @@ import org.junit.Assert; import org.junit.Before; import org.junit.BeforeClass; +import org.junit.FixMethodOrder; import org.junit.Test; import org.junit.experimental.categories.Category; +import org.junit.runners.MethodSorters; import static org.junit.Assert.*; +@FixMethodOrder(MethodSorters.NAME_ASCENDING) @Category(MediumTests.class) public class TestMultiParallel { private static final Log LOG = LogFactory.getLog(TestMultiParallel.class); @@ -57,6 +59,7 @@ public class TestMultiParallel { UTIL.startMiniCluster(slaves); HTable t = UTIL.createTable(Bytes.toBytes(TEST_TABLE), Bytes.toBytes(FAMILY)); UTIL.createMultiRegions(t, Bytes.toBytes(FAMILY)); + UTIL.waitTableAvailable(Bytes.toBytes(TEST_TABLE), 15 * 1000); t.close(); } @@ -218,6 +221,11 @@ public void testFlushCommitsNoAbort() throws Exception { doTestFlushCommits(false); } + /** + * Set table auto flush to false and test flushing commits + * @param doAbort true if abort one regionserver in the testing + * @throws Exception + */ private void doTestFlushCommits(boolean doAbort) throws Exception { // Load the data LOG.info("get new table"); @@ -232,9 +240,25 @@ private void doTestFlushCommits(boolean doAbort) throws Exception { } LOG.info("puts"); table.flushCommits(); + int liveRScount = UTIL.getMiniHBaseCluster().getLiveRegionServerThreads() + .size(); + assert liveRScount > 0; + JVMClusterUtil.RegionServerThread liveRS = UTIL.getMiniHBaseCluster() + .getLiveRegionServerThreads().get(0); if (doAbort) { LOG.info("Aborted=" + UTIL.getMiniHBaseCluster().abortRegionServer(0)); + // If we waiting for no regions being online after we abort the server, we + // could ensure the master has re-assigned the regions on killed server + // after putting keys successfully, it means the server we abort is dead + // and detected by matser + while (liveRS.getRegionServer().getNumberOfOnlineRegions() != 0) { + Thread.sleep(100); + } + while (UTIL.getMiniHBaseCluster().getLiveRegionServerThreads().size() == liveRScount) { + Thread.sleep(100); + } + // try putting more keys after the abort. same key/qual... just validating // no exceptions thrown puts = constructPutRequests(); @@ -252,23 +276,24 @@ private void doTestFlushCommits(boolean doAbort) throws Exception { List liveRSs = UTIL.getMiniHBaseCluster().getLiveRegionServerThreads(); int count = 0; + int regionCount = 0; for (JVMClusterUtil.RegionServerThread t: liveRSs) { count++; + regionCount += t.getRegionServer().getOnlineRegions().size(); LOG.info("Count=" + count + ", Alive=" + t.getRegionServer()); } LOG.info("Count=" + count); Assert.assertEquals("Server count=" + count + ", abort=" + doAbort, - (doAbort ? 1 : 2), count); - for (JVMClusterUtil.RegionServerThread t: liveRSs) { - int regions = t.getRegionServer().getOnlineRegions().size(); - Assert.assertTrue("Count of regions=" + regions, regions > 10); - } + (doAbort ? (liveRScount - 1) : liveRScount), count); + Assert.assertTrue("Count of regions=" + regionCount, regionCount >= 25); table.close(); LOG.info("done"); } @Test (timeout=300000) - public void testBatchWithPut() throws Exception { + // FIXME: sort test lexicographically to the end. + // otherwise clashed with testFlushCommitsWithAbort + public void testZBatchWithPut() throws Exception { LOG.info("test=testBatchWithPut"); HTable table = new HTable(UTIL.getConfiguration(), TEST_TABLE); @@ -279,7 +304,13 @@ public void testBatchWithPut() throws Exception { validateSizeAndEmpty(results, KEYS.length); if (true) { - UTIL.getMiniHBaseCluster().abortRegionServer(0); + int liveRScount = UTIL.getMiniHBaseCluster().getLiveRegionServerThreads() + .size(); + assert liveRScount > 0; + JVMClusterUtil.RegionServerThread liveRS = UTIL.getMiniHBaseCluster() + .getLiveRegionServerThreads().get(0); + liveRS.getRegionServer().abort("Aborting for tests", + new Exception("testBatchWithPut")); puts = constructPutRequests(); results = table.batch(puts); @@ -416,6 +447,7 @@ public void testBatchWithIncrementAndAppend() throws Exception { validateResult(multiRes[1], QUAL4, Bytes.toBytes("xyz")); validateResult(multiRes[0], QUAL2, Bytes.toBytes(2L)); validateResult(multiRes[0], QUAL3, Bytes.toBytes(1L)); + table.close(); } @Test(timeout=300000) diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestMultipleTimestamps.java b/src/test/java/org/apache/hadoop/hbase/client/TestMultipleTimestamps.java index d479259965e0..e4b29e11b2d2 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestMultipleTimestamps.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestMultipleTimestamps.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java b/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java index 716022d7d40a..550b2e5287af 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -26,10 +25,36 @@ import org.junit.Test; import java.io.IOException; +import java.util.Arrays; import java.util.HashMap; import java.util.List; import java.util.Map; +import org.apache.hadoop.hbase.filter.BinaryComparator; +import org.apache.hadoop.hbase.filter.ColumnCountGetFilter; +import org.apache.hadoop.hbase.filter.ColumnPaginationFilter; +import org.apache.hadoop.hbase.filter.ColumnPrefixFilter; +import org.apache.hadoop.hbase.filter.ColumnRangeFilter; +import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp; +import org.apache.hadoop.hbase.filter.DependentColumnFilter; +import org.apache.hadoop.hbase.filter.FamilyFilter; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterList; +import org.apache.hadoop.hbase.filter.FilterList.Operator; +import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter; +import org.apache.hadoop.hbase.filter.InclusiveStopFilter; +import org.apache.hadoop.hbase.filter.KeyOnlyFilter; +import org.apache.hadoop.hbase.filter.MultipleColumnPrefixFilter; +import org.apache.hadoop.hbase.filter.PageFilter; +import org.apache.hadoop.hbase.filter.PrefixFilter; +import org.apache.hadoop.hbase.filter.QualifierFilter; +import org.apache.hadoop.hbase.filter.RowFilter; +import org.apache.hadoop.hbase.filter.SingleColumnValueFilter; +import org.apache.hadoop.hbase.filter.SingleColumnValueExcludeFilter; +import org.apache.hadoop.hbase.filter.SkipFilter; +import org.apache.hadoop.hbase.filter.TimestampsFilter; +import org.apache.hadoop.hbase.filter.ValueFilter; +import org.apache.hadoop.hbase.filter.WhileMatchFilter; import org.apache.hadoop.hbase.util.Bytes; import org.codehaus.jackson.map.ObjectMapper; @@ -48,6 +73,217 @@ public class TestOperation { private static ObjectMapper mapper = new ObjectMapper(); + private static List TS_LIST = Arrays.asList(2L, 3L, 5L); + private static TimestampsFilter TS_FILTER = new TimestampsFilter(TS_LIST); + private static String STR_TS_FILTER = + TS_FILTER.getClass().getSimpleName() + " (3/3): [2, 3, 5]"; + + private static List L_TS_LIST = + Arrays.asList(0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L); + private static TimestampsFilter L_TS_FILTER = + new TimestampsFilter(L_TS_LIST); + private static String STR_L_TS_FILTER = + L_TS_FILTER.getClass().getSimpleName() + " (5/11): [0, 1, 2, 3, 4]"; + + private static String COL_NAME_1 = "col1"; + private static ColumnPrefixFilter COL_PRE_FILTER = + new ColumnPrefixFilter(COL_NAME_1.getBytes()); + private static String STR_COL_PRE_FILTER = + COL_PRE_FILTER.getClass().getSimpleName() + " " + COL_NAME_1; + + private static String COL_NAME_2 = "col2"; + private static ColumnRangeFilter CR_FILTER = new ColumnRangeFilter( + COL_NAME_1.getBytes(), true, COL_NAME_2.getBytes(), false); + private static String STR_CR_FILTER = CR_FILTER.getClass().getSimpleName() + + " [" + COL_NAME_1 + ", " + COL_NAME_2 + ")"; + + private static int COL_COUNT = 9; + private static ColumnCountGetFilter CCG_FILTER = + new ColumnCountGetFilter(COL_COUNT); + private static String STR_CCG_FILTER = + CCG_FILTER.getClass().getSimpleName() + " " + COL_COUNT; + + private static int LIMIT = 3; + private static int OFFSET = 4; + private static ColumnPaginationFilter CP_FILTER = + new ColumnPaginationFilter(LIMIT, OFFSET); + private static String STR_CP_FILTER = CP_FILTER.getClass().getSimpleName() + + " (" + LIMIT + ", " + OFFSET + ")"; + + private static String STOP_ROW_KEY = "stop"; + private static InclusiveStopFilter IS_FILTER = + new InclusiveStopFilter(STOP_ROW_KEY.getBytes()); + private static String STR_IS_FILTER = + IS_FILTER.getClass().getSimpleName() + " " + STOP_ROW_KEY; + + private static String PREFIX = "prefix"; + private static PrefixFilter PREFIX_FILTER = + new PrefixFilter(PREFIX.getBytes()); + private static String STR_PREFIX_FILTER = "PrefixFilter " + PREFIX; + + private static byte[][] PREFIXES = { + "0".getBytes(), "1".getBytes(), "2".getBytes()}; + private static MultipleColumnPrefixFilter MCP_FILTER = + new MultipleColumnPrefixFilter(PREFIXES); + private static String STR_MCP_FILTER = + MCP_FILTER.getClass().getSimpleName() + " (3/3): [0, 1, 2]"; + + private static byte[][] L_PREFIXES = { + "0".getBytes(), "1".getBytes(), "2".getBytes(), "3".getBytes(), + "4".getBytes(), "5".getBytes(), "6".getBytes(), "7".getBytes()}; + private static MultipleColumnPrefixFilter L_MCP_FILTER = + new MultipleColumnPrefixFilter(L_PREFIXES); + private static String STR_L_MCP_FILTER = + L_MCP_FILTER.getClass().getSimpleName() + " (5/8): [0, 1, 2, 3, 4]"; + + private static int PAGE_SIZE = 9; + private static PageFilter PAGE_FILTER = new PageFilter(PAGE_SIZE); + private static String STR_PAGE_FILTER = + PAGE_FILTER.getClass().getSimpleName() + " " + PAGE_SIZE; + + private static SkipFilter SKIP_FILTER = new SkipFilter(L_TS_FILTER); + private static String STR_SKIP_FILTER = + SKIP_FILTER.getClass().getSimpleName() + " " + STR_L_TS_FILTER; + + private static WhileMatchFilter WHILE_FILTER = + new WhileMatchFilter(L_TS_FILTER); + private static String STR_WHILE_FILTER = + WHILE_FILTER.getClass().getSimpleName() + " " + STR_L_TS_FILTER; + + private static KeyOnlyFilter KEY_ONLY_FILTER = new KeyOnlyFilter(); + private static String STR_KEY_ONLY_FILTER = + KEY_ONLY_FILTER.getClass().getSimpleName(); + + private static FirstKeyOnlyFilter FIRST_KEY_ONLY_FILTER = + new FirstKeyOnlyFilter(); + private static String STR_FIRST_KEY_ONLY_FILTER = + FIRST_KEY_ONLY_FILTER.getClass().getSimpleName(); + + private static CompareOp CMP_OP = CompareOp.EQUAL; + private static byte[] CMP_VALUE = "value".getBytes(); + private static BinaryComparator BC = new BinaryComparator(CMP_VALUE); + private static DependentColumnFilter DC_FILTER = + new DependentColumnFilter(FAMILY, QUALIFIER, true, CMP_OP, BC); + private static String STR_DC_FILTER = String.format( + "%s (%s, %s, %s, %s, %s)", DC_FILTER.getClass().getSimpleName(), + Bytes.toStringBinary(FAMILY), Bytes.toStringBinary(QUALIFIER), true, + CMP_OP.name(), Bytes.toStringBinary(BC.getValue())); + + private static FamilyFilter FAMILY_FILTER = new FamilyFilter(CMP_OP, BC); + private static String STR_FAMILY_FILTER = + FAMILY_FILTER.getClass().getSimpleName() + " (EQUAL, value)"; + + private static QualifierFilter QUALIFIER_FILTER = + new QualifierFilter(CMP_OP, BC); + private static String STR_QUALIFIER_FILTER = + QUALIFIER_FILTER.getClass().getSimpleName() + " (EQUAL, value)"; + + private static RowFilter ROW_FILTER = new RowFilter(CMP_OP, BC); + private static String STR_ROW_FILTER = + ROW_FILTER.getClass().getSimpleName() + " (EQUAL, value)"; + + private static ValueFilter VALUE_FILTER = new ValueFilter(CMP_OP, BC); + private static String STR_VALUE_FILTER = + VALUE_FILTER.getClass().getSimpleName() + " (EQUAL, value)"; + + private static SingleColumnValueFilter SCV_FILTER = + new SingleColumnValueFilter(FAMILY, QUALIFIER, CMP_OP, CMP_VALUE); + private static String STR_SCV_FILTER = String.format("%s (%s, %s, %s, %s)", + SCV_FILTER.getClass().getSimpleName(), Bytes.toStringBinary(FAMILY), + Bytes.toStringBinary(QUALIFIER), CMP_OP.name(), + Bytes.toStringBinary(CMP_VALUE)); + + private static SingleColumnValueExcludeFilter SCVE_FILTER = + new SingleColumnValueExcludeFilter(FAMILY, QUALIFIER, CMP_OP, CMP_VALUE); + private static String STR_SCVE_FILTER = String.format("%s (%s, %s, %s, %s)", + SCVE_FILTER.getClass().getSimpleName(), Bytes.toStringBinary(FAMILY), + Bytes.toStringBinary(QUALIFIER), CMP_OP.name(), + Bytes.toStringBinary(CMP_VALUE)); + + private static FilterList AND_FILTER_LIST = new FilterList( + Operator.MUST_PASS_ALL, Arrays.asList((Filter) TS_FILTER, L_TS_FILTER, + CR_FILTER)); + private static String STR_AND_FILTER_LIST = String.format( + "%s AND (3/3): [%s, %s, %s]", AND_FILTER_LIST.getClass().getSimpleName(), + STR_TS_FILTER, STR_L_TS_FILTER, STR_CR_FILTER); + + private static FilterList OR_FILTER_LIST = new FilterList( + Operator.MUST_PASS_ONE, Arrays.asList((Filter) TS_FILTER, L_TS_FILTER, + CR_FILTER)); + private static String STR_OR_FILTER_LIST = String.format( + "%s OR (3/3): [%s, %s, %s]", AND_FILTER_LIST.getClass().getSimpleName(), + STR_TS_FILTER, STR_L_TS_FILTER, STR_CR_FILTER); + + private static FilterList L_FILTER_LIST = new FilterList( + Arrays.asList((Filter) TS_FILTER, L_TS_FILTER, CR_FILTER, COL_PRE_FILTER, + CCG_FILTER, CP_FILTER, PREFIX_FILTER, PAGE_FILTER)); + private static String STR_L_FILTER_LIST = String.format( + "%s AND (5/8): [%s, %s, %s, %s, %s]", + L_FILTER_LIST.getClass().getSimpleName(), STR_TS_FILTER, STR_L_TS_FILTER, + STR_CR_FILTER, STR_COL_PRE_FILTER, STR_CCG_FILTER, STR_CP_FILTER); + + private static Filter[] FILTERS = { + TS_FILTER, // TimestampsFilter + L_TS_FILTER, // TimestampsFilter + COL_PRE_FILTER, // ColumnPrefixFilter + CP_FILTER, // ColumnPaginationFilter + CR_FILTER, // ColumnRangeFilter + CCG_FILTER, // ColumnCountGetFilter + IS_FILTER, // InclusiveStopFilter + PREFIX_FILTER, // PrefixFilter + PAGE_FILTER, // PageFilter + SKIP_FILTER, // SkipFilter + WHILE_FILTER, // WhileMatchFilter + KEY_ONLY_FILTER, // KeyOnlyFilter + FIRST_KEY_ONLY_FILTER, // FirstKeyOnlyFilter + MCP_FILTER, // MultipleColumnPrefixFilter + L_MCP_FILTER, // MultipleColumnPrefixFilter + DC_FILTER, // DependentColumnFilter + FAMILY_FILTER, // FamilyFilter + QUALIFIER_FILTER, // QualifierFilter + ROW_FILTER, // RowFilter + VALUE_FILTER, // ValueFilter + SCV_FILTER, // SingleColumnValueFilter + SCVE_FILTER, // SingleColumnValueExcludeFilter + AND_FILTER_LIST, // FilterList + OR_FILTER_LIST, // FilterList + L_FILTER_LIST, // FilterList + }; + + private static String[] FILTERS_INFO = { + STR_TS_FILTER, // TimestampsFilter + STR_L_TS_FILTER, // TimestampsFilter + STR_COL_PRE_FILTER, // ColumnPrefixFilter + STR_CP_FILTER, // ColumnPaginationFilter + STR_CR_FILTER, // ColumnRangeFilter + STR_CCG_FILTER, // ColumnCountGetFilter + STR_IS_FILTER, // InclusiveStopFilter + STR_PREFIX_FILTER, // PrefixFilter + STR_PAGE_FILTER, // PageFilter + STR_SKIP_FILTER, // SkipFilter + STR_WHILE_FILTER, // WhileMatchFilter + STR_KEY_ONLY_FILTER, // KeyOnlyFilter + STR_FIRST_KEY_ONLY_FILTER, // FirstKeyOnlyFilter + STR_MCP_FILTER, // MultipleColumnPrefixFilter + STR_L_MCP_FILTER, // MultipleColumnPrefixFilter + STR_DC_FILTER, // DependentColumnFilter + STR_FAMILY_FILTER, // FamilyFilter + STR_QUALIFIER_FILTER, // QualifierFilter + STR_ROW_FILTER, // RowFilter + STR_VALUE_FILTER, // ValueFilter + STR_SCV_FILTER, // SingleColumnValueFilter + STR_SCVE_FILTER, // SingleColumnValueExcludeFilter + STR_AND_FILTER_LIST, // FilterList + STR_OR_FILTER_LIST, // FilterList + STR_L_FILTER_LIST, // FilterList + }; + + static { + assertEquals("The sizes of static arrays do not match: " + + "[FILTERS: %d <=> FILTERS_INFO: %d]", + FILTERS.length, FILTERS_INFO.length); + } + /** * Test the client Operations' JSON encoding to ensure that produced JSON is * parseable and that the details are present and not corrupted. diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestPutDotHas.java b/src/test/java/org/apache/hadoop/hbase/client/TestPutDotHas.java new file mode 100644 index 000000000000..49cfcdc2cc13 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestPutDotHas.java @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(SmallTests.class) +/** + * Addresses HBASE-6047 + * We test put.has call with all of its polymorphic magic + */ +public class TestPutDotHas { + + public static final byte[] ROW_01 = Bytes.toBytes("row-01"); + public static final byte[] QUALIFIER_01 = Bytes.toBytes("qualifier-01"); + public static final byte[] VALUE_01 = Bytes.toBytes("value-01"); + public static final byte[] FAMILY_01 = Bytes.toBytes("family-01"); + public static final long TS = 1234567L; + public Put put = new Put(ROW_01); + + @Before + public void setUp() { + put.add(FAMILY_01, QUALIFIER_01, TS, VALUE_01); + } + + @Test + public void testHasIgnoreValueIgnoreTS() { + Assert.assertTrue(put.has(FAMILY_01, QUALIFIER_01)); + Assert.assertFalse(put.has(QUALIFIER_01, FAMILY_01)); + } + + @Test + public void testHasIgnoreValue() { + Assert.assertTrue(put.has(FAMILY_01, QUALIFIER_01, TS)); + Assert.assertFalse(put.has(FAMILY_01, QUALIFIER_01, TS + 1)); + } + + @Test + public void testHasIgnoreTS() { + Assert.assertTrue(put.has(FAMILY_01, QUALIFIER_01, VALUE_01)); + Assert.assertFalse(put.has(FAMILY_01, VALUE_01, QUALIFIER_01)); + } + + @Test + public void testHas() { + Assert.assertTrue(put.has(FAMILY_01, QUALIFIER_01, TS, VALUE_01)); + // Bad TS + Assert.assertFalse(put.has(FAMILY_01, QUALIFIER_01, TS + 1, VALUE_01)); + // Bad Value + Assert.assertFalse(put.has(FAMILY_01, QUALIFIER_01, TS, QUALIFIER_01)); + // Bad Family + Assert.assertFalse(put.has(QUALIFIER_01, QUALIFIER_01, TS, VALUE_01)); + // Bad Qual + Assert.assertFalse(put.has(FAMILY_01, FAMILY_01, TS, VALUE_01)); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java b/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java new file mode 100644 index 000000000000..68768eeb003f --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java @@ -0,0 +1,277 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.fail; + +import java.io.IOException; +import java.util.HashSet; +import java.util.Set; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.master.MasterFileSystem; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException; +import org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException; +import org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test clone/restore snapshots from the client + */ +@Category(LargeTests.class) +public class TestRestoreSnapshotFromClient { + final Log LOG = LogFactory.getLog(getClass()); + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + private final byte[] FAMILY = Bytes.toBytes("cf"); + + private byte[] emptySnapshot; + private byte[] snapshotName0; + private byte[] snapshotName1; + private byte[] snapshotName2; + private int snapshot0Rows; + private int snapshot1Rows; + private byte[] tableName; + private HBaseAdmin admin; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.getConfiguration().setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + TEST_UTIL.getConfiguration().setBoolean("hbase.online.schema.update.enable", true); + TEST_UTIL.getConfiguration().setInt("hbase.hstore.compactionThreshold", 10); + TEST_UTIL.getConfiguration().setInt("hbase.regionserver.msginterval", 100); + TEST_UTIL.getConfiguration().setInt("hbase.client.pause", 250); + TEST_UTIL.getConfiguration().setInt("hbase.client.retries.number", 6); + TEST_UTIL.getConfiguration().setBoolean( + "hbase.master.enabletable.roundrobin", true); + TEST_UTIL.startMiniCluster(3); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * Initialize the tests with a table filled with some data + * and two snapshots (snapshotName0, snapshotName1) of different states. + * The tableName, snapshotNames and the number of rows in the snapshot are initialized. + */ + @Before + public void setup() throws Exception { + this.admin = TEST_UTIL.getHBaseAdmin(); + + long tid = System.currentTimeMillis(); + tableName = Bytes.toBytes("testtb-" + tid); + emptySnapshot = Bytes.toBytes("emptySnaptb-" + tid); + snapshotName0 = Bytes.toBytes("snaptb0-" + tid); + snapshotName1 = Bytes.toBytes("snaptb1-" + tid); + snapshotName2 = Bytes.toBytes("snaptb2-" + tid); + + // create Table and disable it + SnapshotTestingUtils.createTable(TEST_UTIL, tableName, FAMILY); + admin.disableTable(tableName); + + // take an empty snapshot + admin.snapshot(emptySnapshot, tableName); + + HTable table = new HTable(TEST_UTIL.getConfiguration(), tableName); + // enable table and insert data + admin.enableTable(tableName); + SnapshotTestingUtils.loadData(TEST_UTIL, table, 500, FAMILY); + snapshot0Rows = TEST_UTIL.countRows(table); + admin.disableTable(tableName); + + // take a snapshot + admin.snapshot(snapshotName0, tableName); + + // enable table and insert more data + admin.enableTable(tableName); + SnapshotTestingUtils.loadData(TEST_UTIL, table, 500, FAMILY); + snapshot1Rows = TEST_UTIL.countRows(table); + table.close(); + } + + @After + public void tearDown() throws Exception { + TEST_UTIL.deleteTable(tableName); + SnapshotTestingUtils.deleteAllSnapshots(TEST_UTIL.getHBaseAdmin()); + SnapshotTestingUtils.deleteArchiveDirectory(TEST_UTIL); + } + + @Test + public void testRestoreSnapshot() throws IOException { + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, tableName, snapshot1Rows); + admin.disableTable(tableName); + admin.snapshot(snapshotName1, tableName); + // Restore from snapshot-0 + admin.restoreSnapshot(snapshotName0); + admin.enableTable(tableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, tableName, snapshot0Rows); + + // Restore from emptySnapshot + admin.disableTable(tableName); + admin.restoreSnapshot(emptySnapshot); + admin.enableTable(tableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, tableName, 0); + + // Restore from snapshot-1 + admin.disableTable(tableName); + admin.restoreSnapshot(snapshotName1); + admin.enableTable(tableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, tableName, snapshot1Rows); + } + + @Test + public void testRestoreSchemaChange() throws Exception { + byte[] TEST_FAMILY2 = Bytes.toBytes("cf2"); + + HTable table = new HTable(TEST_UTIL.getConfiguration(), tableName); + + // Add one column family and put some data in it + admin.disableTable(tableName); + admin.addColumn(tableName, new HColumnDescriptor(TEST_FAMILY2)); + admin.enableTable(tableName); + assertEquals(2, table.getTableDescriptor().getFamilies().size()); + HTableDescriptor htd = admin.getTableDescriptor(tableName); + assertEquals(2, htd.getFamilies().size()); + SnapshotTestingUtils.loadData(TEST_UTIL, table, 500, TEST_FAMILY2); + long snapshot2Rows = snapshot1Rows + 500; + assertEquals(snapshot2Rows, TEST_UTIL.countRows(table)); + assertEquals(500, TEST_UTIL.countRows(table, TEST_FAMILY2)); + Set fsFamilies = getFamiliesFromFS(tableName); + assertEquals(2, fsFamilies.size()); + table.close(); + + // Take a snapshot + admin.disableTable(tableName); + admin.snapshot(snapshotName2, tableName); + + // Restore the snapshot (without the cf) + admin.restoreSnapshot(snapshotName0); + assertEquals(1, table.getTableDescriptor().getFamilies().size()); + admin.enableTable(tableName); + try { + TEST_UTIL.countRows(table, TEST_FAMILY2); + fail("family '" + Bytes.toString(TEST_FAMILY2) + "' should not exists"); + } catch (NoSuchColumnFamilyException e) { + // expected + } + assertEquals(snapshot0Rows, TEST_UTIL.countRows(table)); + htd = admin.getTableDescriptor(tableName); + assertEquals(1, htd.getFamilies().size()); + fsFamilies = getFamiliesFromFS(tableName); + assertEquals(1, fsFamilies.size()); + table.close(); + + // Restore back the snapshot (with the cf) + admin.disableTable(tableName); + admin.restoreSnapshot(snapshotName2); + admin.enableTable(tableName); + htd = admin.getTableDescriptor(tableName); + assertEquals(2, htd.getFamilies().size()); + assertEquals(2, table.getTableDescriptor().getFamilies().size()); + assertEquals(500, TEST_UTIL.countRows(table, TEST_FAMILY2)); + assertEquals(snapshot2Rows, TEST_UTIL.countRows(table)); + fsFamilies = getFamiliesFromFS(tableName); + assertEquals(2, fsFamilies.size()); + table.close(); + } + + @Test + public void testCloneSnapshotOfCloned() throws IOException, InterruptedException { + byte[] clonedTableName = Bytes.toBytes("clonedtb-" + System.currentTimeMillis()); + admin.cloneSnapshot(snapshotName0, clonedTableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, clonedTableName, snapshot0Rows); + admin.disableTable(clonedTableName); + admin.snapshot(snapshotName2, clonedTableName); + admin.deleteTable(clonedTableName); + waitCleanerRun(); + + admin.cloneSnapshot(snapshotName2, clonedTableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, clonedTableName, snapshot0Rows); + TEST_UTIL.deleteTable(clonedTableName); + } + + @Test + public void testCloneAndRestoreSnapshot() throws IOException, InterruptedException { + TEST_UTIL.deleteTable(tableName); + waitCleanerRun(); + + admin.cloneSnapshot(snapshotName0, tableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, tableName, snapshot0Rows); + waitCleanerRun(); + + admin.disableTable(tableName); + admin.restoreSnapshot(snapshotName0); + admin.enableTable(tableName); + SnapshotTestingUtils.verifyRowCount(TEST_UTIL, tableName, snapshot0Rows); + } + + @Test + public void testCorruptedSnapshot() throws IOException, InterruptedException { + SnapshotTestingUtils.corruptSnapshot(TEST_UTIL, Bytes.toString(snapshotName0)); + byte[] cloneName = Bytes.toBytes("corruptedClone-" + System.currentTimeMillis()); + try { + admin.cloneSnapshot(snapshotName0, cloneName); + fail("Expected CorruptedSnapshotException, got succeeded cloneSnapshot()"); + } catch (CorruptedSnapshotException e) { + // Got the expected corruption exception. + // check for no references of the cloned table. + assertFalse(admin.tableExists(cloneName)); + } catch (Exception e) { + fail("Expected CorruptedSnapshotException got: " + e); + } + } + + // ========================================================================== + // Helpers + // ========================================================================== + private void waitCleanerRun() throws InterruptedException { + TEST_UTIL.getMiniHBaseCluster().getMaster().getHFileCleaner().choreForTesting(); + } + + private Set getFamiliesFromFS(final byte[] tableName) throws IOException { + MasterFileSystem mfs = TEST_UTIL.getMiniHBaseCluster().getMaster().getMasterFileSystem(); + Set families = new HashSet(); + Path tableDir = HTableDescriptor.getTableDir(mfs.getRootDir(), tableName); + for (Path regionDir: FSUtils.getRegionDirs(mfs.getFileSystem(), tableDir)) { + for (Path familyDir: FSUtils.getFamilyDirs(mfs.getFileSystem(), regionDir)) { + families.add(familyDir.getName()); + } + } + return families; + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestResult.java b/src/test/java/org/apache/hadoop/hbase/client/TestResult.java index f9e29c267e37..685767b971db 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestResult.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestResult.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -21,8 +20,10 @@ package org.apache.hadoop.hbase.client; import junit.framework.TestCase; + import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.io.ImmutableBytesWritable; import org.apache.hadoop.hbase.util.Bytes; import org.junit.experimental.categories.Category; @@ -122,8 +123,27 @@ public void testCompareResults() throws Exception { } } + /** + * Verify that Result.getBytes(...) behaves correctly. + */ + public void testResultGetBytes() throws Exception { + byte [] value1 = Bytes.toBytes("value1"); + byte [] qual = Bytes.toBytes("qual"); + + KeyValue kv1 = new KeyValue(row, family, qual, value); + KeyValue kv2 = new KeyValue(row, family, qual, value1); + + Result r1 = new Result(new KeyValue[] {kv1, kv2}); + + ImmutableBytesWritable bytes = r1.getBytes(); + assertNotNull(bytes); + + Result r2 = new Result(bytes); + // no exception thrown + Result.compareResults(r1, r2); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); } - diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestScan.java b/src/test/java/org/apache/hadoop/hbase/client/TestScan.java index cdb40bb526d5..15a6d8da72a7 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestScan.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestScan.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java b/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java index 1f9358324973..3fcd3f29b91b 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -25,7 +24,9 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.LargeTests; import org.apache.hadoop.hbase.catalog.MetaReader; import org.apache.hadoop.hbase.regionserver.HRegionServer; import org.apache.hadoop.hbase.util.Bytes; @@ -50,7 +51,7 @@ public class TestScannerTimeout { private final static int NB_ROWS = 10; // Be careful w/ what you set this timer too... it can get in the way of // the mini cluster coming up -- the verification in particular. - private final static int SCANNER_TIMEOUT = 10000; + private final static int SCANNER_TIMEOUT = 15000; private final static int SCANNER_CACHING = 5; /** @@ -84,7 +85,7 @@ public static void tearDownAfterClass() throws Exception { */ @Before public void setUp() throws Exception { - TEST_UTIL.ensureSomeRegionServersAvailable(2); + TEST_UTIL.ensureSomeNonStoppedRegionServersAvailable(2); } /** @@ -147,7 +148,7 @@ public void test2772() throws Exception { LOG.info("END ************ test2772"); } - + /** * Test that scanner won't miss any rows if the region server it was reading * from failed. Before 3686, it would skip rows in the scan. @@ -163,7 +164,14 @@ public void test3686a() throws Exception { scan.setCaching(SCANNER_CACHING); LOG.info("************ TEST3686A"); MetaReader.fullScanMetaAndPrint(TEST_UTIL.getHBaseCluster().getMaster().getCatalogTracker()); - HTable table = new HTable(TEST_UTIL.getConfiguration(), TABLE_NAME); + // Set a very high timeout, we want to test what happens when a RS + // fails but the region is recovered before the lease times out. + // Since the RS is already created, this conf is client-side only for + // this new table + Configuration conf = new Configuration(TEST_UTIL.getConfiguration()); + conf.setInt( + HConstants.HBASE_REGIONSERVER_LEASE_PERIOD_KEY, SCANNER_TIMEOUT*100); + HTable table = new HTable(conf, TABLE_NAME); LOG.info("START ************ TEST3686A---22"); ResultScanner r = table.getScanner(scan); @@ -183,7 +191,7 @@ public void test3686a() throws Exception { table.close(); LOG.info("************ END TEST3686A"); } - + /** * Make sure that no rows are lost if the scanner timeout is longer on the * client than the server, and the scan times out on the server but not the diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestShell.java b/src/test/java/org/apache/hadoop/hbase/client/TestShell.java index 624913bdf87d..f20e3b1b53a6 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestShell.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestShell.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java b/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java new file mode 100644 index 000000000000..af9ad98e8d9a --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java @@ -0,0 +1,151 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.ipc.HMasterInterface; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.snapshot.HSnapshotDescription; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +import com.google.protobuf.RpcController; + +/** + * Test snapshot logic from the client + */ +@Category(SmallTests.class) +public class TestSnapshotFromAdmin { + + private static final Log LOG = LogFactory.getLog(TestSnapshotFromAdmin.class); + + /** + * Test that the logic for doing 'correct' back-off based on exponential increase and the max-time + * passed from the server ensures the correct overall waiting for the snapshot to finish. + * @throws Exception + */ + @Test(timeout = 60000) + public void testBackoffLogic() throws Exception { + final int maxWaitTime = 7500; + final int numRetries = 10; + final int pauseTime = 500; + // calculate the wait time, if we just do straight backoff (ignoring the expected time from + // master) + long ignoreExpectedTime = 0; + for (int i = 0; i < 6; i++) { + ignoreExpectedTime += HConstants.RETRY_BACKOFF[i] * pauseTime; + } + // the correct wait time, capping at the maxTime/tries + fudge room + final long time = pauseTime * 3 + ((maxWaitTime / numRetries) * 3) + 300; + assertTrue("Capped snapshot wait time isn't less that the uncapped backoff time " + + "- further testing won't prove anything.", time < ignoreExpectedTime); + + // setup the mocks + HConnectionManager.HConnectionImplementation mockConnection = Mockito + .mock(HConnectionManager.HConnectionImplementation.class); + Configuration conf = HBaseConfiguration.create(); + // setup the conf to match the expected properties + conf.setInt("hbase.client.retries.number", numRetries); + conf.setLong("hbase.client.pause", pauseTime); + // mock the master admin to our mock + HMasterInterface mockMaster = Mockito.mock(HMasterInterface.class); + Mockito.when(mockConnection.getConfiguration()).thenReturn(conf); + Mockito.when(mockConnection.getMaster()).thenReturn(mockMaster); + // set the max wait time for the snapshot to complete + Mockito + .when( + mockMaster.snapshot( + Mockito.any(HSnapshotDescription.class))).thenReturn((long)maxWaitTime); + + // first five times, we return false, last we get success + Mockito.when( + mockMaster.isSnapshotDone( + Mockito.any(HSnapshotDescription.class))).thenReturn(false, false, + false, false, false, true); + + // setup the admin and run the test + HBaseAdmin admin = new HBaseAdmin(mockConnection); + String snapshot = "snapshot"; + String table = "table"; + // get start time + long start = System.currentTimeMillis(); + admin.snapshot(snapshot, table); + long finish = System.currentTimeMillis(); + long elapsed = (finish - start); + assertTrue("Elapsed time:" + elapsed + " is more than expected max:" + time, elapsed <= time); + admin.close(); + } + + /** + * Make sure that we validate the snapshot name and the table name before we pass anything across + * the wire + * @throws Exception on failure + */ + @Test + public void testValidateSnapshotName() throws Exception { + HConnectionManager.HConnectionImplementation mockConnection = Mockito + .mock(HConnectionManager.HConnectionImplementation.class); + Configuration conf = HBaseConfiguration.create(); + Mockito.when(mockConnection.getConfiguration()).thenReturn(conf); + HBaseAdmin admin = new HBaseAdmin(mockConnection); + SnapshotDescription.Builder builder = SnapshotDescription.newBuilder(); + // check that invalid snapshot names fail + failSnapshotStart(admin, builder.setName(HConstants.SNAPSHOT_DIR_NAME).build()); + failSnapshotStart(admin, builder.setName("-snapshot").build()); + failSnapshotStart(admin, builder.setName("snapshot fails").build()); + failSnapshotStart(admin, builder.setName("snap$hot").build()); + // check the table name also get verified + failSnapshotStart(admin, builder.setName("snapshot").setTable(".table").build()); + failSnapshotStart(admin, builder.setName("snapshot").setTable("-table").build()); + failSnapshotStart(admin, builder.setName("snapshot").setTable("table fails").build()); + failSnapshotStart(admin, builder.setName("snapshot").setTable("tab%le").build()); + + // mock the master connection + HMasterInterface master = Mockito.mock(HMasterInterface.class); + Mockito.when(mockConnection.getMaster()).thenReturn(master); + + Mockito.when( + master.snapshot(Mockito.any(HSnapshotDescription.class))).thenReturn((long)0); + Mockito.when( + master.isSnapshotDone( + Mockito.any(HSnapshotDescription.class))).thenReturn(true); + + // make sure that we can use valid names + admin.snapshot(builder.setName("snapshot").setTable("table").build()); + } + + private void failSnapshotStart(HBaseAdmin admin, SnapshotDescription snapshot) throws IOException { + try { + admin.snapshot(snapshot); + fail("Snapshot should not have succeed with name:" + snapshot.getName()); + } catch (IllegalArgumentException e) { + LOG.debug("Correctly failed to start snapshot:" + e.getMessage()); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java b/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java new file mode 100644 index 000000000000..c8ff1cd9a011 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java @@ -0,0 +1,227 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import static org.junit.Assert.fail; + +import java.io.IOException; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.TableNotFoundException; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy; +import org.apache.hadoop.hbase.snapshot.HSnapshotDescription; +import org.apache.hadoop.hbase.snapshot.SnapshotCreationException; +import org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.JVMClusterUtil.RegionServerThread; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test create/using/deleting snapshots from the client + *

+ * This is an end-to-end test for the snapshot utility + */ +@Category(LargeTests.class) +public class TestSnapshotFromClient { + private static final Log LOG = LogFactory.getLog(TestSnapshotFromClient.class); + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static final int NUM_RS = 2; + private static final String STRING_TABLE_NAME = "test"; + private static final byte[] TEST_FAM = Bytes.toBytes("fam"); + private static final byte[] TABLE_NAME = Bytes.toBytes(STRING_TABLE_NAME); + + /** + * Setup the config for the cluster + * @throws Exception on failure + */ + @BeforeClass + public static void setupCluster() throws Exception { + setupConf(UTIL.getConfiguration()); + UTIL.startMiniCluster(NUM_RS); + } + + private static void setupConf(Configuration conf) { + // disable the ui + conf.setInt("hbase.regionsever.info.port", -1); + // change the flush size to a small amount, regulating number of store files + conf.setInt("hbase.hregion.memstore.flush.size", 25000); + // so make sure we get a compaction when doing a load, but keep around some + // files in the store + conf.setInt("hbase.hstore.compaction.min", 10); + conf.setInt("hbase.hstore.compactionThreshold", 10); + // block writes if we get to 12 store files + conf.setInt("hbase.hstore.blockingStoreFiles", 12); + // drop the number of attempts for the hbase admin + conf.setInt("hbase.client.retries.number", 1); + // Enable snapshot + conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + // prevent aggressive region split + conf.set(HConstants.HBASE_REGION_SPLIT_POLICY_KEY, + ConstantSizeRegionSplitPolicy.class.getName()); + } + + @Before + public void setup() throws Exception { + UTIL.createTable(TABLE_NAME, TEST_FAM); + } + + @After + public void tearDown() throws Exception { + UTIL.deleteTable(TABLE_NAME); + SnapshotTestingUtils.deleteAllSnapshots(UTIL.getHBaseAdmin()); + SnapshotTestingUtils.deleteArchiveDirectory(UTIL); + } + + @AfterClass + public static void cleanupTest() throws Exception { + try { + UTIL.shutdownMiniCluster(); + } catch (Exception e) { + LOG.warn("failure shutting down cluster", e); + } + } + + /** + * Test snapshotting not allowed .META. and -ROOT- + * @throws Exception + */ + @Test + public void testMetaTablesSnapshot() throws Exception { + HBaseAdmin admin = UTIL.getHBaseAdmin(); + byte[] snapshotName = Bytes.toBytes("metaSnapshot"); + + try { + admin.snapshot(snapshotName, HConstants.META_TABLE_NAME); + fail("taking a snapshot of .META. should not be allowed"); + } catch (IllegalArgumentException e) { + // expected + } + + try { + admin.snapshot(snapshotName, HConstants.ROOT_TABLE_NAME); + fail("taking a snapshot of -ROOT- should not be allowed"); + } catch (IllegalArgumentException e) { + // expected + } + } + + /** + * Test snapshotting a table that is offline + * @throws Exception + */ + @Test + public void testOfflineTableSnapshot() throws Exception { + HBaseAdmin admin = UTIL.getHBaseAdmin(); + // make sure we don't fail on listing snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + + // put some stuff in the table + HTable table = new HTable(UTIL.getConfiguration(), TABLE_NAME); + UTIL.loadTable(table, TEST_FAM); + + // get the name of all the regionservers hosting the snapshotted table + Set snapshotServers = new HashSet(); + List servers = UTIL.getMiniHBaseCluster().getLiveRegionServerThreads(); + for (RegionServerThread server : servers) { + if (server.getRegionServer().getOnlineRegions(TABLE_NAME).size() > 0) { + snapshotServers.add(server.getRegionServer().getServerName().toString()); + } + } + + LOG.debug("FS state before disable:"); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + // XXX if this is flakey, might want to consider using the async version and looping as + // disableTable can succeed and still timeout. + admin.disableTable(TABLE_NAME); + + LOG.debug("FS state before snapshot:"); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + + // take a snapshot of the disabled table + byte[] snapshot = Bytes.toBytes("offlineTableSnapshot"); + admin.snapshot(snapshot, TABLE_NAME); + LOG.debug("Snapshot completed."); + + // make sure we have the snapshot + List snapshots = SnapshotTestingUtils.assertOneSnapshotThatMatches(admin, + snapshot, TABLE_NAME); + + // make sure its a valid snapshot + FileSystem fs = UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getFileSystem(); + Path rootDir = UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getRootDir(); + LOG.debug("FS state after snapshot:"); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + + SnapshotTestingUtils.confirmSnapshotValid(snapshots.get(0), TABLE_NAME, TEST_FAM, rootDir, + admin, fs, false, new Path(rootDir, HConstants.HREGION_LOGDIR_NAME), snapshotServers); + + admin.deleteSnapshot(snapshot); + snapshots = admin.listSnapshots(); + SnapshotTestingUtils.assertNoSnapshots(admin); + } + + @Test + public void testSnapshotFailsOnNonExistantTable() throws Exception { + HBaseAdmin admin = UTIL.getHBaseAdmin(); + // make sure we don't fail on listing snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + String tableName = "_not_a_table"; + + // make sure the table doesn't exist + boolean fail = false; + do { + try { + admin.getTableDescriptor(Bytes.toBytes(tableName)); + fail = true; + LOG.error("Table:" + tableName + " already exists, checking a new name"); + tableName = tableName+"!"; + } catch (TableNotFoundException e) { + fail = false; + } + } while (fail); + + // snapshot the non-existant table + try { + admin.snapshot("fail", tableName); + fail("Snapshot succeeded even though there is not table."); + } catch (SnapshotCreationException e) { + LOG.info("Correctly failed to snapshot a non-existant table:" + e.getMessage()); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotsFromAdmin.java b/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotsFromAdmin.java new file mode 100644 index 000000000000..2c1c85eabd1f --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotsFromAdmin.java @@ -0,0 +1,136 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.ipc.HMasterInterface; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.snapshot.HSnapshotDescription; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +import com.google.protobuf.RpcController; + +/** + * Test snapshot logic from the client + */ +@Category(SmallTests.class) +public class TestSnapshotsFromAdmin { + + private static final Log LOG = LogFactory.getLog(TestSnapshotsFromAdmin.class); + + /** + * Test that the logic for doing 'correct' back-off based on exponential increase and the max-time + * passed from the server ensures the correct overall waiting for the snapshot to finish. + * @throws Exception + */ + @Test(timeout = 60000) + public void testBackoffLogic() throws Exception { + final int maxWaitTime = 7500; + final int numRetries = 10; + final int pauseTime = 500; + // calculate the wait time, if we just do straight backoff (ignoring the expected time from + // master) + long ignoreExpectedTime = 0; + for (int i = 0; i < 6; i++) { + ignoreExpectedTime += HConstants.RETRY_BACKOFF[i] * pauseTime; + } + // the correct wait time, capping at the maxTime/tries + fudge room + final long time = pauseTime * 3 + ((maxWaitTime / numRetries) * 3) + 300; + assertTrue("Capped snapshot wait time isn't less that the uncapped backoff time " + + "- further testing won't prove anything.", time < ignoreExpectedTime); + + // setup the mocks + HConnectionManager.HConnectionImplementation mockConnection = Mockito + .mock(HConnectionManager.HConnectionImplementation.class); + Configuration conf = HBaseConfiguration.create(); + // setup the conf to match the expected properties + conf.setInt("hbase.client.retries.number", numRetries); + conf.setLong("hbase.client.pause", pauseTime); + // mock the master admin to our mock + HMasterInterface mockMaster = Mockito.mock(HMasterInterface.class); + Mockito.when(mockConnection.getConfiguration()).thenReturn(conf); + Mockito.when(mockConnection.getMaster()).thenReturn(mockMaster); + // set the max wait time for the snapshot to complete + Mockito + .when( + mockMaster.snapshot( + Mockito.any(HSnapshotDescription.class))).thenReturn((long)maxWaitTime); + // first five times, we return false, last we get success + Mockito.when( + mockMaster.isSnapshotDone( + Mockito.any(HSnapshotDescription.class))).thenReturn(false, false, + false, false, false, true); + + // setup the admin and run the test + HBaseAdmin admin = new HBaseAdmin(mockConnection); + String snapshot = "snasphot"; + String table = "table"; + // get start time + long start = System.currentTimeMillis(); + admin.snapshot(snapshot, table); + long finish = System.currentTimeMillis(); + long elapsed = (finish - start); + assertTrue("Elapsed time:" + elapsed + " is more than expected max:" + time, elapsed <= time); + } + + /** + * Make sure that we validate the snapshot name and the table name before we pass anything across + * the wire + * @throws IOException on failure + */ + @Test + public void testValidateSnapshotName() throws IOException { + HConnectionManager.HConnectionImplementation mockConnection = Mockito + .mock(HConnectionManager.HConnectionImplementation.class); + Configuration conf = HBaseConfiguration.create(); + Mockito.when(mockConnection.getConfiguration()).thenReturn(conf); + HBaseAdmin admin = new HBaseAdmin(mockConnection); + SnapshotDescription.Builder builder = SnapshotDescription.newBuilder(); + // check that invalid snapshot names fail + failSnapshotStart(admin, builder.setName(HConstants.SNAPSHOT_DIR_NAME).build()); + failSnapshotStart(admin, builder.setName("-snapshot").build()); + failSnapshotStart(admin, builder.setName("snapshot fails").build()); + failSnapshotStart(admin, builder.setName("snap$hot").build()); + // check the table name also get verified + failSnapshotStart(admin, builder.setName("snapshot").setTable(".table").build()); + failSnapshotStart(admin, builder.setName("snapshot").setTable("-table").build()); + failSnapshotStart(admin, builder.setName("snapshot").setTable("table fails").build()); + failSnapshotStart(admin, builder.setName("snapshot").setTable("tab%le").build()); + } + + private void failSnapshotStart(HBaseAdmin admin, SnapshotDescription snapshot) throws IOException { + try { + admin.snapshot(snapshot); + fail("Snapshot should not have succeed with name:" + snapshot.getName()); + } catch (IllegalArgumentException e) { + LOG.debug("Correctly failed to start snapshot:" + e.getMessage()); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java b/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java index 837ce2824fcb..ec482da42e71 100644 --- a/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java b/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java index 47f7e33427db..e0f2569fb846 100644 --- a/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java +++ b/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java @@ -94,7 +94,7 @@ public void testConstraintPasses() throws Exception { * @throws Exception */ @SuppressWarnings("unchecked") - @Test(timeout = 10000) + @Test(timeout = 60000) public void testConstraintFails() throws Exception { // create the table diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java index 38c55aaacc7d..5897e5c63af9 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java index 658c07bb6f08..5b38e0b8f060 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/SampleRegionWALObserver.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/SampleRegionWALObserver.java index ff9c502f2bea..130443d4cddf 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/SampleRegionWALObserver.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/SampleRegionWALObserver.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java index dacb9361cb3c..5aba66416347 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,6 +19,8 @@ package org.apache.hadoop.hbase.coprocessor; +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; @@ -27,12 +28,15 @@ import java.util.List; import java.util.Map; import java.util.Arrays; +import java.util.NavigableSet; import com.google.common.collect.ImmutableList; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.CoprocessorEnvironment; import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Mutation; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Increment; @@ -40,11 +44,16 @@ import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.InternalScanner; +import org.apache.hadoop.hbase.regionserver.KeyValueScanner; +import org.apache.hadoop.hbase.regionserver.Leases; +import org.apache.hadoop.hbase.regionserver.MiniBatchOperationInProgress; import org.apache.hadoop.hbase.regionserver.RegionScanner; +import org.apache.hadoop.hbase.regionserver.ScanType; import org.apache.hadoop.hbase.regionserver.Store; import org.apache.hadoop.hbase.regionserver.StoreFile; import org.apache.hadoop.hbase.regionserver.wal.WALEdit; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; /** * A sample region observer that tests the RegionObserver interface. @@ -60,11 +69,13 @@ public class SimpleRegionObserver extends BaseRegionObserver { boolean hadPreClose; boolean hadPostClose; boolean hadPreFlush; + boolean hadPreFlushScannerOpen; boolean hadPostFlush; boolean hadPreSplit; boolean hadPostSplit; boolean hadPreCompactSelect; boolean hadPostCompactSelect; + boolean hadPreCompactScanner; boolean hadPreCompact; boolean hadPostCompact; boolean hadPreGet = false; @@ -84,7 +95,22 @@ public class SimpleRegionObserver extends BaseRegionObserver { boolean hadPreScannerClose = false; boolean hadPostScannerClose = false; boolean hadPreScannerOpen = false; + boolean hadPreStoreScannerOpen = false; boolean hadPostScannerOpen = false; + boolean hadPreBulkLoadHFile = false; + boolean hadPostBulkLoadHFile = false; + boolean hadPreBatchMutate = false; + boolean hadPostBatchMutate = false; + + @Override + public void start(CoprocessorEnvironment e) throws IOException { + // this only makes sure that leases and locks are available to coprocessors + // from external packages + RegionCoprocessorEnvironment re = (RegionCoprocessorEnvironment)e; + Leases leases = re.getRegionServerServices().getLeases(); + leases.createLease("x", null); + leases.cancelLease("x"); + } @Override public void preOpen(ObserverContext c) { @@ -115,12 +141,20 @@ public boolean wasClosed() { } @Override - public void preFlush(ObserverContext c) { + public InternalScanner preFlush(ObserverContext c, Store store, InternalScanner scanner) { hadPreFlush = true; + return scanner; + } + + @Override + public InternalScanner preFlushScannerOpen(final ObserverContext c, + Store store, KeyValueScanner memstoreScanner, InternalScanner s) throws IOException { + hadPreFlushScannerOpen = true; + return null; } @Override - public void postFlush(ObserverContext c) { + public void postFlush(ObserverContext c, Store store, StoreFile resultFile) { hadPostFlush = true; } @@ -161,6 +195,14 @@ public InternalScanner preCompact(ObserverContext return scanner; } + @Override + public InternalScanner preCompactScannerOpen(final ObserverContext c, + Store store, List scanners, ScanType scanType, long earliestPutTs, + InternalScanner s) throws IOException { + hadPreCompactScanner = true; + return null; + } + @Override public void postCompact(ObserverContext e, Store store, StoreFile resultFile) { @@ -179,6 +221,14 @@ public RegionScanner preScannerOpen(final ObserverContext c, + final Store store, final Scan scan, final NavigableSet targetCols, + final KeyValueScanner s) throws IOException { + hadPreStoreScannerOpen = true; + return null; + } + @Override public RegionScanner postScannerOpen(final ObserverContext c, final Scan scan, final RegionScanner s) @@ -344,6 +394,26 @@ public void postDelete(final ObserverContext c, hadPostDeleted = true; } + @Override + public void preBatchMutate(ObserverContext c, + MiniBatchOperationInProgress> miniBatchOp) throws IOException { + RegionCoprocessorEnvironment e = c.getEnvironment(); + assertNotNull(e); + assertNotNull(e.getRegion()); + assertNotNull(miniBatchOp); + hadPreBatchMutate = true; + } + + @Override + public void postBatchMutate(final ObserverContext c, + final MiniBatchOperationInProgress> miniBatchOp) throws IOException { + RegionCoprocessorEnvironment e = c.getEnvironment(); + assertNotNull(e); + assertNotNull(e.getRegion()); + assertNotNull(miniBatchOp); + hadPostBatchMutate = true; + } + @Override public void preGetClosestRowBefore(final ObserverContext c, final byte[] row, final byte[] family, final Result result) @@ -384,6 +454,43 @@ public Result postIncrement(final ObserverContext return result; } + @Override + public void preBulkLoadHFile(ObserverContext ctx, + List> familyPaths) throws IOException { + RegionCoprocessorEnvironment e = ctx.getEnvironment(); + assertNotNull(e); + assertNotNull(e.getRegion()); + if (Arrays.equals(e.getRegion().getTableDesc().getName(), + TestRegionObserverInterface.TEST_TABLE)) { + assertNotNull(familyPaths); + assertEquals(1,familyPaths.size()); + assertArrayEquals(familyPaths.get(0).getFirst(), TestRegionObserverInterface.A); + String familyPath = familyPaths.get(0).getSecond(); + String familyName = Bytes.toString(TestRegionObserverInterface.A); + assertEquals(familyPath.substring(familyPath.length()-familyName.length()-1),"/"+familyName); + } + hadPreBulkLoadHFile = true; + } + + @Override + public boolean postBulkLoadHFile(ObserverContext ctx, + List> familyPaths, boolean hasLoaded) throws IOException { + RegionCoprocessorEnvironment e = ctx.getEnvironment(); + assertNotNull(e); + assertNotNull(e.getRegion()); + if (Arrays.equals(e.getRegion().getTableDesc().getName(), + TestRegionObserverInterface.TEST_TABLE)) { + assertNotNull(familyPaths); + assertEquals(1,familyPaths.size()); + assertArrayEquals(familyPaths.get(0).getFirst(), TestRegionObserverInterface.A); + String familyPath = familyPaths.get(0).getSecond(); + String familyName = Bytes.toString(TestRegionObserverInterface.A); + assertEquals(familyPath.substring(familyPath.length()-familyName.length()-1),"/"+familyName); + } + hadPostBulkLoadHFile = true; + return hasLoaded; + } + public boolean hadPreGet() { return hadPreGet; } @@ -399,6 +506,15 @@ public boolean hadPrePut() { public boolean hadPostPut() { return hadPostPut; } + + public boolean hadPreBatchMutate() { + return hadPreBatchMutate; + } + + public boolean hadPostBatchMutate() { + return hadPostBatchMutate; + } + public boolean hadDelete() { return !beforeDelete; } @@ -430,4 +546,12 @@ public boolean wasScannerOpenCalled() { public boolean hadDeleted() { return hadPreDeleted && hadPostDeleted; } + + public boolean hadPostBulkLoadHFile() { + return hadPostBulkLoadHFile; + } + + public boolean hadPreBulkLoadHFile() { + return hadPreBulkLoadHFile; + } } diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java index 28af3161e3e4..edb641713ea8 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestBigDecimalColumnInterpreter.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestBigDecimalColumnInterpreter.java new file mode 100644 index 000000000000..62a0d00e83e2 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestBigDecimalColumnInterpreter.java @@ -0,0 +1,670 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.coprocessor; + +import static org.junit.Assert.assertEquals; +import java.math.BigDecimal; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.coprocessor.AggregationClient; +import org.apache.hadoop.hbase.client.coprocessor.BigDecimalColumnInterpreter; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.PrefixFilter; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * A test class to test BigDecimalColumnInterpreter for AggregationsProtocol + */ +@Category(MediumTests.class) +public class TestBigDecimalColumnInterpreter { + protected static Log myLog = LogFactory.getLog(TestBigDecimalColumnInterpreter.class); + + /** + * Creating the test infrastructure. + */ + private static final byte[] TEST_TABLE = Bytes.toBytes("TestTable"); + private static final byte[] TEST_FAMILY = Bytes.toBytes("TestFamily"); + private static final byte[] TEST_QUALIFIER = Bytes.toBytes("TestQualifier"); + private static final byte[] TEST_MULTI_CQ = Bytes.toBytes("TestMultiCQ"); + + private static byte[] ROW = Bytes.toBytes("testRow"); + private static final int ROWSIZE = 20; + private static final int rowSeperator1 = 5; + private static final int rowSeperator2 = 12; + private static byte[][] ROWS = makeN(ROW, ROWSIZE); + + private static HBaseTestingUtility util = new HBaseTestingUtility(); + private static Configuration conf = util.getConfiguration(); + + /** + * A set up method to start the test cluster. AggregateProtocolImpl is registered and will be + * loaded during region startup. + * @throws Exception + */ + @BeforeClass + public static void setupBeforeClass() throws Exception { + + conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, + "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"); + + util.startMiniCluster(2); + HTable table = util.createTable(TEST_TABLE, TEST_FAMILY); + util.createMultiRegions(util.getConfiguration(), table, TEST_FAMILY, new byte[][] { + HConstants.EMPTY_BYTE_ARRAY, ROWS[rowSeperator1], ROWS[rowSeperator2] }); + /** + * The testtable has one CQ which is always populated and one variable CQ for each row rowkey1: + * CF:CQ CF:CQ1 rowKey2: CF:CQ CF:CQ2 + */ + for (int i = 0; i < ROWSIZE; i++) { + Put put = new Put(ROWS[i]); + put.setWriteToWAL(false); + BigDecimal bd = new BigDecimal(i); + put.add(TEST_FAMILY, TEST_QUALIFIER, Bytes.toBytes(bd)); + table.put(put); + Put p2 = new Put(ROWS[i]); + put.setWriteToWAL(false); + p2.add(TEST_FAMILY, Bytes.add(TEST_MULTI_CQ, Bytes.toBytes(bd)), + Bytes.toBytes(bd.multiply(new BigDecimal("0.10")))); + table.put(p2); + } + table.close(); + } + + /** + * Shutting down the cluster + * @throws Exception + */ + @AfterClass + public static void tearDownAfterClass() throws Exception { + util.shutdownMiniCluster(); + } + + /** + * an infrastructure method to prepare rows for the testtable. + * @param base + * @param n + * @return + */ + private static byte[][] makeN(byte[] base, int n) { + byte[][] ret = new byte[n][]; + for (int i = 0; i < n; i++) { + ret[i] = Bytes.add(base, Bytes.toBytes(i)); + } + return ret; + } + + /** + * ****************** Test cases for Median ********************** + */ + /** + * @throws Throwable + */ + @Test + public void testMedianWithValidRange() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal median = aClient.median(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("8.00"), median); + } + + /** + * ***************Test cases for Maximum ******************* + */ + + /** + * give max for the entire table. + * @throws Throwable + */ + @Test + public void testMax() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal maximum = aClient.max(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("19.00"), maximum); + } + + /** + * @throws Throwable + */ + @Test + public void testMaxWithValidRange2() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + scan.setStartRow(ROWS[5]); + scan.setStopRow(ROWS[15]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal max = aClient.max(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("14.00"), max); + } + + @Test + public void testMaxWithValidRangeWithoutCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal maximum = aClient.max(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("19.00"), maximum); + } + + @Test + public void testMaxWithValidRange2WithoutCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[7]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal max = aClient.max(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("6.00"), max); + } + + @Test + public void testMaxWithValidRangeWithNullCF() { + AggregationClient aClient = new AggregationClient(conf); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + Scan scan = new Scan(); + BigDecimal max = null; + try { + max = aClient.max(TEST_TABLE, ci, scan); + } catch (Throwable e) { + max = null; + } + assertEquals(null, max);// CP will throw an IOException about the + // null column family, and max will be set to 0 + } + + @Test + public void testMaxWithInvalidRange() { + AggregationClient aClient = new AggregationClient(conf); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + Scan scan = new Scan(); + scan.setStartRow(ROWS[4]); + scan.setStopRow(ROWS[2]); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + BigDecimal max = new BigDecimal(Long.MIN_VALUE); + ; + try { + max = aClient.max(TEST_TABLE, ci, scan); + } catch (Throwable e) { + max = BigDecimal.ZERO; + } + assertEquals(BigDecimal.ZERO, max);// control should go to the catch block + } + + @Test + public void testMaxWithInvalidRange2() throws Throwable { + BigDecimal max = new BigDecimal(Long.MIN_VALUE); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + scan.setStartRow(ROWS[4]); + scan.setStopRow(ROWS[4]); + try { + AggregationClient aClient = new AggregationClient(conf); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + max = aClient.max(TEST_TABLE, ci, scan); + } catch (Exception e) { + max = BigDecimal.ZERO; + } + assertEquals(BigDecimal.ZERO, max);// control should go to the catch block + } + + @Test + public void testMaxWithFilter() throws Throwable { + BigDecimal max = BigDecimal.ZERO; + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + Filter f = new PrefixFilter(Bytes.toBytes("foo:bar")); + scan.setFilter(f); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + max = aClient.max(TEST_TABLE, ci, scan); + assertEquals(null, max); + } + + /** + * **************************Test cases for Minimum *********************** + */ + + /** + * @throws Throwable + */ + @Test + public void testMinWithValidRange() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + scan.setStartRow(HConstants.EMPTY_START_ROW); + scan.setStopRow(HConstants.EMPTY_END_ROW); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal min = aClient.min(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("0.00"), min); + } + + /** + * @throws Throwable + */ + @Test + public void testMinWithValidRange2() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + scan.setStartRow(ROWS[5]); + scan.setStopRow(ROWS[15]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal min = aClient.min(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("5.00"), min); + } + + @Test + public void testMinWithValidRangeWithNoCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(HConstants.EMPTY_START_ROW); + scan.setStopRow(HConstants.EMPTY_END_ROW); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal min = aClient.min(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("0.00"), min); + } + + @Test + public void testMinWithValidRange2WithNoCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[7]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal min = aClient.min(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("0.60"), min); + } + + @Test + public void testMinWithValidRangeWithNullCF() { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.setStartRow(ROWS[5]); + scan.setStopRow(ROWS[15]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal min = null; + try { + min = aClient.min(TEST_TABLE, ci, scan); + } catch (Throwable e) { + } + assertEquals(null, min);// CP will throw an IOException about the + // null column family, and max will be set to 0 + } + + @Test + public void testMinWithInvalidRange() { + AggregationClient aClient = new AggregationClient(conf); + BigDecimal min = null; + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(ROWS[4]); + scan.setStopRow(ROWS[2]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + try { + min = aClient.min(TEST_TABLE, ci, scan); + } catch (Throwable e) { + } + assertEquals(null, min);// control should go to the catch block + } + + @Test + public void testMinWithInvalidRange2() { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[6]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal min = null; + try { + min = aClient.min(TEST_TABLE, ci, scan); + } catch (Throwable e) { + } + assertEquals(null, min);// control should go to the catch block + } + + @Test + public void testMinWithFilter() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + Filter f = new PrefixFilter(Bytes.toBytes("foo:bar")); + scan.setFilter(f); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal min = null; + min = aClient.min(TEST_TABLE, ci, scan); + assertEquals(null, min); + } + + /** + * *************** Test cases for Sum ********************* + */ + /** + * @throws Throwable + */ + @Test + public void testSumWithValidRange() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal sum = aClient.sum(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("190.00"), sum); + } + + /** + * @throws Throwable + */ + @Test + public void testSumWithValidRange2() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + scan.setStartRow(ROWS[5]); + scan.setStopRow(ROWS[15]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal sum = aClient.sum(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("95.00"), sum); + } + + @Test + public void testSumWithValidRangeWithNoCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal sum = aClient.sum(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("209.00"), sum); // 190 + 19 + } + + @Test + public void testSumWithValidRange2WithNoCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[7]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal sum = aClient.sum(TEST_TABLE, ci, scan); + assertEquals(new BigDecimal("6.60"), sum); // 6 + 60 + } + + @Test + public void testSumWithValidRangeWithNullCF() { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[7]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal sum = null; + try { + sum = aClient.sum(TEST_TABLE, ci, scan); + } catch (Throwable e) { + } + assertEquals(null, sum);// CP will throw an IOException about the + // null column family, and max will be set to 0 + } + + @Test + public void testSumWithInvalidRange() { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[2]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal sum = null; + try { + sum = aClient.sum(TEST_TABLE, ci, scan); + } catch (Throwable e) { + } + assertEquals(null, sum);// control should go to the catch block + } + + @Test + public void testSumWithFilter() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Filter f = new PrefixFilter(Bytes.toBytes("foo:bar")); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setFilter(f); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + BigDecimal sum = null; + sum = aClient.sum(TEST_TABLE, ci, scan); + assertEquals(null, sum); + } + + /** + * ****************************** Test Cases for Avg ************** + */ + /** + * @throws Throwable + */ + @Test + public void testAvgWithValidRange() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + double avg = aClient.avg(TEST_TABLE, ci, scan); + assertEquals(9.5, avg, 0); + } + + /** + * @throws Throwable + */ + @Test + public void testAvgWithValidRange2() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + scan.setStartRow(ROWS[5]); + scan.setStopRow(ROWS[15]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + double avg = aClient.avg(TEST_TABLE, ci, scan); + assertEquals(9.5, avg, 0); + } + + @Test + public void testAvgWithValidRangeWithNoCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + double avg = aClient.avg(TEST_TABLE, ci, scan); + assertEquals(10.45, avg, 0.01); + } + + @Test + public void testAvgWithValidRange2WithNoCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[7]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + double avg = aClient.avg(TEST_TABLE, ci, scan); + assertEquals(6 + 0.60, avg, 0); + } + + @Test + public void testAvgWithValidRangeWithNullCF() { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + Double avg = null; + try { + avg = aClient.avg(TEST_TABLE, ci, scan); + } catch (Throwable e) { + } + assertEquals(null, avg);// CP will throw an IOException about the + // null column family, and max will be set to 0 + } + + @Test + public void testAvgWithInvalidRange() { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + scan.setStartRow(ROWS[5]); + scan.setStopRow(ROWS[1]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + Double avg = null; + try { + avg = aClient.avg(TEST_TABLE, ci, scan); + } catch (Throwable e) { + } + assertEquals(null, avg);// control should go to the catch block + } + + @Test + public void testAvgWithFilter() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + Filter f = new PrefixFilter(Bytes.toBytes("foo:bar")); + scan.setFilter(f); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + Double avg = null; + avg = aClient.avg(TEST_TABLE, ci, scan); + assertEquals(Double.NaN, avg, 0); + } + + /** + * ****************** Test cases for STD ********************** + */ + /** + * @throws Throwable + */ + @Test + public void testStdWithValidRange() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + double std = aClient.std(TEST_TABLE, ci, scan); + assertEquals(5.766, std, 0.05d); + } + + /** + * need to change this + * @throws Throwable + */ + @Test + public void testStdWithValidRange2() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addColumn(TEST_FAMILY, TEST_QUALIFIER); + scan.setStartRow(ROWS[5]); + scan.setStopRow(ROWS[15]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + double std = aClient.std(TEST_TABLE, ci, scan); + assertEquals(2.87, std, 0.05d); + } + + /** + * need to change this + * @throws Throwable + */ + @Test + public void testStdWithValidRangeWithNoCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + double std = aClient.std(TEST_TABLE, ci, scan); + assertEquals(6.342, std, 0.05d); + } + + @Test + public void testStdWithValidRange2WithNoCQ() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[7]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + double std = aClient.std(TEST_TABLE, ci, scan); + System.out.println("std is:" + std); + assertEquals(0, std, 0.05d); + } + + @Test + public void testStdWithValidRangeWithNullCF() { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[17]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + Double std = null; + try { + std = aClient.std(TEST_TABLE, ci, scan); + } catch (Throwable e) { + } + assertEquals(null, std);// CP will throw an IOException about the + // null column family, and max will be set to 0 + } + + @Test + public void testStdWithInvalidRange() { + AggregationClient aClient = new AggregationClient(conf); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setStartRow(ROWS[6]); + scan.setStopRow(ROWS[1]); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + Double std = null; + try { + std = aClient.std(TEST_TABLE, ci, scan); + } catch (Throwable e) { + } + assertEquals(null, std);// control should go to the catch block + } + + @Test + public void testStdWithFilter() throws Throwable { + AggregationClient aClient = new AggregationClient(conf); + Filter f = new PrefixFilter(Bytes.toBytes("foo:bar")); + Scan scan = new Scan(); + scan.addFamily(TEST_FAMILY); + scan.setFilter(f); + final ColumnInterpreter ci = new BigDecimalColumnInterpreter(); + Double std = null; + std = aClient.std(TEST_TABLE, ci, scan); + assertEquals(Double.NaN, std, 0); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java index 368a0e524983..0d87f4b398c5 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -26,20 +25,22 @@ import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.regionserver.HRegion; + +import org.apache.hadoop.hbase.util.ClassLoaderTestHelper; +import org.apache.hadoop.hbase.util.CoprocessorClassLoader; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import javax.tools.*; import java.io.*; import java.util.*; -import java.util.Arrays; import java.util.jar.*; import org.junit.*; import org.junit.experimental.categories.Category; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; import static org.junit.Assert.assertFalse; @@ -51,7 +52,6 @@ public class TestClassLoading { private static final Log LOG = LogFactory.getLog(TestClassLoading.class); private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); - private static Configuration conf; private static MiniDFSCluster cluster; static final int BUFFER_SIZE = 4096; @@ -61,28 +61,22 @@ public class TestClassLoading { static final String cpName3 = "TestCP3"; static final String cpName4 = "TestCP4"; static final String cpName5 = "TestCP5"; + static final String cpName6 = "TestCP6"; + static final String cpNameInvalid = "TestCPInvalid"; - private static Class regionCoprocessor1 = ColumnAggregationEndpoint.class; - private static Class regionCoprocessor2 = GenericEndpoint.class; - private static Class regionServerCoprocessor = SampleRegionWALObserver.class; - private static Class masterCoprocessor = BaseMasterObserver.class; + private static Class regionCoprocessor1 = ColumnAggregationEndpoint.class; + private static Class regionCoprocessor2 = GenericEndpoint.class; + private static Class regionServerCoprocessor = SampleRegionWALObserver.class; + private static Class masterCoprocessor = BaseMasterObserver.class; private static final String[] regionServerSystemCoprocessors = new String[]{ - regionCoprocessor1.getSimpleName(), - regionServerCoprocessor.getSimpleName() - }; - - private static final String[] regionServerSystemAndUserCoprocessors = - new String[] { - regionCoprocessor1.getSimpleName(), - regionCoprocessor2.getSimpleName(), regionServerCoprocessor.getSimpleName() }; @BeforeClass public static void setUpBeforeClass() throws Exception { - conf = TEST_UTIL.getConfiguration(); + Configuration conf = TEST_UTIL.getConfiguration(); // regionCoprocessor1 will be loaded on all regionservers, since it is // loaded for any tables (user or meta). @@ -108,91 +102,11 @@ public static void tearDownAfterClass() throws Exception { TEST_UTIL.shutdownMiniCluster(); } - // generate jar file - private boolean createJarArchive(File archiveFile, File[] tobeJared) { - try { - byte buffer[] = new byte[BUFFER_SIZE]; - // Open archive file - FileOutputStream stream = new FileOutputStream(archiveFile); - JarOutputStream out = new JarOutputStream(stream, new Manifest()); - - for (int i = 0; i < tobeJared.length; i++) { - if (tobeJared[i] == null || !tobeJared[i].exists() - || tobeJared[i].isDirectory()) { - continue; - } - - // Add archive entry - JarEntry jarAdd = new JarEntry(tobeJared[i].getName()); - jarAdd.setTime(tobeJared[i].lastModified()); - out.putNextEntry(jarAdd); - - // Write file to archive - FileInputStream in = new FileInputStream(tobeJared[i]); - while (true) { - int nRead = in.read(buffer, 0, buffer.length); - if (nRead <= 0) - break; - out.write(buffer, 0, nRead); - } - in.close(); - } - out.close(); - stream.close(); - LOG.info("Adding classes to jar file completed"); - return true; - } catch (Exception ex) { - LOG.error("Error: " + ex.getMessage()); - return false; - } - } - - private File buildCoprocessorJar(String className) throws Exception { - // compose a java source file. - String javaCode = "import org.apache.hadoop.hbase.coprocessor.*;" + + static File buildCoprocessorJar(String className) throws Exception { + String code = "import org.apache.hadoop.hbase.coprocessor.*;" + "public class " + className + " extends BaseRegionObserver {}"; - Path baseDir = TEST_UTIL.getDataTestDir(); - Path srcDir = new Path(TEST_UTIL.getDataTestDir(), "src"); - File srcDirPath = new File(srcDir.toString()); - srcDirPath.mkdirs(); - File sourceCodeFile = new File(srcDir.toString(), className + ".java"); - BufferedWriter bw = new BufferedWriter(new FileWriter(sourceCodeFile)); - bw.write(javaCode); - bw.close(); - - // compile it by JavaCompiler - JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); - ArrayList srcFileNames = new ArrayList(); - srcFileNames.add(sourceCodeFile.toString()); - StandardJavaFileManager fm = compiler.getStandardFileManager(null, null, - null); - Iterable cu = - fm.getJavaFileObjects(sourceCodeFile); - List options = new ArrayList(); - options.add("-classpath"); - // only add hbase classes to classpath. This is a little bit tricky: assume - // the classpath is {hbaseSrc}/target/classes. - String currentDir = new File(".").getAbsolutePath(); - String classpath = - currentDir + Path.SEPARATOR + "target"+ Path.SEPARATOR + "classes" + - System.getProperty("path.separator") + - System.getProperty("surefire.test.class.path"); - options.add(classpath); - LOG.debug("Setting classpath to: "+classpath); - - JavaCompiler.CompilationTask task = compiler.getTask(null, fm, null, - options, null, cu); - assertTrue("Compile file " + sourceCodeFile + " failed.", task.call()); - - // build a jar file by the classes files - String jarFileName = className + ".jar"; - File jarFile = new File(baseDir.toString(), jarFileName); - if (!createJarArchive(jarFile, - new File[]{new File(srcDir.toString(), className + ".class")})){ - assertTrue("Build jar file failed.", false); - } - - return jarFile; + return ClassLoaderTestHelper.buildJar( + TEST_UTIL.getDataTestDir().toString(), className, code); } @Test @@ -208,16 +122,18 @@ public void testClassLoadingFromHDFS() throws Exception { new Path(fs.getUri().toString() + Path.SEPARATOR)); String jarFileOnHDFS1 = fs.getUri().toString() + Path.SEPARATOR + jarFile1.getName(); + Path pathOnHDFS1 = new Path(jarFileOnHDFS1); assertTrue("Copy jar file to HDFS failed.", - fs.exists(new Path(jarFileOnHDFS1))); + fs.exists(pathOnHDFS1)); LOG.info("Copied jar file to HDFS: " + jarFileOnHDFS1); fs.copyFromLocalFile(new Path(jarFile2.getPath()), new Path(fs.getUri().toString() + Path.SEPARATOR)); String jarFileOnHDFS2 = fs.getUri().toString() + Path.SEPARATOR + jarFile2.getName(); + Path pathOnHDFS2 = new Path(jarFileOnHDFS2); assertTrue("Copy jar file to HDFS failed.", - fs.exists(new Path(jarFileOnHDFS2))); + fs.exists(pathOnHDFS2)); LOG.info("Copied jar file to HDFS: " + jarFileOnHDFS2); // create a table that references the coprocessors @@ -229,40 +145,82 @@ public void testClassLoadingFromHDFS() throws Exception { // with configuration values htd.setValue("COPROCESSOR$2", jarFileOnHDFS2.toString() + "|" + cpName2 + "|" + Coprocessor.PRIORITY_USER + "|k1=v1,k2=v2,k3=v3"); - HBaseAdmin admin = new HBaseAdmin(this.conf); + // same jar but invalid class name (should fail to load this class) + htd.setValue("COPROCESSOR$3", jarFileOnHDFS2.toString() + "|" + cpNameInvalid + + "|" + Coprocessor.PRIORITY_USER); + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); if (admin.tableExists(tableName)) { admin.disableTable(tableName); admin.deleteTable(tableName); } - admin.createTable(htd); + CoprocessorClassLoader.clearCache(); + byte[] startKey = {10, 63}; + byte[] endKey = {12, 43}; + admin.createTable(htd, startKey, endKey, 4); + waitForTable(htd.getName()); // verify that the coprocessors were loaded - boolean found1 = false, found2 = false, found2_k1 = false, - found2_k2 = false, found2_k3 = false; + boolean foundTableRegion=false; + boolean found_invalid = true, found1 = true, found2 = true, found2_k1 = true, + found2_k2 = true, found2_k3 = true; + Map> regionsActiveClassLoaders = + new HashMap>(); MiniHBaseCluster hbase = TEST_UTIL.getHBaseCluster(); for (HRegion region: hbase.getRegionServer(0).getOnlineRegionsLocalContext()) { if (region.getRegionNameAsString().startsWith(tableName)) { + foundTableRegion = true; CoprocessorEnvironment env; env = region.getCoprocessorHost().findCoprocessorEnvironment(cpName1); - if (env != null) { - found1 = true; - } + found1 = found1 && (env != null); env = region.getCoprocessorHost().findCoprocessorEnvironment(cpName2); + found2 = found2 && (env != null); if (env != null) { - found2 = true; Configuration conf = env.getConfiguration(); - found2_k1 = conf.get("k1") != null; - found2_k2 = conf.get("k2") != null; - found2_k3 = conf.get("k3") != null; + found2_k1 = found2_k1 && (conf.get("k1") != null); + found2_k2 = found2_k2 && (conf.get("k2") != null); + found2_k3 = found2_k3 && (conf.get("k3") != null); + } else { + found2_k1 = found2_k2 = found2_k3 = false; } + env = region.getCoprocessorHost().findCoprocessorEnvironment(cpNameInvalid); + found_invalid = found_invalid && (env != null); + + regionsActiveClassLoaders + .put(region, ((CoprocessorHost) region.getCoprocessorHost()).getExternalClassLoaders()); } } + + assertTrue("No region was found for table " + tableName, foundTableRegion); assertTrue("Class " + cpName1 + " was missing on a region", found1); assertTrue("Class " + cpName2 + " was missing on a region", found2); + //an invalid CP class name is defined for this table, validate that it is not loaded + assertFalse("Class " + cpNameInvalid + " was found on a region", found_invalid); assertTrue("Configuration key 'k1' was missing on a region", found2_k1); assertTrue("Configuration key 'k2' was missing on a region", found2_k2); assertTrue("Configuration key 'k3' was missing on a region", found2_k3); + // check if CP classloaders are cached + assertNotNull(jarFileOnHDFS1 + " was not cached", + CoprocessorClassLoader.getIfCached(pathOnHDFS1)); + assertNotNull(jarFileOnHDFS2 + " was not cached", + CoprocessorClassLoader.getIfCached(pathOnHDFS2)); + //two external jar used, should be one classloader per jar + assertEquals("The number of cached classloaders should be equal to the number" + + " of external jar files", + 2, CoprocessorClassLoader.getAllCached().size()); + //check if region active classloaders are shared across all RS regions + Set externalClassLoaders = new HashSet( + CoprocessorClassLoader.getAllCached()); + for (Map.Entry> regionCP : regionsActiveClassLoaders.entrySet()) { + assertTrue("Some CP classloaders for region " + regionCP.getKey() + " are not cached." + + " ClassLoader Cache:" + externalClassLoaders + + " Region ClassLoaders:" + regionCP.getValue(), + externalClassLoaders.containsAll(regionCP.getValue())); + } + } + + private String getLocalPath(File file) { + return new Path(file.toURI()).toString(); } @Test @@ -273,10 +231,11 @@ public void testClassLoadingFromLocalFS() throws Exception { // create a table that references the jar HTableDescriptor htd = new HTableDescriptor(cpName3); htd.addFamily(new HColumnDescriptor("test")); - htd.setValue("COPROCESSOR$1", jarFile.toString() + "|" + cpName3 + "|" + + htd.setValue("COPROCESSOR$1", getLocalPath(jarFile) + "|" + cpName3 + "|" + Coprocessor.PRIORITY_USER); - HBaseAdmin admin = new HBaseAdmin(this.conf); + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); admin.createTable(htd); + waitForTable(htd.getName()); // verify that the coprocessor was loaded boolean found = false; @@ -290,6 +249,37 @@ public void testClassLoadingFromLocalFS() throws Exception { assertTrue("Class " + cpName3 + " was missing on a region", found); } + @Test + // HBASE-6308: Test CP classloader is the CoprocessorClassLoader + public void testPrivateClassLoader() throws Exception { + File jarFile = buildCoprocessorJar(cpName4); + + // create a table that references the jar + HTableDescriptor htd = new HTableDescriptor(cpName4); + htd.addFamily(new HColumnDescriptor("test")); + htd.setValue("COPROCESSOR$1", getLocalPath(jarFile) + "|" + cpName4 + "|" + + Coprocessor.PRIORITY_USER); + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + admin.createTable(htd); + waitForTable(htd.getName()); + + // verify that the coprocessor was loaded correctly + boolean found = false; + MiniHBaseCluster hbase = TEST_UTIL.getHBaseCluster(); + for (HRegion region: + hbase.getRegionServer(0).getOnlineRegionsLocalContext()) { + if (region.getRegionNameAsString().startsWith(cpName4)) { + Coprocessor cp = region.getCoprocessorHost().findCoprocessor(cpName4); + if (cp != null) { + found = true; + assertEquals("Class " + cpName4 + " was not loaded by CoprocessorClassLoader", + cp.getClass().getClassLoader().getClass(), CoprocessorClassLoader.class); + } + } + } + assertTrue("Class " + cpName4 + " was missing on a region", found); + } + @Test // HBase-3810: Registering a Coprocessor at HTableDescriptor should be // less strict @@ -298,16 +288,16 @@ public void testHBase3810() throws Exception { File jarFile1 = buildCoprocessorJar(cpName1); File jarFile2 = buildCoprocessorJar(cpName2); - File jarFile4 = buildCoprocessorJar(cpName4); File jarFile5 = buildCoprocessorJar(cpName5); + File jarFile6 = buildCoprocessorJar(cpName6); String cpKey1 = "COPROCESSOR$1"; String cpKey2 = " Coprocessor$2 "; String cpKey3 = " coprocessor$03 "; - String cpValue1 = jarFile1.toString() + "|" + cpName1 + "|" + + String cpValue1 = getLocalPath(jarFile1) + "|" + cpName1 + "|" + Coprocessor.PRIORITY_USER; - String cpValue2 = jarFile2.toString() + " | " + cpName2 + " | "; + String cpValue2 = getLocalPath(jarFile2) + " | " + cpName2 + " | "; // load from default class loader String cpValue3 = " | org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver | | k=v "; @@ -322,27 +312,28 @@ public void testHBase3810() throws Exception { htd.setValue(cpKey3, cpValue3); // add 2 coprocessor by using new htd.addCoprocessor() api - htd.addCoprocessor(cpName4, new Path(jarFile4.getPath()), + htd.addCoprocessor(cpName5, new Path(getLocalPath(jarFile5)), Coprocessor.PRIORITY_USER, null); Map kvs = new HashMap(); kvs.put("k1", "v1"); kvs.put("k2", "v2"); kvs.put("k3", "v3"); - htd.addCoprocessor(cpName5, new Path(jarFile5.getPath()), + htd.addCoprocessor(cpName6, new Path(getLocalPath(jarFile6)), Coprocessor.PRIORITY_USER, kvs); - HBaseAdmin admin = new HBaseAdmin(this.conf); + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); if (admin.tableExists(tableName)) { admin.disableTable(tableName); admin.deleteTable(tableName); } admin.createTable(htd); + waitForTable(htd.getName()); // verify that the coprocessor was loaded boolean found_2 = false, found_1 = false, found_3 = false, - found_4 = false, found_5 = false; - boolean found5_k1 = false, found5_k2 = false, found5_k3 = false, - found5_k4 = false; + found_5 = false, found_6 = false; + boolean found6_k1 = false, found6_k2 = false, found6_k3 = false, + found6_k4 = false; MiniHBaseCluster hbase = TEST_UTIL.getHBaseCluster(); for (HRegion region: @@ -355,17 +346,17 @@ public void testHBase3810() throws Exception { found_3 = found_3 || (region.getCoprocessorHost().findCoprocessor("SimpleRegionObserver") != null); - found_4 = found_4 || - (region.getCoprocessorHost().findCoprocessor(cpName4) != null); + found_5 = found_5 || + (region.getCoprocessorHost().findCoprocessor(cpName5) != null); CoprocessorEnvironment env = - region.getCoprocessorHost().findCoprocessorEnvironment(cpName5); + region.getCoprocessorHost().findCoprocessorEnvironment(cpName6); if (env != null) { - found_5 = true; + found_6 = true; Configuration conf = env.getConfiguration(); - found5_k1 = conf.get("k1") != null; - found5_k2 = conf.get("k2") != null; - found5_k3 = conf.get("k3") != null; + found6_k1 = conf.get("k1") != null; + found6_k2 = conf.get("k2") != null; + found6_k3 = conf.get("k3") != null; } } } @@ -373,90 +364,122 @@ public void testHBase3810() throws Exception { assertTrue("Class " + cpName1 + " was missing on a region", found_1); assertTrue("Class " + cpName2 + " was missing on a region", found_2); assertTrue("Class SimpleRegionObserver was missing on a region", found_3); - assertTrue("Class " + cpName4 + " was missing on a region", found_4); assertTrue("Class " + cpName5 + " was missing on a region", found_5); + assertTrue("Class " + cpName6 + " was missing on a region", found_6); - assertTrue("Configuration key 'k1' was missing on a region", found5_k1); - assertTrue("Configuration key 'k2' was missing on a region", found5_k2); - assertTrue("Configuration key 'k3' was missing on a region", found5_k3); - assertFalse("Configuration key 'k4' wasn't configured", found5_k4); + assertTrue("Configuration key 'k1' was missing on a region", found6_k1); + assertTrue("Configuration key 'k2' was missing on a region", found6_k2); + assertTrue("Configuration key 'k3' was missing on a region", found6_k3); + assertFalse("Configuration key 'k4' wasn't configured", found6_k4); } @Test - public void testRegionServerCoprocessorsReported() throws Exception { - // HBASE 4070: Improve region server metrics to report loaded coprocessors - // to master: verify that each regionserver is reporting the correct set of - // loaded coprocessors. + public void testClassLoadingFromLibDirInJar() throws Exception { + loadingClassFromLibDirInJar("/lib/"); + } - // We rely on the fact that getCoprocessors() will return a sorted - // display of the coprocessors' names, so for example, regionCoprocessor1's - // name "ColumnAggregationEndpoint" will appear before regionCoprocessor2's - // name "GenericEndpoint" because "C" is before "G" lexicographically. - - HBaseAdmin admin = new HBaseAdmin(this.conf); - - // disable all user tables, if any are loaded. - for (HTableDescriptor htd: admin.listTables()) { - if (!htd.isMetaTable()) { - String tableName = htd.getNameAsString(); - if (admin.isTableEnabled(tableName)) { - try { - admin.disableTable(htd.getNameAsString()); - } catch (TableNotEnabledException e) { - // ignoring this exception for now : not sure why it's happening. - } - } - } - } + @Test + public void testClassLoadingFromRelativeLibDirInJar() throws Exception { + loadingClassFromLibDirInJar("lib/"); + } - // should only be system coprocessors loaded at this point. - assertAllRegionServers(regionServerSystemCoprocessors,null); + void loadingClassFromLibDirInJar(String libPrefix) throws Exception { + FileSystem fs = cluster.getFileSystem(); - // The next two tests enable and disable user tables to see if coprocessor - // load reporting changes as coprocessors are loaded and unloaded. - // - - // Create a table. - // should cause regionCoprocessor2 to be loaded, since we've specified it - // for loading on any user table with USER_REGION_COPROCESSOR_CONF_KEY - // in setUpBeforeClass(). - String userTable1 = "userTable1"; - HTableDescriptor userTD1 = new HTableDescriptor(userTable1); - admin.createTable(userTD1); - // table should be enabled now. - assertTrue(admin.isTableEnabled(userTable1)); - assertAllRegionServers(regionServerSystemAndUserCoprocessors, userTable1); - - // unload and make sure we're back to only system coprocessors again. - admin.disableTable(userTable1); - assertAllRegionServers(regionServerSystemCoprocessors,null); + File innerJarFile1 = buildCoprocessorJar(cpName1); + File innerJarFile2 = buildCoprocessorJar(cpName2); + File outerJarFile = new File(TEST_UTIL.getDataTestDir().toString(), "outer.jar"); + + byte buffer[] = new byte[BUFFER_SIZE]; + // TODO: code here and elsewhere in this file is duplicated w/TestClassFinder. + // Some refactoring may be in order... + // Open archive file + FileOutputStream stream = new FileOutputStream(outerJarFile); + JarOutputStream out = new JarOutputStream(stream, new Manifest()); + + for (File jarFile: new File[] { innerJarFile1, innerJarFile2 }) { + // Add archive entry + JarEntry jarAdd = new JarEntry(libPrefix + jarFile.getName()); + jarAdd.setTime(jarFile.lastModified()); + out.putNextEntry(jarAdd); + + // Write file to archive + FileInputStream in = new FileInputStream(jarFile); + while (true) { + int nRead = in.read(buffer, 0, buffer.length); + if (nRead <= 0) + break; + out.write(buffer, 0, nRead); + } + in.close(); + } + out.close(); + stream.close(); + LOG.info("Adding jar file to outer jar file completed"); - // create another table, with its own specified coprocessor. - String userTable2 = "userTable2"; - HTableDescriptor htd2 = new HTableDescriptor(userTable2); + // copy the jars into dfs + fs.copyFromLocalFile(new Path(outerJarFile.getPath()), + new Path(fs.getUri().toString() + Path.SEPARATOR)); + String jarFileOnHDFS = fs.getUri().toString() + Path.SEPARATOR + + outerJarFile.getName(); + assertTrue("Copy jar file to HDFS failed.", + fs.exists(new Path(jarFileOnHDFS))); + LOG.info("Copied jar file to HDFS: " + jarFileOnHDFS); - String userTableCP = "userTableCP"; - File jarFile1 = buildCoprocessorJar(userTableCP); - htd2.addFamily(new HColumnDescriptor("myfamily")); - htd2.setValue("COPROCESSOR$1", jarFile1.toString() + "|" + userTableCP + + // create a table that references the coprocessors + HTableDescriptor htd = new HTableDescriptor(tableName); + htd.addFamily(new HColumnDescriptor("test")); + // without configuration values + htd.setValue("COPROCESSOR$1", jarFileOnHDFS.toString() + "|" + cpName1 + "|" + Coprocessor.PRIORITY_USER); - admin.createTable(htd2); - // table should be enabled now. - assertTrue(admin.isTableEnabled(userTable2)); - - ArrayList existingCPsPlusNew = - new ArrayList(Arrays.asList(regionServerSystemAndUserCoprocessors)); - existingCPsPlusNew.add(userTableCP); - String[] existingCPsPlusNewArray = new String[existingCPsPlusNew.size()]; - assertAllRegionServers(existingCPsPlusNew.toArray(existingCPsPlusNewArray), - userTable2); + // with configuration values + htd.setValue("COPROCESSOR$2", jarFileOnHDFS.toString() + "|" + cpName2 + + "|" + Coprocessor.PRIORITY_USER + "|k1=v1,k2=v2,k3=v3"); + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + if (admin.tableExists(tableName)) { + admin.disableTable(tableName); + admin.deleteTable(tableName); + } + admin.createTable(htd); + waitForTable(htd.getName()); - admin.disableTable(userTable2); - assertTrue(admin.isTableDisabled(userTable2)); + // verify that the coprocessors were loaded + boolean found1 = false, found2 = false, found2_k1 = false, + found2_k2 = false, found2_k3 = false; + MiniHBaseCluster hbase = TEST_UTIL.getHBaseCluster(); + for (HRegion region: + hbase.getRegionServer(0).getOnlineRegionsLocalContext()) { + if (region.getRegionNameAsString().startsWith(tableName)) { + CoprocessorEnvironment env; + env = region.getCoprocessorHost().findCoprocessorEnvironment(cpName1); + if (env != null) { + found1 = true; + } + env = region.getCoprocessorHost().findCoprocessorEnvironment(cpName2); + if (env != null) { + found2 = true; + Configuration conf = env.getConfiguration(); + found2_k1 = conf.get("k1") != null; + found2_k2 = conf.get("k2") != null; + found2_k3 = conf.get("k3") != null; + } + } + } + assertTrue("Class " + cpName1 + " was missing on a region", found1); + assertTrue("Class " + cpName2 + " was missing on a region", found2); + assertTrue("Configuration key 'k1' was missing on a region", found2_k1); + assertTrue("Configuration key 'k2' was missing on a region", found2_k2); + assertTrue("Configuration key 'k3' was missing on a region", found2_k3); + } - // we should be back to only system coprocessors again. - assertAllRegionServers(regionServerSystemCoprocessors, null); + @Test + public void testRegionServerCoprocessorsReported() throws Exception { + // This was a test for HBASE-4070. + // We are removing coprocessors from region load in HBASE-5258. + // Therefore, this test now only checks system coprocessors. + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + assertAllRegionServers(regionServerSystemCoprocessors,null); } /** @@ -503,7 +526,7 @@ void assertAllRegionServers(String[] expectedCoprocessors, String tableName) } boolean any_failed = false; for(Map.Entry server: servers.entrySet()) { - actualCoprocessors = server.getValue().getCoprocessors(); + actualCoprocessors = server.getValue().getRsCoprocessors(); if (!Arrays.equals(actualCoprocessors, expectedCoprocessors)) { LOG.debug("failed comparison: actual: " + Arrays.toString(actualCoprocessors) + @@ -535,6 +558,13 @@ public void testMasterCoprocessorsReported() { assertEquals(loadedMasterCoprocessorsVerify, loadedMasterCoprocessors); } + private void waitForTable(byte[] name) throws InterruptedException, IOException { + // First wait until all regions are online + TEST_UTIL.waitTableEnabled(name, 5000); + // Now wait a bit longer for the coprocessor hosts to load the CPs + Thread.sleep(1000); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java index e1f52c1bace9..03d80a78cbe5 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -26,10 +25,12 @@ import java.util.Map; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HConstants; import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Scan; @@ -75,6 +76,8 @@ public static void setupBeforeClass() throws Exception { conf.setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, "org.apache.hadoop.hbase.coprocessor.ColumnAggregationEndpoint", "org.apache.hadoop.hbase.coprocessor.GenericEndpoint"); + conf.setStrings(CoprocessorHost.MASTER_COPROCESSOR_CONF_KEY, + "org.apache.hadoop.hbase.coprocessor.GenericEndpoint"); util.startMiniCluster(2); cluster = util.getMiniHBaseCluster(); @@ -135,6 +138,34 @@ public void testGeneric() throws Throwable { table.close(); } + @Test + public void testMasterGeneric() throws Throwable { + HBaseAdmin admin = new HBaseAdmin(util.getConfiguration()); + GenericProtocol protocol = admin.coprocessorProxy(GenericProtocol.class); + String workResult1 = protocol.doWork("foo"); + assertEquals("foo", workResult1); + byte[] workResult2 = protocol.doWork(new byte[]{1}); + assertArrayEquals(new byte[]{1}, workResult2); + byte workResult3 = protocol.doWork((byte)1); + assertEquals((byte)1, workResult3); + char workResult4 = protocol.doWork('c'); + assertEquals('c', workResult4); + boolean workResult5 = protocol.doWork(true); + assertEquals(true, workResult5); + short workResult6 = protocol.doWork((short)1); + assertEquals((short)1, workResult6); + int workResult7 = protocol.doWork(5); + assertEquals(5, workResult7); + long workResult8 = protocol.doWork(5l); + assertEquals(5l, workResult8); + double workResult9 = protocol.doWork(6d); + assertEquals(6d, workResult9, 0.01); + float workResult10 = protocol.doWork(6f); + assertEquals(6f, workResult10, 0.01); + Text workResult11 = protocol.doWork(new Text("foo")); + assertEquals(new Text("foo"), workResult11); + } + @Ignore @Test public void testAggregation() throws Throwable { HTable table = new HTable(util.getConfiguration(), TEST_TABLE); @@ -200,6 +231,7 @@ public void testExecDeserialization() throws IOException { dib.reset(dob.getData(), dob.getLength()); Exec after = new Exec(); + after.setConf(HBaseConfiguration.create()); after.readFields(dib); // no error thrown assertEquals(after.getProtocolName(), protocolName); diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java index 545d107cbb02..e152cb88b265 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,15 +19,20 @@ package org.apache.hadoop.hbase.coprocessor; +import static org.mockito.Mockito.when; + import java.io.IOException; import java.util.ArrayList; import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentMap; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.regionserver.InternalScanner; import org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost; @@ -42,14 +46,12 @@ import org.junit.experimental.categories.Category; import org.mockito.Mockito; -import static org.mockito.Mockito.when; - @Category(SmallTests.class) public class TestCoprocessorInterface extends HBaseTestCase { static final Log LOG = LogFactory.getLog(TestCoprocessorInterface.class); - static final String DIR = "test/build/data/TestCoprocessorInterface/"; private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + static final Path DIR = TEST_UTIL.getDataTestDir(); private static class CustomScanner implements RegionScanner { @@ -64,11 +66,35 @@ public boolean next(List results) throws IOException { return delegate.next(results); } + @Override + public boolean next(List results, String metric) + throws IOException { + return delegate.next(results, metric); + } + @Override public boolean next(List result, int limit) throws IOException { return delegate.next(result, limit); } + @Override + public boolean next(List result, int limit, String metric) + throws IOException { + return delegate.next(result, limit, metric); + } + + @Override + public boolean nextRaw(List result, int limit, String metric) + throws IOException { + return delegate.nextRaw(result, limit, metric); + } + + @Override + public boolean nextRaw(List result, String metric) + throws IOException { + return delegate.nextRaw(result, metric); + } + @Override public void close() throws IOException { delegate.close(); @@ -84,6 +110,15 @@ public boolean isFilterDone() { return delegate.isFilterDone(); } + @Override + public boolean reseek(byte[] row) throws IOException { + return false; + } + + @Override + public long getMvccReadPoint() { + return delegate.getMvccReadPoint(); + } } public static class CoprocessorImpl extends BaseRegionObserver { @@ -100,14 +135,19 @@ public static class CoprocessorImpl extends BaseRegionObserver { private boolean postFlushCalled; private boolean preSplitCalled; private boolean postSplitCalled; + private ConcurrentMap sharedData; @Override public void start(CoprocessorEnvironment e) { + sharedData = ((RegionCoprocessorEnvironment)e).getSharedData(); + // using new String here, so that there will be new object on each invocation + sharedData.putIfAbsent("test1", new Object()); startCalled = true; } @Override public void stop(CoprocessorEnvironment e) { + sharedData = null; stopCalled = true; } @@ -182,6 +222,104 @@ boolean wasCompacted() { boolean wasSplit() { return (preSplitCalled && postSplitCalled); } + Map getSharedData() { + return sharedData; + } + } + + public static class CoprocessorII extends BaseRegionObserver { + private ConcurrentMap sharedData; + @Override + public void start(CoprocessorEnvironment e) { + sharedData = ((RegionCoprocessorEnvironment)e).getSharedData(); + sharedData.putIfAbsent("test2", new Object()); + } + @Override + public void stop(CoprocessorEnvironment e) { + sharedData = null; + } + @Override + public void preGet(final ObserverContext e, + final Get get, final List results) throws IOException { + if (1/0 == 1) { + e.complete(); + } + } + + Map getSharedData() { + return sharedData; + } + } + + public void testSharedData() throws IOException { + byte [] tableName = Bytes.toBytes("testtable"); + byte [][] families = { fam1, fam2, fam3 }; + + Configuration hc = initSplit(); + HRegion region = initHRegion(tableName, getName(), hc, + new Class[]{}, families); + + for (int i = 0; i < 3; i++) { + addContent(region, fam3); + region.flushcache(); + } + + region.compactStores(); + + byte [] splitRow = region.checkSplit(); + + assertNotNull(splitRow); + HRegion [] regions = split(region, splitRow); + for (int i = 0; i < regions.length; i++) { + regions[i] = reopenRegion(regions[i], CoprocessorImpl.class, CoprocessorII.class); + } + Coprocessor c = regions[0].getCoprocessorHost(). + findCoprocessor(CoprocessorImpl.class.getName()); + Coprocessor c2 = regions[0].getCoprocessorHost(). + findCoprocessor(CoprocessorII.class.getName()); + Object o = ((CoprocessorImpl)c).getSharedData().get("test1"); + Object o2 = ((CoprocessorII)c2).getSharedData().get("test2"); + assertNotNull(o); + assertNotNull(o2); + // to coprocessors get different sharedDatas + assertFalse(((CoprocessorImpl)c).getSharedData() == ((CoprocessorII)c2).getSharedData()); + for (int i = 1; i < regions.length; i++) { + c = regions[i].getCoprocessorHost(). + findCoprocessor(CoprocessorImpl.class.getName()); + c2 = regions[i].getCoprocessorHost(). + findCoprocessor(CoprocessorII.class.getName()); + // make sure that all coprocessor of a class have identical sharedDatas + assertTrue(((CoprocessorImpl)c).getSharedData().get("test1") == o); + assertTrue(((CoprocessorII)c2).getSharedData().get("test2") == o2); + } + // now have all Environments fail + for (int i = 0; i < regions.length; i++) { + try { + Get g = new Get(regions[i].getStartKey()); + regions[i].get(g, null); + fail(); + } catch (DoNotRetryIOException xc) { + } + assertNull(regions[i].getCoprocessorHost(). + findCoprocessor(CoprocessorII.class.getName())); + } + c = regions[0].getCoprocessorHost(). + findCoprocessor(CoprocessorImpl.class.getName()); + assertTrue(((CoprocessorImpl)c).getSharedData().get("test1") == o); + c = c2 = null; + // perform a GC + System.gc(); + // reopen the region + region = reopenRegion(regions[0], CoprocessorImpl.class, CoprocessorII.class); + c = region.getCoprocessorHost(). + findCoprocessor(CoprocessorImpl.class.getName()); + // CPimpl is unaffected, still the same reference + assertTrue(((CoprocessorImpl)c).getSharedData().get("test1") == o); + c2 = region.getCoprocessorHost(). + findCoprocessor(CoprocessorII.class.getName()); + // new map and object created, hence the reference is different + // hence the old entry was indeed removed by the GC and new one has been created + assertFalse(((CoprocessorII)c2).getSharedData().get("test2") == o2); } public void testCoprocessorInterface() throws IOException { @@ -190,7 +328,7 @@ public void testCoprocessorInterface() throws IOException { Configuration hc = initSplit(); HRegion region = initHRegion(tableName, getName(), hc, - CoprocessorImpl.class, families); + new Class[]{CoprocessorImpl.class}, families); for (int i = 0; i < 3; i++) { addContent(region, fam3); region.flushcache(); @@ -238,12 +376,10 @@ public void testCoprocessorInterface() throws IOException { } } - HRegion reopenRegion(final HRegion closedRegion, Class implClass) + HRegion reopenRegion(final HRegion closedRegion, Class ... implClasses) throws IOException { //HRegionInfo info = new HRegionInfo(tableName, null, null, false); - HRegion r = new HRegion(closedRegion.getTableDir(), closedRegion.getLog(), - closedRegion.getFilesystem(), closedRegion.getConf(), - closedRegion.getRegionInfo(), closedRegion.getTableDesc(), null); + HRegion r = new HRegion(closedRegion); r.initialize(); // this following piece is a hack. currently a coprocessorHost @@ -253,7 +389,9 @@ HRegion reopenRegion(final HRegion closedRegion, Class implClass) RegionCoprocessorHost host = new RegionCoprocessorHost(r, null, conf); r.setCoprocessorHost(host); - host.load(implClass, Coprocessor.PRIORITY_USER, conf); + for (Class implClass : implClasses) { + host.load(implClass, Coprocessor.PRIORITY_USER, conf); + } // we need to manually call pre- and postOpen here since the // above load() is not the real case for CP loading. A CP is // expected to be loaded by default from 1) configuration; or 2) @@ -266,7 +404,7 @@ HRegion reopenRegion(final HRegion closedRegion, Class implClass) } HRegion initHRegion (byte [] tableName, String callingMethod, - Configuration conf, Class implClass, byte [] ... families) + Configuration conf, Class [] implClasses, byte [][] families) throws IOException { HTableDescriptor htd = new HTableDescriptor(tableName); for(byte [] family : families) { @@ -280,10 +418,11 @@ HRegion initHRegion (byte [] tableName, String callingMethod, RegionCoprocessorHost host = new RegionCoprocessorHost(r, null, conf); r.setCoprocessorHost(host); - host.load(implClass, Coprocessor.PRIORITY_USER, conf); - - Coprocessor c = host.findCoprocessor(implClass.getName()); - assertNotNull(c); + for (Class implClass : implClasses) { + host.load(implClass, Coprocessor.PRIORITY_USER, conf); + Coprocessor c = host.findCoprocessor(implClass.getName()); + assertNotNull(c); + } // Here we have to call pre and postOpen explicitly. host.preOpen(); diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorStop.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorStop.java new file mode 100644 index 000000000000..68009863828b --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorStop.java @@ -0,0 +1,125 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.coprocessor; + +import java.io.IOException; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.master.MasterCoprocessorHost; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.FileSystem; + +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import static org.junit.Assert.assertTrue; + + +/** + * Tests for master and regionserver coprocessor stop method + * + */ +@Category(MediumTests.class) +public class TestCoprocessorStop { + private static final Log LOG = LogFactory.getLog(TestCoprocessorStop.class); + private static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static final String MASTER_FILE = + "master" + System.currentTimeMillis(); + private static final String REGIONSERVER_FILE = + "regionserver" + System.currentTimeMillis(); + + public static class FooCoprocessor implements Coprocessor { + @Override + public void start(CoprocessorEnvironment env) throws IOException { + String where = null; + + if (env instanceof MasterCoprocessorEnvironment) { + // if running on HMaster + where = "master"; + } else if (env instanceof RegionServerCoprocessorEnvironment) { + where = "regionserver"; + } else if (env instanceof RegionCoprocessorEnvironment) { + LOG.error("on RegionCoprocessorEnvironment!!"); + } + LOG.info("start coprocessor on " + where); + } + + @Override + public void stop(CoprocessorEnvironment env) throws IOException { + String fileName = null; + + if (env instanceof MasterCoprocessorEnvironment) { + // if running on HMaster + fileName = MASTER_FILE; + } else if (env instanceof RegionServerCoprocessorEnvironment) { + fileName = REGIONSERVER_FILE; + } else if (env instanceof RegionCoprocessorEnvironment) { + LOG.error("on RegionCoprocessorEnvironment!!"); + } + + Configuration conf = UTIL.getConfiguration(); + Path resultFile = new Path(UTIL.getDataTestDir(), fileName); + FileSystem fs = FileSystem.get(conf); + + boolean result = fs.createNewFile(resultFile); + LOG.info("create file " + resultFile + " return rc " + result); + } + } + + @BeforeClass + public static void setupBeforeClass() throws Exception { + Configuration conf = UTIL.getConfiguration(); + + conf.set(CoprocessorHost.MASTER_COPROCESSOR_CONF_KEY, + FooCoprocessor.class.getName()); + conf.set(CoprocessorHost.REGIONSERVER_COPROCESSOR_CONF_KEY, + FooCoprocessor.class.getName()); + + UTIL.startMiniCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + UTIL.shutdownMiniCluster(); + } + + @Test + public void testStopped() throws Exception { + //shutdown hbase only. then check flag file. + MiniHBaseCluster cluster = UTIL.getHBaseCluster(); + LOG.info("shutdown hbase cluster..."); + cluster.shutdown(); + LOG.info("wait for the hbase cluster shutdown..."); + cluster.waitUntilShutDown(); + + Configuration conf = UTIL.getConfiguration(); + FileSystem fs = FileSystem.get(conf); + + Path resultFile = new Path(UTIL.getDataTestDir(), MASTER_FILE); + assertTrue("Master flag file should have been created",fs.exists(resultFile)); + + resultFile = new Path(UTIL.getDataTestDir(), REGIONSERVER_FILE); + assertTrue("RegionServer flag file should have been created",fs.exists(resultFile)); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java new file mode 100644 index 000000000000..fd83023116e2 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java @@ -0,0 +1,323 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.coprocessor; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.Coprocessor; +import org.apache.hadoop.hbase.CoprocessorEnvironment; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.Append; +import org.apache.hadoop.hbase.client.Delete; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.HTableInterface; +import org.apache.hadoop.hbase.client.Increment; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Row; +import org.apache.hadoop.hbase.client.RowMutations; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.master.MasterCoprocessorHost; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.VersionInfo; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import static org.junit.Assert.*; + +/** + * Tests class {@link org.apache.hadoop.hbase.coprocessor.CoprocessorHost.Environment.HTableWrapper} + * by invoking its methods and briefly asserting the result is reasonable. + */ +@Category(MediumTests.class) +public class TestHTableWrapper { + + private static final HBaseTestingUtility util = new HBaseTestingUtility(); + + private static final byte[] TEST_TABLE = Bytes.toBytes("test"); + private static final byte[] TEST_FAMILY = Bytes.toBytes("f1"); + + private static final byte[] ROW_A = Bytes.toBytes("aaa"); + private static final byte[] ROW_B = Bytes.toBytes("bbb"); + private static final byte[] ROW_C = Bytes.toBytes("ccc"); + private static final byte[] ROW_D = Bytes.toBytes("ddd"); + private static final byte[] ROW_E = Bytes.toBytes("eee"); + + private static final byte[] qualifierCol1 = Bytes.toBytes("col1"); + + private static final byte[] bytes1 = Bytes.toBytes(1); + private static final byte[] bytes2 = Bytes.toBytes(2); + private static final byte[] bytes3 = Bytes.toBytes(3); + private static final byte[] bytes4 = Bytes.toBytes(4); + private static final byte[] bytes5 = Bytes.toBytes(5); + + static class DummyRegionObserver extends BaseRegionObserver { + } + + private HTableInterface hTableInterface; + private HTable table; + + @BeforeClass + public static void setupBeforeClass() throws Exception { + util.startMiniCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + util.shutdownMiniCluster(); + } + + @Before + public void before() throws Exception { + table = util.createTable(TEST_TABLE, TEST_FAMILY); + + Put puta = new Put(ROW_A); + puta.add(TEST_FAMILY, qualifierCol1, bytes1); + table.put(puta); + + Put putb = new Put(ROW_B); + putb.add(TEST_FAMILY, qualifierCol1, bytes2); + table.put(putb); + + Put putc = new Put(ROW_C); + putc.add(TEST_FAMILY, qualifierCol1, bytes3); + table.put(putc); + } + + @After + public void after() throws Exception { + try { + if (table != null) { + table.close(); + } + } finally { + util.deleteTable(TEST_TABLE); + } + } + + @Test + public void testHTableInterfaceMethods() throws Exception { + Configuration conf = util.getConfiguration(); + MasterCoprocessorHost cpHost = util.getMiniHBaseCluster().getMaster().getCoprocessorHost(); + Class implClazz = DummyRegionObserver.class; + cpHost.load(implClazz, Coprocessor.PRIORITY_HIGHEST, conf); + CoprocessorEnvironment env = cpHost.findCoprocessorEnvironment(implClazz.getName()); + assertEquals(Coprocessor.VERSION, env.getVersion()); + assertEquals(VersionInfo.getVersion(), env.getHBaseVersion()); + hTableInterface = env.getTable(TEST_TABLE); + checkHTableInterfaceMethods(); + cpHost.shutdown(env); + } + + private void checkHTableInterfaceMethods() throws Exception { + checkConf(); + checkNameAndDescriptor(); + checkAutoFlush(); + checkBufferSize(); + checkExists(); + checkGetRowOrBefore(); + checkAppend(); + checkPutsAndDeletes(); + checkCheckAndPut(); + checkCheckAndDelete(); + checkIncrementColumnValue(); + checkIncrement(); + checkBatch(); + checkMutateRow(); + checkResultScanner(); + + hTableInterface.flushCommits(); + hTableInterface.close(); + } + + private void checkConf() { + Configuration confExpected = util.getConfiguration(); + Configuration confActual = hTableInterface.getConfiguration(); + assertTrue(confExpected == confActual); + } + + private void checkNameAndDescriptor() throws IOException { + assertArrayEquals(TEST_TABLE, hTableInterface.getTableName()); + assertEquals(table.getTableDescriptor(), hTableInterface.getTableDescriptor()); + } + + private void checkAutoFlush() { + boolean initialAutoFlush = hTableInterface.isAutoFlush(); + hTableInterface.setAutoFlush(false); + assertFalse(hTableInterface.isAutoFlush()); + hTableInterface.setAutoFlush(true, true); + assertTrue(hTableInterface.isAutoFlush()); + hTableInterface.setAutoFlush(initialAutoFlush); + } + + private void checkBufferSize() throws IOException { + long initialWriteBufferSize = hTableInterface.getWriteBufferSize(); + hTableInterface.setWriteBufferSize(12345L); + assertEquals(12345L, hTableInterface.getWriteBufferSize()); + hTableInterface.setWriteBufferSize(initialWriteBufferSize); + } + + private void checkExists() throws IOException { + boolean ex = hTableInterface.exists(new Get(ROW_A).addColumn(TEST_FAMILY, qualifierCol1)); + assertTrue(ex); + } + + @SuppressWarnings("deprecation") + private void checkGetRowOrBefore() throws IOException { + Result rowOrBeforeResult = hTableInterface.getRowOrBefore(ROW_A, TEST_FAMILY); + assertArrayEquals(ROW_A, rowOrBeforeResult.getRow()); + } + + private void checkAppend() throws IOException { + final byte[] appendValue = Bytes.toBytes("append"); + Append append = new Append(qualifierCol1).add(TEST_FAMILY, qualifierCol1, appendValue); + Result appendResult = hTableInterface.append(append); + byte[] appendedRow = appendResult.getRow(); + checkRowValue(appendedRow, appendValue); + } + + private void checkPutsAndDeletes() throws IOException { + // put: + Put putD = new Put(ROW_D).add(TEST_FAMILY, qualifierCol1, bytes2); + hTableInterface.put(putD); + checkRowValue(ROW_D, bytes2); + + // delete: + Delete delete = new Delete(ROW_D); + hTableInterface.delete(delete); + checkRowValue(ROW_D, null); + + // multiple puts: + Put[] puts = new Put[] { new Put(ROW_D).add(TEST_FAMILY, qualifierCol1, bytes2), + new Put(ROW_E).add(TEST_FAMILY, qualifierCol1, bytes3) }; + hTableInterface.put(Arrays.asList(puts)); + checkRowsValues(new byte[][] { ROW_D, ROW_E }, new byte[][] { bytes2, bytes3 }); + + // multiple deletes: + Delete[] deletes = new Delete[] { new Delete(ROW_D), new Delete(ROW_E) }; + hTableInterface.delete(new ArrayList(Arrays.asList(deletes))); + checkRowsValues(new byte[][] { ROW_D, ROW_E }, new byte[][] { null, null }); + } + + private void checkCheckAndPut() throws IOException { + Put putC = new Put(ROW_C).add(TEST_FAMILY, qualifierCol1, bytes5); + assertFalse(hTableInterface.checkAndPut(ROW_C, TEST_FAMILY, qualifierCol1, /* expect */ + bytes4, putC/* newValue */)); + assertTrue(hTableInterface.checkAndPut(ROW_C, TEST_FAMILY, qualifierCol1, /* expect */ + bytes3, putC/* newValue */)); + checkRowValue(ROW_C, bytes5); + } + + private void checkCheckAndDelete() throws IOException { + Delete delete = new Delete(ROW_C); + assertFalse(hTableInterface.checkAndDelete(ROW_C, TEST_FAMILY, qualifierCol1, bytes4, delete)); + assertTrue(hTableInterface.checkAndDelete(ROW_C, TEST_FAMILY, qualifierCol1, bytes5, delete)); + checkRowValue(ROW_C, null); + } + + private void checkIncrementColumnValue() throws IOException { + hTableInterface.put(new Put(ROW_A).add(TEST_FAMILY, qualifierCol1, Bytes.toBytes(1L))); + checkRowValue(ROW_A, Bytes.toBytes(1L)); + + final long newVal = hTableInterface + .incrementColumnValue(ROW_A, TEST_FAMILY, qualifierCol1, 10L); + assertEquals(11L, newVal); + checkRowValue(ROW_A, Bytes.toBytes(11L)); + } + + private void checkIncrement() throws IOException { + hTableInterface.increment(new Increment(ROW_A).addColumn(TEST_FAMILY, qualifierCol1, -15L)); + checkRowValue(ROW_A, Bytes.toBytes(-4L)); + } + + private void checkBatch() throws IOException, InterruptedException { + Object[] results1 = hTableInterface.batch(Arrays.asList(new Row[] { + new Increment(ROW_A).addColumn(TEST_FAMILY, qualifierCol1, 2L), + new Increment(ROW_A).addColumn(TEST_FAMILY, qualifierCol1, 2L) })); + assertEquals(2, results1.length); + for (Object r2 : results1) { + assertTrue(r2 instanceof Result); + } + checkRowValue(ROW_A, Bytes.toBytes(0L)); + Object[] results2 = new Result[2]; + hTableInterface.batch( + Arrays.asList(new Row[] { new Increment(ROW_A).addColumn(TEST_FAMILY, qualifierCol1, 2L), + new Increment(ROW_A).addColumn(TEST_FAMILY, qualifierCol1, 2L) }), results2); + for (Object r2 : results2) { + assertTrue(r2 instanceof Result); + } + checkRowValue(ROW_A, Bytes.toBytes(4L)); + } + + private void checkMutateRow() throws IOException { + Put put = new Put(ROW_A).add(TEST_FAMILY, qualifierCol1, bytes1); + RowMutations rowMutations = new RowMutations(ROW_A); + rowMutations.add(put); + hTableInterface.mutateRow(rowMutations); + checkRowValue(ROW_A, bytes1); + } + + private void checkResultScanner() throws IOException { + ResultScanner resultScanner = hTableInterface.getScanner(TEST_FAMILY); + Result[] results = resultScanner.next(10); + assertEquals(3, results.length); + + resultScanner = hTableInterface.getScanner(TEST_FAMILY, qualifierCol1); + results = resultScanner.next(10); + assertEquals(3, results.length); + + resultScanner = hTableInterface.getScanner(new Scan(ROW_A, ROW_C)); + results = resultScanner.next(10); + assertEquals(2, results.length); + } + + private void checkRowValue(byte[] row, byte[] expectedValue) throws IOException { + Get get = new Get(row).addColumn(TEST_FAMILY, qualifierCol1); + Result result = hTableInterface.get(get); + byte[] actualValue = result.getValue(TEST_FAMILY, qualifierCol1); + assertArrayEquals(expectedValue, actualValue); + } + + private void checkRowsValues(byte[][] rows, byte[][] expectedValues) throws IOException { + if (rows.length != expectedValues.length) { + throw new IllegalArgumentException(); + } + Get[] gets = new Get[rows.length]; + for (int i = 0; i < gets.length; i++) { + gets[i] = new Get(rows[i]).addColumn(TEST_FAMILY, qualifierCol1); + } + Result[] results = hTableInterface.get(Arrays.asList(gets)); + for (int i = 0; i < expectedValues.length; i++) { + byte[] actualValue = results[i].getValue(TEST_FAMILY, qualifierCol1); + assertArrayEquals(expectedValues[i], actualValue); + } + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java index 0e0b4228d2c8..fe44ab9a50b9 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java index d7e0f65e3bc9..02068c54ddab 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java index b369c768c275..53445e6a1708 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -40,8 +39,11 @@ import org.apache.hadoop.hbase.master.AssignmentManager; import org.apache.hadoop.hbase.master.HMaster; import org.apache.hadoop.hbase.master.MasterCoprocessorHost; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; import org.apache.hadoop.hbase.regionserver.HRegionServer; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Threads; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -89,6 +91,16 @@ public static class CPMasterObserver implements MasterObserver { private boolean postStartMasterCalled; private boolean startCalled; private boolean stopCalled; + private boolean preSnapshotCalled; + private boolean postSnapshotCalled; + private boolean preCloneSnapshotCalled; + private boolean postCloneSnapshotCalled; + private boolean preRestoreSnapshotCalled; + private boolean postRestoreSnapshotCalled; + private boolean preDeleteSnapshotCalled; + private boolean postDeleteSnapshotCalled; + private boolean preGetTableDescriptorsCalled; + private boolean postGetTableDescriptorsCalled; public void enableBypass(boolean bypass) { this.bypass = bypass; @@ -121,6 +133,16 @@ public void resetStates() { postBalanceCalled = false; preBalanceSwitchCalled = false; postBalanceSwitchCalled = false; + preSnapshotCalled = false; + postSnapshotCalled = false; + preCloneSnapshotCalled = false; + postCloneSnapshotCalled = false; + preRestoreSnapshotCalled = false; + postRestoreSnapshotCalled = false; + preDeleteSnapshotCalled = false; + postDeleteSnapshotCalled = false; + preGetTableDescriptorsCalled = false; + postGetTableDescriptorsCalled = false; } @Override @@ -460,10 +482,98 @@ public void stop(CoprocessorEnvironment env) throws IOException { public boolean wasStarted() { return startCalled; } public boolean wasStopped() { return stopCalled; } + + @Override + public void preSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + preSnapshotCalled = true; + } + + @Override + public void postSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + postSnapshotCalled = true; + } + + public boolean wasSnapshotCalled() { + return preSnapshotCalled && postSnapshotCalled; + } + + @Override + public void preCloneSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + preCloneSnapshotCalled = true; + } + + @Override + public void postCloneSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + postCloneSnapshotCalled = true; + } + + public boolean wasCloneSnapshotCalled() { + return preCloneSnapshotCalled && postCloneSnapshotCalled; + } + + @Override + public void preRestoreSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + preRestoreSnapshotCalled = true; + } + + @Override + public void postRestoreSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot, final HTableDescriptor hTableDescriptor) + throws IOException { + postRestoreSnapshotCalled = true; + } + + public boolean wasRestoreSnapshotCalled() { + return preRestoreSnapshotCalled && postRestoreSnapshotCalled; + } + + @Override + public void preDeleteSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot) throws IOException { + preDeleteSnapshotCalled = true; + } + + @Override + public void postDeleteSnapshot(final ObserverContext ctx, + final SnapshotDescription snapshot) throws IOException { + postDeleteSnapshotCalled = true; + } + + public boolean wasDeleteSnapshotCalled() { + return preDeleteSnapshotCalled && postDeleteSnapshotCalled; + } + + @Override + public void preGetTableDescriptors(ObserverContext ctx, + List tableNamesList, List descriptors) throws IOException { + preGetTableDescriptorsCalled = true; + } + + @Override + public void postGetTableDescriptors(ObserverContext ctx, + List descriptors) throws IOException { + postGetTableDescriptorsCalled = true; + } + + public boolean wasGetTableDescriptorsCalled() { + return preGetTableDescriptorsCalled && postGetTableDescriptorsCalled; + } } private static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static byte[] TEST_SNAPSHOT = Bytes.toBytes("observed_snapshot"); private static byte[] TEST_TABLE = Bytes.toBytes("observed_table"); + private static byte[] TEST_CLONE = Bytes.toBytes("observed_clone"); private static byte[] TEST_FAMILY = Bytes.toBytes("fam1"); private static byte[] TEST_FAMILY2 = Bytes.toBytes("fam2"); @@ -472,6 +582,8 @@ public static void setupBeforeClass() throws Exception { Configuration conf = UTIL.getConfiguration(); conf.set(CoprocessorHost.MASTER_COPROCESSOR_CONF_KEY, CPMasterObserver.class.getName()); + // Enable snapshot + conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); // We need more than one data server on this test UTIL.startMiniCluster(2); } @@ -539,7 +651,7 @@ public void testTableOperations() throws Exception { // modify table htd.setMaxFileSize(512 * 1024 * 1024); - admin.modifyTable(TEST_TABLE, htd); + modifyTableSync(admin, TEST_TABLE, htd); // preModifyTable can't bypass default action. assertTrue("Test table should have been modified", cp.wasModifyTableCalled()); @@ -582,7 +694,7 @@ public void testTableOperations() throws Exception { // modify table htd.setMaxFileSize(512 * 1024 * 1024); - admin.modifyTable(TEST_TABLE, htd); + modifyTableSync(admin, TEST_TABLE, htd); assertTrue("Test table should have been modified", cp.wasModifyTableCalled()); @@ -627,6 +739,19 @@ public void testTableOperations() throws Exception { cp.wasDeleteTableCalled()); } + private void modifyTableSync(HBaseAdmin admin, byte[] tableName, HTableDescriptor htd) + throws IOException { + admin.modifyTable(tableName, htd); + //wait until modify table finishes + for (int t = 0; t < 100; t++) { //10 sec timeout + HTableDescriptor td = admin.getTableDescriptor(htd.getName()); + if (td.equals(htd)) { + break; + } + Threads.sleep(100); + } + } + @Test public void testRegionTransitionOperations() throws Exception { MiniHBaseCluster cluster = UTIL.getHBaseCluster(); @@ -639,8 +764,8 @@ public void testRegionTransitionOperations() throws Exception { cp.resetStates(); HTable table = UTIL.createTable(TEST_TABLE, TEST_FAMILY); - int countOfRegions = UTIL.createMultiRegions(table, TEST_FAMILY); - UTIL.waitUntilAllRegionsAssigned(countOfRegions); + UTIL.createMultiRegions(table, TEST_FAMILY); + UTIL.waitUntilAllRegionsAssigned(TEST_TABLE); NavigableMap regions = table.getRegionLocations(); Map.Entry firstGoodPair = null; @@ -684,6 +809,8 @@ public void testRegionTransitionOperations() throws Exception { // move half the open regions from RS 0 to RS 1 HRegionServer rs = cluster.getRegionServer(0); byte[] destRS = Bytes.toBytes(cluster.getRegionServer(1).getServerName().toString()); + //Make sure no regions are in transition now + waitForRITtoBeZero(master); List openRegions = rs.getOnlineRegions(); int moveCnt = openRegions.size()/2; for (int i=0; i transRegions = @@ -700,13 +893,22 @@ public void testRegionTransitionOperations() throws Exception { for (AssignmentManager.RegionState state : transRegions) { mgr.waitOnRegionToClearRegionsInTransition(state.getRegion()); } + } - // now trigger a balance - master.balanceSwitch(true); - boolean balanceRun = master.balance(); - assertTrue("Coprocessor should be called on region rebalancing", - cp.wasBalanceCalled()); - table.close(); + @Test + public void testTableDescriptorsEnumeration() throws Exception { + MiniHBaseCluster cluster = UTIL.getHBaseCluster(); + + HMaster master = cluster.getMaster(); + MasterCoprocessorHost host = master.getCoprocessorHost(); + CPMasterObserver cp = (CPMasterObserver)host.findCoprocessor( + CPMasterObserver.class.getName()); + cp.resetStates(); + + master.getHTableDescriptors(); + + assertTrue("Coprocessor should be called on table descriptors request", + cp.wasGetTableDescriptorsCalled()); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMultiRowMutationProtocol.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMultiRowMutationProtocol.java new file mode 100644 index 000000000000..f6a44f541a90 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMultiRowMutationProtocol.java @@ -0,0 +1,210 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.coprocessor; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.client.Delete; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Mutation; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Assert; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * A test class to cover multi row mutations protocol + */ +@Category(MediumTests.class) +public class TestMultiRowMutationProtocol { + + private static final byte[] TEST_TABLE = Bytes.toBytes("TestTable"); + private static final byte[] TEST_FAMILY = Bytes.toBytes("TestFamily"); + private static final byte[] INVALID_FAMILY = Bytes.toBytes("InvalidFamily"); + private static final byte[] TEST_QUALIFIER = Bytes.toBytes("TestQualifier"); + private static byte[] ROW = Bytes.toBytes("testRow"); + + private static final int ROWSIZE = 20; + private static final int rowSeperator1 = 5; + private static final int rowSeperator2 = 12; + private static byte[][] ROWS = makeN(ROW, ROWSIZE); + + private static HBaseTestingUtility util = new HBaseTestingUtility(); + private static MiniHBaseCluster cluster = null; + + private HTable table = null; + + @BeforeClass + public static void setupBeforeClass() throws Exception { + // set configure to indicate which cp should be loaded + Configuration conf = util.getConfiguration(); + conf.setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, + "org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint"); + + util.startMiniCluster(2); + cluster = util.getMiniHBaseCluster(); + + HTable table = util.createTable(TEST_TABLE, TEST_FAMILY); + util.createMultiRegions(util.getConfiguration(), table, TEST_FAMILY, + new byte[][] { HConstants.EMPTY_BYTE_ARRAY, + ROWS[rowSeperator1], ROWS[rowSeperator2] }); + + for (int i = 0; i < ROWSIZE; i++) { + Put put = new Put(ROWS[i]); + put.setWriteToWAL(false); + put.add(TEST_FAMILY, TEST_QUALIFIER, Bytes.toBytes(i)); + table.put(put); + } + + // sleep here is an ugly hack to allow region transitions to finish + long timeout = System.currentTimeMillis() + (15 * 1000); + while ((System.currentTimeMillis() < timeout) && + (table.getRegionsInfo().size() != 3)) { + Thread.sleep(250); + } + table.close(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + util.shutdownMiniCluster(); + } + + @Before + public void setup() throws IOException { + table = new HTable(util.getConfiguration(), TEST_TABLE); + for (int i = 0; i < ROWSIZE; i++) { + Put put = new Put(ROWS[i]); + put.setWriteToWAL(false); + put.add(TEST_FAMILY, TEST_QUALIFIER, Bytes.toBytes(i)); + table.put(put); + } + } + + @After + public void tearDown() throws IOException { + table.close(); + } + + @Test + public void testMultiRowMutations() throws IOException { + List mutations = new ArrayList(); + + Put put = new Put(ROWS[1]); + put.add(TEST_FAMILY, TEST_QUALIFIER, Bytes.toBytes(2 * 1)); + mutations.add(put); + Delete del = new Delete(ROWS[3]); + del.deleteColumns(TEST_FAMILY, TEST_QUALIFIER); + mutations.add(del); + + MultiRowMutationProtocol p = + table.coprocessorProxy(MultiRowMutationProtocol.class, mutations.get(0).getRow()); + try { + p.mutateRows(mutations); + } catch (IOException e) { + Assert.assertTrue(false); + } + + Get get = new Get(ROWS[1]); + get.addColumn(TEST_FAMILY, TEST_QUALIFIER); + Result result = table.get(get); + Assert.assertEquals(2, Bytes.toInt(result.getValue(TEST_FAMILY, TEST_QUALIFIER))); + + + get = new Get(ROWS[3]); + get.addColumn(TEST_FAMILY, TEST_QUALIFIER); + result = table.get(get); + Assert.assertNull(result.getValue(TEST_FAMILY, TEST_QUALIFIER)); + } + + @Test + public void testMultiRowMutationsAcrossRegions() throws IOException { + List mutations = new ArrayList(); + + Put put = new Put(ROWS[1]); + put.add(TEST_FAMILY, TEST_QUALIFIER, Bytes.toBytes(2 * 1)); + mutations.add(put); + Delete del = new Delete(ROWS[7]); + del.deleteColumns(TEST_FAMILY, TEST_QUALIFIER); + mutations.add(del); + + MultiRowMutationProtocol p = + table.coprocessorProxy(MultiRowMutationProtocol.class, mutations.get(0).getRow()); + try { + p.mutateRows(mutations); + Assert.assertTrue(false); + } catch (IOException e) { + } + } + + @Test + public void testInvalidFamiliy() throws IOException { + List invalids = new ArrayList(); + Put put = new Put(ROWS[1]); + put.add(INVALID_FAMILY, TEST_QUALIFIER, Bytes.toBytes(2 * 1)); + invalids.add(put); + + MultiRowMutationProtocol p = + table.coprocessorProxy(MultiRowMutationProtocol.class, ROWS[1]); + try { + p.mutateRows(invalids); + Assert.assertTrue(false); + } catch (IOException e) { + } + + List valids = new ArrayList(); + put = new Put(ROWS[1]); + put.add(TEST_FAMILY, TEST_QUALIFIER, Bytes.toBytes(2 * 1)); + valids.add(put); + try { + p.mutateRows(valids); + } catch (IOException e) { + Assert.assertTrue(false); + } + } + + /** + * an infrastructure method to prepare rows for the testtable. + * @param base + * @param n + * @return + */ + private static byte[][] makeN(byte[] base, int n) { + byte[][] ret = new byte[n][]; + for (int i = 0; i < n; i++) { + ret[i] = Bytes.add(base, Bytes.toBytes(i)); + } + return ret; + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestOpenTableInCoprocessor.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestOpenTableInCoprocessor.java new file mode 100644 index 000000000000..c0d4a50c4ef9 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestOpenTableInCoprocessor.java @@ -0,0 +1,195 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.coprocessor; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.Collections; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.SynchronousQueue; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; + +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.HTableInterface; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.regionserver.wal.WALEdit; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Threads; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test that a coprocessor can open a connection and write to another table, inside a hook. + */ +@Category(MediumTests.class) +public class TestOpenTableInCoprocessor { + + private static final byte[] otherTable = Bytes.toBytes("otherTable"); + private static final byte[] primaryTable = Bytes.toBytes("primary"); + private static final byte[] family = new byte[] { 'f' }; + + private static boolean [] completed = new boolean[1]; + + /** + * Custom coprocessor that just copies the write to another table. + */ + public static class SendToOtherTableCoprocessor extends BaseRegionObserver { + + @Override + public void prePut(ObserverContext e, Put put, WALEdit edit, + boolean writeToWAL) throws IOException { + HTableInterface table = e.getEnvironment().getTable(otherTable); + table.put(put); + table.flushCommits(); + completed[0] = true; + table.close(); + } + + } + + private static boolean [] completedWithPool = new boolean [1] ; + + public static class CustomThreadPoolCoprocessor extends BaseRegionObserver { + + /** + * Get a pool that has only ever one thread. A second action added to the pool (running + * concurrently), will cause an exception. + * @return + */ + private ExecutorService getPool() { + int maxThreads = 1; + long keepAliveTime = 60; + ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads, keepAliveTime, + TimeUnit.SECONDS, new SynchronousQueue(), + Threads.newDaemonThreadFactory("hbase-table")); + pool.allowCoreThreadTimeOut(true); + return pool; + } + + @Override + public void prePut(ObserverContext e, Put put, WALEdit edit, + boolean writeToWAL) throws IOException { + HTableInterface table = e.getEnvironment().getTable(otherTable, getPool()); + Put p = new Put(new byte[] { 'a' }); + p.add(family, null, new byte[] { 'a' }); + try { + table.batch(Collections.singletonList(put)); + } catch (InterruptedException e1) { + throw new IOException(e1); + } + completedWithPool[0] = true; + table.close(); + } + } + + private static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @BeforeClass + public static void setupCluster() throws Exception { + UTIL.startMiniCluster(); + } + + @After + public void cleanupTestTable() throws Exception { + UTIL.getHBaseAdmin().disableTable(primaryTable); + UTIL.getHBaseAdmin().deleteTable(primaryTable); + + UTIL.getHBaseAdmin().disableTable(otherTable); + UTIL.getHBaseAdmin().deleteTable(otherTable); + + } + + @AfterClass + public static void teardownCluster() throws Exception{ + UTIL.shutdownMiniCluster(); + } + + @Test + public void testCoprocessorCanCreateConnectionToRemoteTable() throws Throwable { + runCoprocessorConnectionToRemoteTable(SendToOtherTableCoprocessor.class, completed); + } + + @Test + public void testCoprocessorCanCreateConnectionToRemoteTableWithCustomPool() throws Throwable { + runCoprocessorConnectionToRemoteTable(CustomThreadPoolCoprocessor.class, completedWithPool); + } + + private void runCoprocessorConnectionToRemoteTable(Class clazz, + boolean[] completeCheck) throws Throwable { + HTableDescriptor primary = new HTableDescriptor(primaryTable); + primary.addFamily(new HColumnDescriptor(family)); + // add our coprocessor + primary.addCoprocessor(clazz.getName()); + + HTableDescriptor other = new HTableDescriptor(otherTable); + other.addFamily(new HColumnDescriptor(family)); + + + HBaseAdmin admin = UTIL.getHBaseAdmin(); + admin.createTable(primary); + admin.createTable(other); + + HTable table = new HTable(UTIL.getConfiguration(), "primary"); + Put p = new Put(new byte[] { 'a' }); + p.add(family, null, new byte[] { 'a' }); + table.put(p); + table.flushCommits(); + table.close(); + + HTable target = new HTable(UTIL.getConfiguration(), otherTable); + assertTrue("Didn't complete update to target table!", completeCheck[0]); + assertEquals("Didn't find inserted row", 1, getKeyValueCount(target)); + target.close(); + + } + + /** + * Count the number of keyvalue in the table. Scans all possible versions + * @param table table to scan + * @return number of keyvalues over all rows in the table + * @throws IOException + */ + private int getKeyValueCount(HTable table) throws IOException { + Scan scan = new Scan(); + scan.setMaxVersions(Integer.MAX_VALUE - 1); + + ResultScanner results = table.getScanner(scan); + int count = 0; + for (Result res : results) { + count += res.list().size(); + System.out.println(count + ") " + res); + } + results.close(); + + return count; + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java index 7a118dab9dda..ab1b0e0a3308 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java index 1b3b6df3d329..75d89042b8a0 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,6 +19,11 @@ package org.apache.hadoop.hbase.coprocessor; +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + import java.io.IOException; import java.lang.reflect.Method; import java.util.ArrayList; @@ -29,8 +33,13 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.client.*; +import org.apache.hadoop.hbase.io.hfile.CacheConfig; +import org.apache.hadoop.hbase.io.hfile.HFile; +import org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.InternalScanner; import org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost; @@ -39,18 +48,14 @@ import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; import org.apache.hadoop.hbase.util.JVMClusterUtil; - import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; -import static org.junit.Assert.*; - @Category(MediumTests.class) public class TestRegionObserverInterface { static final Log LOG = LogFactory.getLog(TestRegionObserverInterface.class); - static final String DIR = "test/build/data/TestRegionObserver/"; public static final byte[] TEST_TABLE = Bytes.toBytes("TestTable"); public final static byte[] A = Bytes.toBytes("a"); @@ -97,9 +102,9 @@ public void testRegionObserver() throws IOException { verifyMethodResult(SimpleRegionObserver.class, new String[] {"hadPreGet", "hadPostGet", "hadPrePut", "hadPostPut", - "hadDelete"}, + "hadPreBatchMutate", "hadPostBatchMutate", "hadDelete"}, TEST_TABLE, - new Boolean[] {false, false, true, true, false} + new Boolean[] {false, false, true, true, true, true, false} ); Get get = new Get(ROW); @@ -123,9 +128,9 @@ public void testRegionObserver() throws IOException { verifyMethodResult(SimpleRegionObserver.class, new String[] {"hadPreGet", "hadPostGet", "hadPrePut", "hadPostPut", - "hadDelete"}, + "hadPreBatchMutate", "hadPostBatchMutate", "hadDelete"}, TEST_TABLE, - new Boolean[] {true, true, true, true, true} + new Boolean[] {true, true, true, true, true, true, true} ); util.deleteTable(tableName); table.close(); @@ -301,11 +306,24 @@ public boolean next(List results) throws IOException { } @Override - public boolean next(List results, int limit) throws IOException { + public boolean next(List results, String metric) + throws IOException { + return next(results, -1, metric); + } + + @Override + public boolean next(List results, int limit) + throws IOException{ + return next(results, limit, null); + } + + @Override + public boolean next(List results, int limit, String metric) + throws IOException { List internalResults = new ArrayList(); boolean hasMore; do { - hasMore = scanner.next(internalResults, limit); + hasMore = scanner.next(internalResults, limit, metric); if (!internalResults.isEmpty()) { long row = Bytes.toLong(internalResults.get(0).getRow()); if (row % 2 == 0) { @@ -416,6 +434,37 @@ public void testCompactionOverride() throws Exception { table.close(); } + @Test + public void bulkLoadHFileTest() throws Exception { + String testName = TestRegionObserverInterface.class.getName()+".bulkLoadHFileTest"; + byte[] tableName = TEST_TABLE; + Configuration conf = util.getConfiguration(); + HTable table = util.createTable(tableName, new byte[][] {A, B, C}); + + verifyMethodResult(SimpleRegionObserver.class, + new String[] {"hadPreBulkLoadHFile", "hadPostBulkLoadHFile"}, + tableName, + new Boolean[] {false, false} + ); + + FileSystem fs = util.getTestFileSystem(); + final Path dir = util.getDataTestDir(testName).makeQualified(fs); + Path familyDir = new Path(dir, Bytes.toString(A)); + + createHFile(util.getConfiguration(), fs, new Path(familyDir,Bytes.toString(A)), A, A); + + //Bulk load + new LoadIncrementalHFiles(conf).doBulkLoad(dir, new HTable(conf, tableName)); + + verifyMethodResult(SimpleRegionObserver.class, + new String[] {"hadPreBulkLoadHFile", "hadPostBulkLoadHFile"}, + tableName, + new Boolean[] {true, true} + ); + util.deleteTable(tableName); + table.close(); + } + // check each region whether the coprocessor upcalls are called or not. private void verifyMethodResult(Class c, String methodName[], byte[] tableName, Object value[]) throws IOException { @@ -444,6 +493,25 @@ private void verifyMethodResult(Class c, String methodName[], byte[] tableName, } } + private static void createHFile( + Configuration conf, + FileSystem fs, Path path, + byte[] family, byte[] qualifier) throws IOException { + HFile.Writer writer = HFile.getWriterFactory(conf, new CacheConfig(conf)) + .withPath(fs, path) + .withComparator(KeyValue.KEY_COMPARATOR) + .create(); + long now = System.currentTimeMillis(); + try { + for (int i =1;i<=9;i++) { + KeyValue kv = new KeyValue(Bytes.toBytes(i+""), family, qualifier, now, Bytes.toBytes(i+"")); + writer.append(kv); + } + } finally { + writer.close(); + } + } + private static byte [][] makeN(byte [] base, int n) { byte [][] ret = new byte[n][]; for(int i=0;i c, + Store store, Scan scan, NavigableSet targetCols, KeyValueScanner s) + throws IOException { + scan.setFilter(new NoDataFilter()); + return new StoreScanner(store, store.getScanInfo(), scan, targetCols); + } + } + + /** + * Don't allow any data in a flush by creating a custom {@link StoreScanner}. + */ + public static class NoDataFromFlush extends BaseRegionObserver { + @Override + public InternalScanner preFlushScannerOpen(ObserverContext c, + Store store, KeyValueScanner memstoreScanner, InternalScanner s) throws IOException { + Scan scan = new Scan(); + scan.setFilter(new NoDataFilter()); + return new StoreScanner(store, store.getScanInfo(), scan, + Collections.singletonList(memstoreScanner), ScanType.MINOR_COMPACT, store.getHRegion() + .getSmallestReadPoint(), HConstants.OLDEST_TIMESTAMP); + } + } + + /** + * Don't allow any data to be written out in the compaction by creating a custom + * {@link StoreScanner}. + */ + public static class NoDataFromCompaction extends BaseRegionObserver { + @Override + public InternalScanner preCompactScannerOpen(ObserverContext c, + Store store, List scanners, ScanType scanType, + long earliestPutTs, InternalScanner s) throws IOException { + Scan scan = new Scan(); + scan.setFilter(new NoDataFilter()); + return new StoreScanner(store, store.getScanInfo(), scan, scanners, ScanType.MINOR_COMPACT, + store.getHRegion().getSmallestReadPoint(), HConstants.OLDEST_TIMESTAMP); + } + } + + + HRegion initHRegion(byte[] tableName, String callingMethod, Configuration conf, + byte[]... families) throws IOException { + HTableDescriptor htd = new HTableDescriptor(tableName); + for (byte[] family : families) { + htd.addFamily(new HColumnDescriptor(family)); + } + HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); + Path path = new Path(DIR + callingMethod); + HRegion r = HRegion.createHRegion(info, path, conf, htd); + // this following piece is a hack. currently a coprocessorHost + // is secretly loaded at OpenRegionHandler. we don't really + // start a region server here, so just manually create cphost + // and set it to region. + RegionCoprocessorHost host = new RegionCoprocessorHost(r, null, conf); + r.setCoprocessorHost(host); + return r; + } + + @Test + public void testRegionObserverScanTimeStacking() throws Exception { + byte[] ROW = Bytes.toBytes("testRow"); + byte[] TABLE = Bytes.toBytes(getClass().getName()); + byte[] A = Bytes.toBytes("A"); + byte[][] FAMILIES = new byte[][] { A }; + + Configuration conf = HBaseConfiguration.create(); + HRegion region = initHRegion(TABLE, getClass().getName(), conf, FAMILIES); + RegionCoprocessorHost h = region.getCoprocessorHost(); + h.load(NoDataFromScan.class, Coprocessor.PRIORITY_HIGHEST, conf); + h.load(EmptyRegionObsever.class, Coprocessor.PRIORITY_USER, conf); + + Put put = new Put(ROW); + put.add(A, A, A); + region.put(put); + + Get get = new Get(ROW); + Result r = region.get(get); + assertNull( + "Got an unexpected number of rows - no data should be returned with the NoDataFromScan coprocessor. Found: " + + r, r.list()); + } + + @Test + public void testRegionObserverFlushTimeStacking() throws Exception { + byte[] ROW = Bytes.toBytes("testRow"); + byte[] TABLE = Bytes.toBytes(getClass().getName()); + byte[] A = Bytes.toBytes("A"); + byte[][] FAMILIES = new byte[][] { A }; + + Configuration conf = HBaseConfiguration.create(); + HRegion region = initHRegion(TABLE, getClass().getName(), conf, FAMILIES); + RegionCoprocessorHost h = region.getCoprocessorHost(); + h.load(NoDataFromFlush.class, Coprocessor.PRIORITY_HIGHEST, conf); + h.load(EmptyRegionObsever.class, Coprocessor.PRIORITY_USER, conf); + + // put a row and flush it to disk + Put put = new Put(ROW); + put.add(A, A, A); + region.put(put); + region.flushcache(); + Get get = new Get(ROW); + Result r = region.get(get); + assertNull( + "Got an unexpected number of rows - no data should be returned with the NoDataFromScan coprocessor. Found: " + + r, r.list()); + } + + /* + * Custom HRegion which uses CountDownLatch to signal the completion of compaction + */ + public static class CompactionCompletionNotifyingRegion extends HRegion { + private static volatile CountDownLatch compactionStateChangeLatch = null; + + public CompactionCompletionNotifyingRegion(Path tableDir, HLog log, FileSystem fs, + Configuration confParam, HRegionInfo info, HTableDescriptor htd, + RegionServerServices rsServices) { + super(tableDir, log, fs, confParam, info, htd, rsServices); + } + + public CountDownLatch getCompactionStateChangeLatch() { + if (compactionStateChangeLatch == null) compactionStateChangeLatch = new CountDownLatch(1); + return compactionStateChangeLatch; + } + + @Override + public boolean compact(CompactionRequest cr) throws IOException { + boolean ret = super.compact(cr); + if (ret) compactionStateChangeLatch.countDown(); + return ret; + } + } + + /** + * Unfortunately, the easiest way to test this is to spin up a mini-cluster since we want to do + * the usual compaction mechanism on the region, rather than going through the backdoor to the + * region + */ + @Test + @Category(MediumTests.class) + public void testRegionObserverCompactionTimeStacking() throws Exception { + // setup a mini cluster so we can do a real compaction on a region + Configuration conf = UTIL.getConfiguration(); + conf.setClass(HConstants.REGION_IMPL, CompactionCompletionNotifyingRegion.class, HRegion.class); + conf.setInt("hbase.hstore.compaction.min", 2); + UTIL.startMiniCluster(); + String tableName = "testRegionObserverCompactionTimeStacking"; + byte[] ROW = Bytes.toBytes("testRow"); + byte[] A = Bytes.toBytes("A"); + HTableDescriptor desc = new HTableDescriptor(tableName); + desc.addFamily(new HColumnDescriptor(A)); + desc.addCoprocessor(EmptyRegionObsever.class.getName(), null, Coprocessor.PRIORITY_USER, null); + desc.addCoprocessor(NoDataFromCompaction.class.getName(), null, Coprocessor.PRIORITY_HIGHEST, + null); + + HBaseAdmin admin = UTIL.getHBaseAdmin(); + admin.createTable(desc); + + HTable table = new HTable(conf, desc.getName()); + + // put a row and flush it to disk + Put put = new Put(ROW); + put.add(A, A, A); + table.put(put); + table.flushCommits(); + + HRegionServer rs = UTIL.getRSForFirstRegionInTable(desc.getName()); + List regions = rs.getOnlineRegions(desc.getName()); + assertEquals("More than 1 region serving test table with 1 row", 1, regions.size()); + HRegion region = regions.get(0); + admin.flush(region.getRegionName()); + CountDownLatch latch = ((CompactionCompletionNotifyingRegion)region) + .getCompactionStateChangeLatch(); + + // put another row and flush that too + put = new Put(Bytes.toBytes("anotherrow")); + put.add(A, A, A); + table.put(put); + table.flushCommits(); + admin.flush(region.getRegionName()); + + // run a compaction, which normally would should get rid of the data + Store s = region.getStores().get(A); + CompactionRequest request = new CompactionRequest(region, s, Store.PRIORITY_USER); + rs.compactSplitThread.requestCompaction(region, s, + "compact for testRegionObserverCompactionTimeStacking", Store.PRIORITY_USER, request); + // wait for the compaction to complete + latch.await(); + + // check both rows to ensure that they aren't there + Get get = new Get(ROW); + Result r = table.get(get); + assertNull( + "Got an unexpected number of rows - no data should be returned with the NoDataFromScan coprocessor. Found: " + + r, r.list()); + + get = new Get(Bytes.toBytes("anotherrow")); + r = table.get(get); + assertNull( + "Got an unexpected number of rows - no data should be returned with the NoDataFromScan coprocessor Found: " + + r, r.list()); + + table.close(); + UTIL.shutdownMiniCluster(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java index 002d611a4394..4edfd385876e 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -21,24 +20,30 @@ package org.apache.hadoop.hbase.coprocessor; import java.io.IOException; -import java.util.List; -import java.util.Map; + +import junit.framework.TestCase; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.Coprocessor; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.SmallTests; import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost; import org.apache.hadoop.hbase.regionserver.wal.WALEdit; -import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.util.Bytes; - -import junit.framework.TestCase; import org.junit.experimental.categories.Category; @Category(SmallTests.class) public class TestRegionObserverStacking extends TestCase { - static final String DIR = "test/build/data/TestRegionObserverStacking/"; + private static HBaseTestingUtility TEST_UTIL + = new HBaseTestingUtility(); + static final Path DIR = TEST_UTIL.getDataTestDir(); public static class ObserverA extends BaseRegionObserver { long id; diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java index 33756579719f..6d2fc33d3236 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,27 +19,28 @@ package org.apache.hadoop.hbase.coprocessor; +import static org.junit.Assert.fail; + import java.io.IOException; +import junit.framework.Assert; + import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.regionserver.HRegionServer; -import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.regionserver.wal.WALEdit; -import org.apache.hadoop.hbase.zookeeper.ZooKeeperListener; -import org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker; -import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.apache.hadoop.hbase.util.Bytes; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; -import static org.junit.Assert.*; - /** * Tests unhandled exceptions thrown by coprocessors running on a regionserver.. * Expected result is that the regionserver will abort with an informative @@ -49,49 +49,18 @@ */ @Category(MediumTests.class) public class TestRegionServerCoprocessorExceptionWithAbort { - static final Log LOG = LogFactory.getLog(TestRegionObserverInterface.class); - - private class zkwAbortable implements Abortable { - @Override - public void abort(String why, Throwable e) { - throw new RuntimeException("Fatal ZK rs tracker error, why=", e); - } - @Override - public boolean isAborted() { - return false; - } - }; - - private class RSTracker extends ZooKeeperNodeTracker { - public boolean regionZKNodeWasDeleted = false; - public String rsNode; - private Thread mainThread; - - public RSTracker(ZooKeeperWatcher zkw, String rsNode, Thread mainThread) { - super(zkw, rsNode, new zkwAbortable()); - this.rsNode = rsNode; - this.mainThread = mainThread; - } - - @Override - public synchronized void nodeDeleted(String path) { - if (path.equals(rsNode)) { - regionZKNodeWasDeleted = true; - mainThread.interrupt(); - } - } - } - private static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); - static final int timeout = 30000; + static final Log LOG = LogFactory.getLog(TestRegionServerCoprocessorExceptionWithAbort.class); + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final String TABLE_NAME = "observed_table"; @BeforeClass public static void setupBeforeClass() throws Exception { // set configure to indicate which cp should be loaded Configuration conf = TEST_UTIL.getConfiguration(); - conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, - BuggyRegionObserver.class.getName()); + conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2); // Let's fail fast. + conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, BuggyRegionObserver.class.getName()); conf.set("hbase.coprocessor.abortonerror", "true"); - TEST_UTIL.startMiniCluster(2); + TEST_UTIL.startMiniCluster(3); } @AfterClass @@ -101,59 +70,32 @@ public static void teardownAfterClass() throws Exception { @Test public void testExceptionFromCoprocessorDuringPut() - throws IOException { + throws IOException, InterruptedException { // When we try to write to TEST_TABLE, the buggy coprocessor will // cause a NullPointerException, which will cause the regionserver (which // hosts the region we attempted to write to) to abort. - byte[] TEST_TABLE = Bytes.toBytes("observed_table"); + byte[] TEST_TABLE = Bytes.toBytes(TABLE_NAME); byte[] TEST_FAMILY = Bytes.toBytes("aaa"); HTable table = TEST_UTIL.createTable(TEST_TABLE, TEST_FAMILY); - TEST_UTIL.waitUntilAllRegionsAssigned( - TEST_UTIL.createMultiRegions(table, TEST_FAMILY)); + TEST_UTIL.createMultiRegions(table, TEST_FAMILY); + TEST_UTIL.waitUntilAllRegionsAssigned(TEST_TABLE); // Note which regionServer will abort (after put is attempted). - final HRegionServer regionServer = - TEST_UTIL.getRSForFirstRegionInTable(TEST_TABLE); - - // add watch so we can know when this regionserver aborted. - ZooKeeperWatcher zkw = new ZooKeeperWatcher(TEST_UTIL.getConfiguration(), - "unittest", new zkwAbortable()); + final HRegionServer regionServer = TEST_UTIL.getRSForFirstRegionInTable(TEST_TABLE); - RSTracker rsTracker = new RSTracker(zkw, - "/hbase/rs/"+regionServer.getServerName(), Thread.currentThread()); - rsTracker.start(); - zkw.registerListener(rsTracker); + final byte[] ROW = Bytes.toBytes("aaa"); + Put put = new Put(ROW); + put.add(TEST_FAMILY, ROW, ROW); - boolean caughtInterruption = false; + Assert.assertFalse("The region server should be available", regionServer.isAborted()); try { - final byte[] ROW = Bytes.toBytes("aaa"); - Put put = new Put(ROW); - put.add(TEST_FAMILY, ROW, ROW); table.put(put); - } catch (IOException e) { - // Depending on exact timing of the threads involved, zkw's interruption - // might be caught here ... - if (e.getCause().getClass().equals(InterruptedException.class)) { - LOG.debug("caught interruption here (during put())."); - caughtInterruption = true; - } else { - fail("put() failed: " + e); - } - } - if (caughtInterruption == false) { - try { - Thread.sleep(timeout); - fail("RegionServer did not abort within 30 seconds."); - } catch (InterruptedException e) { - // .. or it might be caught here. - LOG.debug("caught interruption here (during sleep())."); - caughtInterruption = true; - } + fail("The put should have failed, as the coprocessor is buggy"); + } catch (IOException ignored) { + // Expected. } - assertTrue("Main thread caught interruption.",caughtInterruption); - assertTrue("RegionServer aborted on coprocessor exception, as expected.", - rsTracker.regionZKNodeWasDeleted); + Assert.assertTrue("The region server should have aborted", regionServer.isAborted()); table.close(); } @@ -162,11 +104,9 @@ public static class BuggyRegionObserver extends SimpleRegionObserver { public void prePut(final ObserverContext c, final Put put, final WALEdit edit, final boolean writeToWAL) { - String tableName = - c.getEnvironment().getRegion().getRegionInfo().getTableNameAsString(); - if (tableName.equals("observed_table")) { - Integer i = null; - i = i + 1; + String tableName = c.getEnvironment().getRegion().getRegionInfo().getTableNameAsString(); + if (TABLE_NAME.equals(tableName)) { + throw new NullPointerException("Buggy coprocessor"); } } } diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java index 10f44a6eb3f3..d60ce9c626d2 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,23 +19,26 @@ package org.apache.hadoop.hbase.coprocessor; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + import java.io.IOException; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException; import org.apache.hadoop.hbase.regionserver.HRegionServer; -import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.regionserver.wal.WALEdit; +import org.apache.hadoop.hbase.util.Bytes; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; -import static org.junit.Assert.*; - /** * Tests unhandled exceptions thrown by coprocessors running on regionserver. * Expected result is that the master will remove the buggy coprocessor from @@ -77,7 +79,7 @@ public static void teardownAfterClass() throws Exception { TEST_UTIL.shutdownMiniCluster(); } - @Test(timeout=30000) + @Test(timeout=60000) public void testExceptionFromCoprocessorDuringPut() throws IOException { // Set watches on the zookeeper nodes for all of the regionservers in the @@ -92,8 +94,8 @@ public void testExceptionFromCoprocessorDuringPut() byte[] TEST_FAMILY = Bytes.toBytes("aaa"); HTable table = TEST_UTIL.createTable(TEST_TABLE, TEST_FAMILY); - TEST_UTIL.waitUntilAllRegionsAssigned( - TEST_UTIL.createMultiRegions(table, TEST_FAMILY)); + TEST_UTIL.createMultiRegions(table, TEST_FAMILY); + TEST_UTIL.waitUntilAllRegionsAssigned(TEST_TABLE); // Note which regionServer that should survive the buggy coprocessor's // prePut(). HRegionServer regionServer = diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java index 36dd28938a78..ff2622bd1d74 100644 --- a/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java new file mode 100644 index 000000000000..8b4d51348361 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java @@ -0,0 +1,407 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.coprocessor.example; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.coprocessor.Batch; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; +import org.apache.hadoop.hbase.coprocessor.example.BulkDeleteProtocol.DeleteType; +import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp; +import org.apache.hadoop.hbase.filter.FilterList; +import org.apache.hadoop.hbase.filter.FilterList.Operator; +import org.apache.hadoop.hbase.filter.SingleColumnValueFilter; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(MediumTests.class) +public class TestBulkDeleteProtocol { + private static final byte[] FAMILY1 = Bytes.toBytes("cf1"); + private static final byte[] FAMILY2 = Bytes.toBytes("cf2"); + private static final byte[] QUALIFIER1 = Bytes.toBytes("c1"); + private static final byte[] QUALIFIER2 = Bytes.toBytes("c2"); + private static final byte[] QUALIFIER3 = Bytes.toBytes("c3"); + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + @BeforeClass + public static void setupBeforeClass() throws Exception { + TEST_UTIL.getConfiguration().set(CoprocessorHost.USER_REGION_COPROCESSOR_CONF_KEY, + BulkDeleteEndpoint.class.getName()); + TEST_UTIL.startMiniCluster(2); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + @Test + public void testBulkDeleteEndpoint() throws Throwable { + byte[] tableName = Bytes.toBytes("testBulkDeleteEndpoint"); + HTable ht = createTable(tableName); + List puts = new ArrayList(100); + for (int j = 0; j < 100; j++) { + byte[] rowkey = Bytes.toBytes(j); + puts.add(createPut(rowkey, "v1")); + } + ht.put(puts); + // Deleting all the rows. + long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, new Scan(), 500, DeleteType.ROW, + null); + assertEquals(100, noOfRowsDeleted); + + int rows = 0; + for (Result result : ht.getScanner(new Scan())) { + rows++; + } + assertEquals(0, rows); + } + + @Test + public void testBulkDeleteEndpointWhenRowBatchSizeLessThanRowsToDeleteFromARegion() + throws Throwable { + byte[] tableName = Bytes + .toBytes("testBulkDeleteEndpointWhenRowBatchSizeLessThanRowsToDeleteFromARegion"); + HTable ht = createTable(tableName); + List puts = new ArrayList(100); + for (int j = 0; j < 100; j++) { + byte[] rowkey = Bytes.toBytes(j); + puts.add(createPut(rowkey, "v1")); + } + ht.put(puts); + // Deleting all the rows. + long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, new Scan(), 10, DeleteType.ROW, null); + assertEquals(100, noOfRowsDeleted); + + int rows = 0; + for (Result result : ht.getScanner(new Scan())) { + rows++; + } + assertEquals(0, rows); + } + + private long invokeBulkDeleteProtocol(byte[] tableName, final Scan scan, final int rowBatchSize, + final byte deleteType, final Long timeStamp) throws Throwable { + HTable ht = new HTable(TEST_UTIL.getConfiguration(), tableName); + long noOfDeletedRows = 0L; + Batch.Call callable = + new Batch.Call() { + public BulkDeleteResponse call(BulkDeleteProtocol instance) throws IOException { + return instance.delete(scan, deleteType, timeStamp, rowBatchSize); + } + }; + Map result = ht.coprocessorExec(BulkDeleteProtocol.class, + scan.getStartRow(), scan.getStopRow(), callable); + for (BulkDeleteResponse response : result.values()) { + noOfDeletedRows += response.getRowsDeleted(); + } + return noOfDeletedRows; + } + + @Test + public void testBulkDeleteWithConditionBasedDelete() throws Throwable { + byte[] tableName = Bytes.toBytes("testBulkDeleteWithConditionBasedDelete"); + HTable ht = createTable(tableName); + List puts = new ArrayList(100); + for (int j = 0; j < 100; j++) { + byte[] rowkey = Bytes.toBytes(j); + String value = (j % 10 == 0) ? "v1" : "v2"; + puts.add(createPut(rowkey, value)); + } + ht.put(puts); + Scan scan = new Scan(); + FilterList fl = new FilterList(Operator.MUST_PASS_ALL); + SingleColumnValueFilter scvf = new SingleColumnValueFilter(FAMILY1, QUALIFIER3, + CompareOp.EQUAL, Bytes.toBytes("v1")); + //fl.addFilter(new FirstKeyOnlyFilter()); + fl.addFilter(scvf); + scan.setFilter(fl); + // Deleting all the rows where cf1:c1=v1 + long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, scan, 500, DeleteType.ROW, null); + assertEquals(10, noOfRowsDeleted); + + int rows = 0; + for (Result result : ht.getScanner(new Scan())) { + rows++; + } + assertEquals(90, rows); + } + + @Test + public void testBulkDeleteColumn() throws Throwable { + byte[] tableName = Bytes.toBytes("testBulkDeleteColumn"); + HTable ht = createTable(tableName); + List puts = new ArrayList(100); + for (int j = 0; j < 100; j++) { + byte[] rowkey = Bytes.toBytes(j); + String value = (j % 10 == 0) ? "v1" : "v2"; + puts.add(createPut(rowkey, value)); + } + ht.put(puts); + Scan scan = new Scan (); + scan.addColumn(FAMILY1, QUALIFIER2); + // Delete the column cf1:col2 + long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, scan, 500, DeleteType.COLUMN, null); + assertEquals(100, noOfRowsDeleted); + + int rows = 0; + for (Result result : ht.getScanner(new Scan())) { + assertEquals(2, result.getFamilyMap(FAMILY1).size()); + assertTrue(result.getColumn(FAMILY1, QUALIFIER2).isEmpty()); + assertEquals(1, result.getColumn(FAMILY1, QUALIFIER1).size()); + assertEquals(1, result.getColumn(FAMILY1, QUALIFIER3).size()); + rows++; + } + assertEquals(100, rows); + } + + @Test + public void testBulkDeleteFamily() throws Throwable { + byte[] tableName = Bytes.toBytes("testBulkDeleteFamily"); + HTableDescriptor htd = new HTableDescriptor(tableName); + htd.addFamily(new HColumnDescriptor(FAMILY1)); + htd.addFamily(new HColumnDescriptor(FAMILY2)); + TEST_UTIL.getHBaseAdmin().createTable(htd, Bytes.toBytes(0), Bytes.toBytes(120), 5); + HTable ht = new HTable(TEST_UTIL.getConfiguration(), tableName); + List puts = new ArrayList(100); + for (int j = 0; j < 100; j++) { + Put put = new Put(Bytes.toBytes(j)); + put.add(FAMILY1, QUALIFIER1, "v1".getBytes()); + put.add(FAMILY2, QUALIFIER2, "v2".getBytes()); + puts.add(put); + } + ht.put(puts); + Scan scan = new Scan (); + scan.addFamily(FAMILY1); + // Delete the column family cf1 + long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, scan, 500, DeleteType.FAMILY, null); + assertEquals(100, noOfRowsDeleted); + int rows = 0; + for (Result result : ht.getScanner(new Scan())) { + assertTrue(result.getFamilyMap(FAMILY1).isEmpty()); + assertEquals(1, result.getColumn(FAMILY2, QUALIFIER2).size()); + rows++; + } + assertEquals(100, rows); + } + + @Test + public void testBulkDeleteColumnVersion() throws Throwable { + byte[] tableName = Bytes.toBytes("testBulkDeleteColumnVersion"); + HTable ht = createTable(tableName); + List puts = new ArrayList(100); + for (int j = 0; j < 100; j++) { + Put put = new Put(Bytes.toBytes(j)); + byte[] value = "v1".getBytes(); + put.add(FAMILY1, QUALIFIER1, 1234L, value); + put.add(FAMILY1, QUALIFIER2, 1234L, value); + put.add(FAMILY1, QUALIFIER3, 1234L, value); + // Latest version values + value = "v2".getBytes(); + put.add(FAMILY1, QUALIFIER1, value); + put.add(FAMILY1, QUALIFIER2, value); + put.add(FAMILY1, QUALIFIER3, value); + put.add(FAMILY1, null, value); + puts.add(put); + } + ht.put(puts); + Scan scan = new Scan (); + scan.addFamily(FAMILY1); + // Delete the latest version values of all the columns in family cf1. + long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, scan, 500, DeleteType.VERSION, + HConstants.LATEST_TIMESTAMP); + assertEquals(100, noOfRowsDeleted); + int rows = 0; + scan = new Scan (); + scan.setMaxVersions(); + for (Result result : ht.getScanner(scan)) { + assertEquals(3, result.getFamilyMap(FAMILY1).size()); + List column = result.getColumn(FAMILY1, QUALIFIER1); + assertEquals(1, column.size()); + assertTrue(Bytes.equals("v1".getBytes(), column.get(0).getValue())); + + column = result.getColumn(FAMILY1, QUALIFIER2); + assertEquals(1, column.size()); + assertTrue(Bytes.equals("v1".getBytes(), column.get(0).getValue())); + + column = result.getColumn(FAMILY1, QUALIFIER3); + assertEquals(1, column.size()); + assertTrue(Bytes.equals("v1".getBytes(), column.get(0).getValue())); + rows++; + } + assertEquals(100, rows); + } + + @Test + public void testBulkDeleteColumnVersionBasedOnTS() throws Throwable { + byte[] tableName = Bytes.toBytes("testBulkDeleteColumnVersionBasedOnTS"); + HTable ht = createTable(tableName); + List puts = new ArrayList(100); + for (int j = 0; j < 100; j++) { + Put put = new Put(Bytes.toBytes(j)); + // TS = 1000L + byte[] value = "v1".getBytes(); + put.add(FAMILY1, QUALIFIER1, 1000L, value); + put.add(FAMILY1, QUALIFIER2, 1000L, value); + put.add(FAMILY1, QUALIFIER3, 1000L, value); + // TS = 1234L + value = "v2".getBytes(); + put.add(FAMILY1, QUALIFIER1, 1234L, value); + put.add(FAMILY1, QUALIFIER2, 1234L, value); + put.add(FAMILY1, QUALIFIER3, 1234L, value); + // Latest version values + value = "v3".getBytes(); + put.add(FAMILY1, QUALIFIER1, value); + put.add(FAMILY1, QUALIFIER2, value); + put.add(FAMILY1, QUALIFIER3, value); + puts.add(put); + } + ht.put(puts); + Scan scan = new Scan (); + scan.addColumn(FAMILY1, QUALIFIER3); + // Delete the column cf1:c3's one version at TS=1234 + long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, scan, 500, DeleteType.VERSION, 1234L); + assertEquals(100, noOfRowsDeleted); + int rows = 0; + scan = new Scan (); + scan.setMaxVersions(); + for (Result result : ht.getScanner(scan)) { + assertEquals(3, result.getFamilyMap(FAMILY1).size()); + assertEquals(3, result.getColumn(FAMILY1, QUALIFIER1).size()); + assertEquals(3, result.getColumn(FAMILY1, QUALIFIER2).size()); + List column = result.getColumn(FAMILY1, QUALIFIER3); + assertEquals(2, column.size()); + assertTrue(Bytes.equals("v3".getBytes(), column.get(0).getValue())); + assertTrue(Bytes.equals("v1".getBytes(), column.get(1).getValue())); + rows++; + } + assertEquals(100, rows); + } + + @Test + public void testBulkDeleteWithNumberOfVersions() throws Throwable { + byte[] tableName = Bytes.toBytes("testBulkDeleteWithNumberOfVersions"); + HTable ht = createTable(tableName); + List puts = new ArrayList(100); + for (int j = 0; j < 100; j++) { + Put put = new Put(Bytes.toBytes(j)); + // TS = 1000L + byte[] value = "v1".getBytes(); + put.add(FAMILY1, QUALIFIER1, 1000L, value); + put.add(FAMILY1, QUALIFIER2, 1000L, value); + put.add(FAMILY1, QUALIFIER3, 1000L, value); + // TS = 1234L + value = "v2".getBytes(); + put.add(FAMILY1, QUALIFIER1, 1234L, value); + put.add(FAMILY1, QUALIFIER2, 1234L, value); + put.add(FAMILY1, QUALIFIER3, 1234L, value); + // TS = 2000L + value = "v3".getBytes(); + put.add(FAMILY1, QUALIFIER1, 2000L, value); + put.add(FAMILY1, QUALIFIER2, 2000L, value); + put.add(FAMILY1, QUALIFIER3, 2000L, value); + // Latest version values + value = "v4".getBytes(); + put.add(FAMILY1, QUALIFIER1, value); + put.add(FAMILY1, QUALIFIER2, value); + put.add(FAMILY1, QUALIFIER3, value); + puts.add(put); + } + ht.put(puts); + + // Delete all the versions of columns cf1:c1 and cf1:c2 falling with the time range + // [1000,2000) + final Scan scan = new Scan(); + scan.addColumn(FAMILY1, QUALIFIER1); + scan.addColumn(FAMILY1, QUALIFIER2); + scan.setTimeRange(1000L, 2000L); + scan.setMaxVersions(); + + long noOfDeletedRows = 0L; + long noOfVersionsDeleted = 0L; + Batch.Call callable = + new Batch.Call() { + public BulkDeleteResponse call(BulkDeleteProtocol instance) throws IOException { + return instance.delete(scan, DeleteType.VERSION, null, 500); + } + }; + Map result = ht.coprocessorExec(BulkDeleteProtocol.class, + scan.getStartRow(), scan.getStopRow(), callable); + for (BulkDeleteResponse response : result.values()) { + noOfDeletedRows += response.getRowsDeleted(); + noOfVersionsDeleted += response.getVersionsDeleted(); + } + assertEquals(100, noOfDeletedRows); + assertEquals(400, noOfVersionsDeleted); + + int rows = 0; + Scan scan1 = new Scan (); + scan1.setMaxVersions(); + for (Result res : ht.getScanner(scan1)) { + assertEquals(3, res.getFamilyMap(FAMILY1).size()); + List column = res.getColumn(FAMILY1, QUALIFIER1); + assertEquals(2, column.size()); + assertTrue(Bytes.equals("v4".getBytes(), column.get(0).getValue())); + assertTrue(Bytes.equals("v3".getBytes(), column.get(1).getValue())); + column = res.getColumn(FAMILY1, QUALIFIER2); + assertEquals(2, column.size()); + assertTrue(Bytes.equals("v4".getBytes(), column.get(0).getValue())); + assertTrue(Bytes.equals("v3".getBytes(), column.get(1).getValue())); + assertEquals(4, res.getColumn(FAMILY1, QUALIFIER3).size()); + rows++; + } + assertEquals(100, rows); + } + + private HTable createTable(byte[] tableName) throws IOException { + HTableDescriptor htd = new HTableDescriptor(tableName); + HColumnDescriptor hcd = new HColumnDescriptor(FAMILY1); + hcd.setMaxVersions(10);// Just setting 10 as I am not testing with more than 10 versions here + htd.addFamily(hcd); + TEST_UTIL.getHBaseAdmin().createTable(htd, Bytes.toBytes(0), Bytes.toBytes(120), 5); + HTable ht = new HTable(TEST_UTIL.getConfiguration(), tableName); + return ht; + } + + private Put createPut(byte[] rowkey, String value) throws IOException { + Put put = new Put(rowkey); + put.add(FAMILY1, QUALIFIER1, value.getBytes()); + put.add(FAMILY1, QUALIFIER2, value.getBytes()); + put.add(FAMILY1, QUALIFIER3, value.getBytes()); + return put; + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestZooKeeperScanPolicyObserver.java b/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestZooKeeperScanPolicyObserver.java new file mode 100644 index 000000000000..ee05cb7da111 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestZooKeeperScanPolicyObserver.java @@ -0,0 +1,130 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.coprocessor.example; + +import static org.junit.Assert.assertEquals; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HConnectionManager; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; +import org.apache.hadoop.hbase.zookeeper.ZKUtil; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.apache.zookeeper.ZooKeeper; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(MediumTests.class) +public class TestZooKeeperScanPolicyObserver { + private static final Log LOG = LogFactory.getLog(TestZooKeeperScanPolicyObserver.class); + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final byte[] F = Bytes.toBytes("fam"); + private static final byte[] Q = Bytes.toBytes("qual"); + private static final byte[] R = Bytes.toBytes("row"); + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + // Test we can first start the ZK cluster by itself + Configuration conf = TEST_UTIL.getConfiguration(); + conf.setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, + ZooKeeperScanPolicyObserver.class.getName()); + TEST_UTIL.startMiniZKCluster(); + TEST_UTIL.startMiniCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + @Test + public void testScanPolicyObserver() throws Exception { + byte[] tableName = Bytes.toBytes("testScanPolicyObserver"); + HTableDescriptor desc = new HTableDescriptor(tableName); + HColumnDescriptor hcd = new HColumnDescriptor(F) + .setMaxVersions(10) + .setTimeToLive(1); + desc.addFamily(hcd); + TEST_UTIL.getHBaseAdmin().createTable(desc); + HTable t = new HTable(new Configuration(TEST_UTIL.getConfiguration()), tableName); + long now = EnvironmentEdgeManager.currentTimeMillis(); + + ZooKeeperWatcher zkw = HConnectionManager.getConnection(TEST_UTIL.getConfiguration()) + .getZooKeeperWatcher(); + ZooKeeper zk = zkw.getRecoverableZooKeeper().getZooKeeper(); + ZKUtil.createWithParents(zkw, ZooKeeperScanPolicyObserver.node); + // let's say test last backup was 1h ago + // using plain ZK here, because RecoverableZooKeeper add extra encoding to the data + zk.setData(ZooKeeperScanPolicyObserver.node, Bytes.toBytes(now - 3600*1000), -1); + + LOG.debug("Set time: "+Bytes.toLong(Bytes.toBytes(now - 3600*1000))); + + // sleep for 1s to give the ZK change a chance to reach the watcher in the observer. + // TODO: Better to wait for the data to be propagated + Thread.sleep(1000); + + long ts = now - 2000; + Put p = new Put(R); + p.add(F, Q, ts, Q); + t.put(p); + p = new Put(R); + p.add(F, Q, ts+1, Q); + t.put(p); + + // these two should be expired but for the override + // (their ts was 2s in the past) + Get g = new Get(R); + g.setMaxVersions(10); + Result r = t.get(g); + // still there? + assertEquals(2, r.size()); + + TEST_UTIL.flush(tableName); + TEST_UTIL.compact(tableName, true); + + g = new Get(R); + g.setMaxVersions(10); + r = t.get(g); + // still there? + assertEquals(2, r.size()); + zk.setData(ZooKeeperScanPolicyObserver.node, Bytes.toBytes(now), -1); + LOG.debug("Set time: "+now); + + TEST_UTIL.compact(tableName, true); + + g = new Get(R); + g.setMaxVersions(10); + r = t.get(g); + // should be gone now + assertEquals(0, r.size()); + t.close(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionDispatcher.java b/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionDispatcher.java new file mode 100644 index 000000000000..e5c47b13996a --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionDispatcher.java @@ -0,0 +1,123 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.errorhandling; + +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +/** + * Test that we propagate errors through an dispatcher exactly once via different failure + * injection mechanisms. + */ +@Category(SmallTests.class) +public class TestForeignExceptionDispatcher { + private static final Log LOG = LogFactory.getLog(TestForeignExceptionDispatcher.class); + + /** + * Exception thrown from the test + */ + final ForeignException EXTEXN = new ForeignException("FORTEST", new IllegalArgumentException("FORTEST")); + final ForeignException EXTEXN2 = new ForeignException("FORTEST2", new IllegalArgumentException("FORTEST2")); + + /** + * Tests that a dispatcher only dispatches only the first exception, and does not propagate + * subsequent exceptions. + */ + @Test + public void testErrorPropagation() { + ForeignExceptionListener listener1 = Mockito.mock(ForeignExceptionListener.class); + ForeignExceptionListener listener2 = Mockito.mock(ForeignExceptionListener.class); + ForeignExceptionDispatcher dispatcher = new ForeignExceptionDispatcher(); + + // add the listeners + dispatcher.addListener(listener1); + dispatcher.addListener(listener2); + + // create an artificial error + dispatcher.receive(EXTEXN); + + // make sure the listeners got the error + Mockito.verify(listener1, Mockito.times(1)).receive(EXTEXN); + Mockito.verify(listener2, Mockito.times(1)).receive(EXTEXN); + + // make sure that we get an exception + try { + dispatcher.rethrowException(); + fail("Monitor should have thrown an exception after getting error."); + } catch (ForeignException ex) { + assertTrue("Got an unexpected exception:" + ex, ex.getCause() == EXTEXN.getCause()); + LOG.debug("Got the testing exception!"); + } + + // push another error, which should be not be passed to listeners + dispatcher.receive(EXTEXN2); + Mockito.verify(listener1, Mockito.never()).receive(EXTEXN2); + Mockito.verify(listener2, Mockito.never()).receive(EXTEXN2); + } + + @Test + public void testSingleDispatcherWithTimer() { + ForeignExceptionListener listener1 = Mockito.mock(ForeignExceptionListener.class); + ForeignExceptionListener listener2 = Mockito.mock(ForeignExceptionListener.class); + + ForeignExceptionDispatcher monitor = new ForeignExceptionDispatcher(); + + // add the listeners + monitor.addListener(listener1); + monitor.addListener(listener2); + + TimeoutExceptionInjector timer = new TimeoutExceptionInjector(monitor, 1000); + timer.start(); + timer.trigger(); + + assertTrue("Monitor didn't get timeout", monitor.hasException()); + + // verify that that we propagated the error + Mockito.verify(listener1).receive(Mockito.any(ForeignException.class)); + Mockito.verify(listener2).receive(Mockito.any(ForeignException.class)); + } + + /** + * Test that the dispatcher can receive an error via the timer mechanism. + */ + @Test + public void testAttemptTimer() { + ForeignExceptionListener listener1 = Mockito.mock(ForeignExceptionListener.class); + ForeignExceptionListener listener2 = Mockito.mock(ForeignExceptionListener.class); + ForeignExceptionDispatcher orchestrator = new ForeignExceptionDispatcher(); + + // add the listeners + orchestrator.addListener(listener1); + orchestrator.addListener(listener2); + + // now create a timer and check for that error + TimeoutExceptionInjector timer = new TimeoutExceptionInjector(orchestrator, 1000); + timer.start(); + timer.trigger(); + // make sure that we got the timer error + Mockito.verify(listener1, Mockito.times(1)).receive(Mockito.any(ForeignException.class)); + Mockito.verify(listener2, Mockito.times(1)).receive(Mockito.any(ForeignException.class)); + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionSerialization.java b/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionSerialization.java new file mode 100644 index 000000000000..11363fed437f --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionSerialization.java @@ -0,0 +1,82 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.errorhandling; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import com.google.protobuf.InvalidProtocolBufferException; + +/** + * Test that we correctly serialize exceptions from a remote source + */ +@Category(SmallTests.class) +public class TestForeignExceptionSerialization { + private static final String srcName = "someNode"; + + /** + * Verify that we get back similar stack trace information before an after serialization. + * @throws InvalidProtocolBufferException + */ + @Test + public void testSimpleException() throws InvalidProtocolBufferException { + String data = "some bytes"; + ForeignException in = new ForeignException("SRC", new IllegalArgumentException(data)); + // check that we get the data back out + ForeignException e = ForeignException.deserialize(ForeignException.serialize(srcName, in)); + assertNotNull(e); + + // now check that we get the right stack trace + StackTraceElement elem = new StackTraceElement(this.getClass().toString(), "method", "file", 1); + in.setStackTrace(new StackTraceElement[] { elem }); + e = ForeignException.deserialize(ForeignException.serialize(srcName, in)); + + assertNotNull(e); + assertEquals("Stack trace got corrupted", elem, e.getCause().getStackTrace()[0]); + assertEquals("Got an unexpectedly long stack trace", 1, e.getCause().getStackTrace().length); + } + + /** + * Compare that a generic exception's stack trace has the same stack trace elements after + * serialization and deserialization + * @throws InvalidProtocolBufferException + */ + @Test + public void testRemoteFromLocal() throws InvalidProtocolBufferException { + String errorMsg = "some message"; + Exception generic = new Exception(errorMsg); + generic.printStackTrace(); + assertTrue(generic.getMessage().contains(errorMsg)); + + ForeignException e = ForeignException.deserialize(ForeignException.serialize(srcName, generic)); + assertArrayEquals("Local stack trace got corrupted", generic.getStackTrace(), e.getCause().getStackTrace()); + + e.printStackTrace(); // should have ForeignException and source node in it. + assertTrue(e.getCause().getCause() == null); + + // verify that original error message is present in Foreign exception message + assertTrue(e.getCause().getMessage().contains(errorMsg)); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/errorhandling/TestTimeoutExceptionInjector.java b/src/test/java/org/apache/hadoop/hbase/errorhandling/TestTimeoutExceptionInjector.java new file mode 100644 index 000000000000..641dbe042014 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/errorhandling/TestTimeoutExceptionInjector.java @@ -0,0 +1,103 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.errorhandling; + +import static org.junit.Assert.fail; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +/** + * Test the {@link TimeoutExceptionInjector} to ensure we fulfill contracts + */ +@Category(SmallTests.class) +public class TestTimeoutExceptionInjector { + + private static final Log LOG = LogFactory.getLog(TestTimeoutExceptionInjector.class); + + /** + * Test that a manually triggered timer fires an exception. + */ + @Test(timeout = 60000) + public void testTimerTrigger() { + final long time = 10000000; // pick a value that is very far in the future + ForeignExceptionListener listener = Mockito.mock(ForeignExceptionListener.class); + TimeoutExceptionInjector timer = new TimeoutExceptionInjector(listener, time); + timer.start(); + timer.trigger(); + Mockito.verify(listener, Mockito.times(1)).receive(Mockito.any(ForeignException.class)); + } + + /** + * Test that a manually triggered exception with data fires with the data in receiveError. + */ + @Test + public void testTimerPassesOnErrorInfo() { + final long time = 1000000; + ForeignExceptionListener listener = Mockito.mock(ForeignExceptionListener.class); + TimeoutExceptionInjector timer = new TimeoutExceptionInjector(listener, time); + timer.start(); + timer.trigger(); + Mockito.verify(listener).receive(Mockito.any(ForeignException.class)); + } + + /** + * Demonstrate TimeoutExceptionInjector semantics -- completion means no more exceptions passed to + * error listener. + */ + @Test(timeout = 60000) + public void testStartAfterComplete() throws InterruptedException { + final long time = 10; + ForeignExceptionListener listener = Mockito.mock(ForeignExceptionListener.class); + TimeoutExceptionInjector timer = new TimeoutExceptionInjector(listener, time); + timer.complete(); + try { + timer.start(); + fail("Timer should fail to start after complete."); + } catch (IllegalStateException e) { + LOG.debug("Correctly failed timer: " + e.getMessage()); + } + Thread.sleep(time + 1); + Mockito.verifyZeroInteractions(listener); + } + + /** + * Demonstrate TimeoutExceptionInjector semantics -- triggering fires exception and completes + * the timer. + */ + @Test(timeout = 60000) + public void testStartAfterTrigger() throws InterruptedException { + final long time = 10; + ForeignExceptionListener listener = Mockito.mock(ForeignExceptionListener.class); + TimeoutExceptionInjector timer = new TimeoutExceptionInjector(listener, time); + timer.trigger(); + try { + timer.start(); + fail("Timer should fail to start after complete."); + } catch (IllegalStateException e) { + LOG.debug("Correctly failed timer: " + e.getMessage()); + } + Thread.sleep(time * 2); + Mockito.verify(listener, Mockito.times(1)).receive(Mockito.any(ForeignException.class)); + Mockito.verifyNoMoreInteractions(listener); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/executor/TestExecutorService.java b/src/test/java/org/apache/hadoop/hbase/executor/TestExecutorService.java index 805421f7a4be..29201e6999bc 100644 --- a/src/test/java/org/apache/hadoop/hbase/executor/TestExecutorService.java +++ b/src/test/java/org/apache/hadoop/hbase/executor/TestExecutorService.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/filter/FilterAllFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/FilterAllFilter.java new file mode 100644 index 000000000000..e883bd0849ea --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/filter/FilterAllFilter.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.filter; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; + +import org.apache.hadoop.hbase.KeyValue; + +public class FilterAllFilter extends FilterBase { + + public FilterAllFilter() { + } + + @Override + public ReturnCode filterKeyValue(KeyValue ignored) { + return ReturnCode.SKIP; + } + + @Override + public boolean filterRow() { + return true; + } + + @Override + public void readFields(DataInput in) throws IOException { + } + + @Override + public void write(DataOutput out) throws IOException { + } + +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestBitComparator.java b/src/test/java/org/apache/hadoop/hbase/filter/TestBitComparator.java index 9007f074aa3e..aff6cf2f5e81 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestBitComparator.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestBitComparator.java @@ -1,18 +1,20 @@ -/** - * Copyright 2010 The Apache Software Foundation - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * under the License. +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. */ package org.apache.hadoop.hbase.filter; @@ -32,6 +34,11 @@ public class TestBitComparator extends TestCase { private static byte[] data1 = new byte[]{15, 0, 0, 0, 0, 0}; private static byte[] data2 = new byte[]{0, 0, 0, 0, 0, 15}; private static byte[] data3 = new byte[]{15, 15, 15, 15, 15}; + + // data for testing compareTo method with offset and length parameters + private static byte[] data1_2 = new byte[]{15, 15, 0, 0, 0, 0, 0, 15}; + private static byte[] data2_2 = new byte[]{15, 0, 0, 0, 0, 0, 15, 15}; + private final int Equal = 0; private final int NotEqual = 1; @@ -65,6 +72,26 @@ private void testOperation(byte[] data, byte[] comparatorBytes, BitComparator.Bi assertEquals(comparator.compareTo(data), expected); } + public void testANDOperationWithOffset() { + testOperationWithOffset(data1_2, ones, BitComparator.BitwiseOp.AND, Equal); + testOperationWithOffset(data1_2, data0, BitComparator.BitwiseOp.AND, NotEqual); + testOperationWithOffset(data2_2, data1, BitComparator.BitwiseOp.AND, NotEqual); + } + + public void testOROperationWithOffset() { + testOperationWithOffset(data1_2, zeros, BitComparator.BitwiseOp.OR, Equal); + testOperationWithOffset(data2_2, data1, BitComparator.BitwiseOp.OR, Equal); + } + + public void testXOROperationWithOffset() { + testOperationWithOffset(data2_2, data1, BitComparator.BitwiseOp.XOR, Equal); + } + + private void testOperationWithOffset(byte[] data, byte[] comparatorBytes, BitComparator.BitwiseOp operator, int expected) { + BitComparator comparator = new BitComparator(comparatorBytes, operator); + assertEquals(comparator.compareTo(data, 1, comparatorBytes.length), expected); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestColumnCountGetFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestColumnCountGetFilter.java new file mode 100644 index 000000000000..f725e618f600 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestColumnCountGetFilter.java @@ -0,0 +1,149 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.filter; + +import static org.junit.Assert.assertEquals; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.KeyValueTestUtil; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.InternalScanner; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(SmallTests.class) +public class TestColumnCountGetFilter { + + private final static HBaseTestingUtility TEST_UTIL = new + HBaseTestingUtility(); + + @Test + public void testColumnCountGetFilter() throws IOException { + String family = "Family"; + HTableDescriptor htd = new HTableDescriptor("testColumnCountGetFilter"); + htd.addFamily(new HColumnDescriptor(family)); + HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); + HRegion region = HRegion.createHRegion(info, TEST_UTIL. + getDataTestDir(), TEST_UTIL.getConfiguration(), htd); + try { + String valueString = "ValueString"; + String row = "row-1"; + List columns = generateRandomWords(10000, "column"); + Put p = new Put(Bytes.toBytes(row)); + p.setWriteToWAL(false); + for (String column : columns) { + KeyValue kv = KeyValueTestUtil.create(row, family, column, 0, valueString); + p.add(kv); + } + region.put(p); + + Get get = new Get(row.getBytes()); + Filter filter = new ColumnCountGetFilter(100); + get.setFilter(filter); + Scan scan = new Scan(get); + InternalScanner scanner = region.getScanner(scan); + List results = new ArrayList(); + scanner.next(results); + assertEquals(100, results.size()); + } finally { + region.close(); + region.getLog().closeAndDelete(); + } + + region.close(); + region.getLog().closeAndDelete(); + } + + @Test + public void testColumnCountGetFilterWithFilterList() throws IOException { + String family = "Family"; + HTableDescriptor htd = new HTableDescriptor("testColumnCountGetFilter"); + htd.addFamily(new HColumnDescriptor(family)); + HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); + HRegion region = HRegion.createHRegion(info, TEST_UTIL. + getDataTestDir(), TEST_UTIL.getConfiguration(), htd); + try { + String valueString = "ValueString"; + String row = "row-1"; + List columns = generateRandomWords(10000, "column"); + Put p = new Put(Bytes.toBytes(row)); + p.setWriteToWAL(false); + for (String column : columns) { + KeyValue kv = KeyValueTestUtil.create(row, family, column, 0, valueString); + p.add(kv); + } + region.put(p); + + Get get = new Get(row.getBytes()); + FilterList filterLst = new FilterList (); + filterLst.addFilter( new ColumnCountGetFilter(100)); + get.setFilter(filterLst); + Scan scan = new Scan(get); + InternalScanner scanner = region.getScanner(scan); + List results = new ArrayList(); + scanner.next(results); + assertEquals(100, results.size()); + } finally { + region.close(); + region.getLog().closeAndDelete(); + } + + region.close(); + region.getLog().closeAndDelete(); + } + + List generateRandomWords(int numberOfWords, String suffix) { + Set wordSet = new HashSet(); + for (int i = 0; i < numberOfWords; i++) { + int lengthOfWords = (int) (Math.random()*2) + 1; + char[] wordChar = new char[lengthOfWords]; + for (int j = 0; j < wordChar.length; j++) { + wordChar[j] = (char) (Math.random() * 26 + 97); + } + String word; + if (suffix == null) { + word = new String(wordChar); + } else { + word = new String(wordChar) + suffix; + } + wordSet.add(word); + } + List wordList = new ArrayList(wordSet); + return wordList; + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} + diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java index e26ce7e383ae..0d4208567bda 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -81,7 +80,7 @@ private Filter serializationTest(Filter filter) throws Exception { private void basicFilterTests(ColumnPaginationFilter filter) throws Exception { KeyValue kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_1); - assertTrue("basicFilter1", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE); + assertTrue("basicFilter1", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE_AND_NEXT_COL); } /** diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java index 333766c4b052..9f899143b770 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java @@ -50,52 +50,56 @@ public void testColumnPrefixFilter() throws IOException { HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); HRegion region = HRegion.createHRegion(info, TEST_UTIL. getDataTestDir(), TEST_UTIL.getConfiguration(), htd); - - List rows = generateRandomWords(100, "row"); - List columns = generateRandomWords(10000, "column"); - long maxTimestamp = 2; - - List kvList = new ArrayList(); - - Map> prefixMap = new HashMap>(); - - prefixMap.put("p", new ArrayList()); - prefixMap.put("s", new ArrayList()); - - String valueString = "ValueString"; - - for (String row: rows) { - Put p = new Put(Bytes.toBytes(row)); - p.setWriteToWAL(false); - for (String column: columns) { - for (long timestamp = 1; timestamp <= maxTimestamp; timestamp++) { - KeyValue kv = KeyValueTestUtil.create(row, family, column, timestamp, - valueString); - p.add(kv); - kvList.add(kv); - for (String s: prefixMap.keySet()) { - if (column.startsWith(s)) { - prefixMap.get(s).add(kv); + try { + List rows = generateRandomWords(100, "row"); + List columns = generateRandomWords(10000, "column"); + long maxTimestamp = 2; + + List kvList = new ArrayList(); + + Map> prefixMap = new HashMap>(); + + prefixMap.put("p", new ArrayList()); + prefixMap.put("s", new ArrayList()); + + String valueString = "ValueString"; + + for (String row: rows) { + Put p = new Put(Bytes.toBytes(row)); + p.setWriteToWAL(false); + for (String column: columns) { + for (long timestamp = 1; timestamp <= maxTimestamp; timestamp++) { + KeyValue kv = KeyValueTestUtil.create(row, family, column, timestamp, + valueString); + p.add(kv); + kvList.add(kv); + for (String s: prefixMap.keySet()) { + if (column.startsWith(s)) { + prefixMap.get(s).add(kv); + } } } } + region.put(p); } - region.put(p); - } - ColumnPrefixFilter filter; - Scan scan = new Scan(); - scan.setMaxVersions(); - for (String s: prefixMap.keySet()) { - filter = new ColumnPrefixFilter(Bytes.toBytes(s)); + ColumnPrefixFilter filter; + Scan scan = new Scan(); + scan.setMaxVersions(); + for (String s: prefixMap.keySet()) { + filter = new ColumnPrefixFilter(Bytes.toBytes(s)); - scan.setFilter(filter); + scan.setFilter(filter); - InternalScanner scanner = region.getScanner(scan); - List results = new ArrayList(); - while(scanner.next(results)); - assertEquals(prefixMap.get(s).size(), results.size()); + InternalScanner scanner = region.getScanner(scan); + List results = new ArrayList(); + while(scanner.next(results)); + assertEquals(prefixMap.get(s).size(), results.size()); + } + } finally { + region.close(); + region.getLog().closeAndDelete(); } region.close(); @@ -110,55 +114,59 @@ public void testColumnPrefixFilterWithFilterList() throws IOException { HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); HRegion region = HRegion.createHRegion(info, TEST_UTIL. getDataTestDir(), TEST_UTIL.getConfiguration(), htd); - - List rows = generateRandomWords(100, "row"); - List columns = generateRandomWords(10000, "column"); - long maxTimestamp = 2; - - List kvList = new ArrayList(); - - Map> prefixMap = new HashMap>(); - - prefixMap.put("p", new ArrayList()); - prefixMap.put("s", new ArrayList()); - - String valueString = "ValueString"; - - for (String row: rows) { - Put p = new Put(Bytes.toBytes(row)); - p.setWriteToWAL(false); - for (String column: columns) { - for (long timestamp = 1; timestamp <= maxTimestamp; timestamp++) { - KeyValue kv = KeyValueTestUtil.create(row, family, column, timestamp, - valueString); - p.add(kv); - kvList.add(kv); - for (String s: prefixMap.keySet()) { - if (column.startsWith(s)) { - prefixMap.get(s).add(kv); + try { + List rows = generateRandomWords(100, "row"); + List columns = generateRandomWords(10000, "column"); + long maxTimestamp = 2; + + List kvList = new ArrayList(); + + Map> prefixMap = new HashMap>(); + + prefixMap.put("p", new ArrayList()); + prefixMap.put("s", new ArrayList()); + + String valueString = "ValueString"; + + for (String row: rows) { + Put p = new Put(Bytes.toBytes(row)); + p.setWriteToWAL(false); + for (String column: columns) { + for (long timestamp = 1; timestamp <= maxTimestamp; timestamp++) { + KeyValue kv = KeyValueTestUtil.create(row, family, column, timestamp, + valueString); + p.add(kv); + kvList.add(kv); + for (String s: prefixMap.keySet()) { + if (column.startsWith(s)) { + prefixMap.get(s).add(kv); + } } } } + region.put(p); } - region.put(p); - } - ColumnPrefixFilter filter; - Scan scan = new Scan(); - scan.setMaxVersions(); - for (String s: prefixMap.keySet()) { - filter = new ColumnPrefixFilter(Bytes.toBytes(s)); - - //this is how this test differs from the one above - FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL); - filterList.addFilter(filter); - scan.setFilter(filterList); - - InternalScanner scanner = region.getScanner(scan); - List results = new ArrayList(); - while(scanner.next(results)); - assertEquals(prefixMap.get(s).size(), results.size()); + ColumnPrefixFilter filter; + Scan scan = new Scan(); + scan.setMaxVersions(); + for (String s: prefixMap.keySet()) { + filter = new ColumnPrefixFilter(Bytes.toBytes(s)); + + //this is how this test differs from the one above + FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL); + filterList.addFilter(filter); + scan.setFilter(filterList); + + InternalScanner scanner = region.getScanner(scan); + List results = new ArrayList(); + while(scanner.next(results)); + assertEquals(prefixMap.get(s).size(), results.size()); + } + } finally { + region.close(); + region.getLog().closeAndDelete(); } region.close(); diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java index 8e13a5b34ef0..2d205bfbd1db 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -243,6 +242,31 @@ public void testFilterDropping() throws Exception { assertEquals("check cell retention", 2, accepted.size()); } + /** + * Test for HBASE-8794. Avoid NullPointerException in DependentColumnFilter.toString(). + */ + public void testToStringWithNullComparator() { + // Test constructor that implicitly sets a null comparator + Filter filter = new DependentColumnFilter(FAMILIES[0], QUALIFIER); + assertNotNull(filter.toString()); + assertTrue("check string contains 'null' as compatator is null", + filter.toString().contains("null")); + + // Test constructor with explicit null comparator + filter = new DependentColumnFilter(FAMILIES[0], QUALIFIER, true, CompareOp.EQUAL, null); + assertNotNull(filter.toString()); + assertTrue("check string contains 'null' as compatator is null", + filter.toString().contains("null")); + } + + public void testToStringWithNonNullComparator() { + Filter filter = + new DependentColumnFilter(FAMILIES[0], QUALIFIER, true, CompareOp.EQUAL, + new BinaryComparator(MATCH_VAL)); + assertNotNull(filter.toString()); + assertTrue("check string contains comparator value", filter.toString().contains("match")); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java index 0e19d4d1b4aa..c95bc0aced0b 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,33 +19,39 @@ package org.apache.hadoop.hbase.filter; +import java.io.DataInput; +import java.io.DataOutput; import java.io.IOException; import java.util.ArrayList; import java.util.Arrays; import java.util.List; -import java.util.concurrent.atomic.AtomicInteger; import junit.framework.Assert; + import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.client.Delete; +import org.apache.hadoop.hbase.client.Durability; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp; import org.apache.hadoop.hbase.filter.FilterList.Operator; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.InternalScanner; +import org.apache.hadoop.hbase.regionserver.RegionScanner; import org.apache.hadoop.hbase.regionserver.wal.HLog; import org.apache.hadoop.hbase.util.Bytes; import org.junit.experimental.categories.Category; +import com.google.common.base.Throwables; + /** * Test filters at the HRegion doorstep. */ @Category(SmallTests.class) public class TestFilter extends HBaseTestCase { - private final Log LOG = LogFactory.getLog(this.getClass()); + private final static Log LOG = LogFactory.getLog(TestFilter.class); private HRegion region; // @@ -63,10 +68,24 @@ public class TestFilter extends HBaseTestCase { Bytes.toBytes("testRowTwo-2"), Bytes.toBytes("testRowTwo-3") }; + private static final byte [][] ROWS_THREE = { + Bytes.toBytes("testRowThree-0"), Bytes.toBytes("testRowThree-1"), + Bytes.toBytes("testRowThree-2"), Bytes.toBytes("testRowThree-3") + }; + + private static final byte [][] ROWS_FOUR = { + Bytes.toBytes("testRowFour-0"), Bytes.toBytes("testRowFour-1"), + Bytes.toBytes("testRowFour-2"), Bytes.toBytes("testRowFour-3") + }; + private static final byte [][] FAMILIES = { Bytes.toBytes("testFamilyOne"), Bytes.toBytes("testFamilyTwo") }; + private static final byte [][] FAMILIES_1 = { + Bytes.toBytes("testFamilyThree"), Bytes.toBytes("testFamilyFour") + }; + private static final byte [][] QUALIFIERS_ONE = { Bytes.toBytes("testQualifierOne-0"), Bytes.toBytes("testQualifierOne-1"), Bytes.toBytes("testQualifierOne-2"), Bytes.toBytes("testQualifierOne-3") @@ -77,10 +96,24 @@ public class TestFilter extends HBaseTestCase { Bytes.toBytes("testQualifierTwo-2"), Bytes.toBytes("testQualifierTwo-3") }; + private static final byte [][] QUALIFIERS_THREE = { + Bytes.toBytes("testQualifierThree-0"), Bytes.toBytes("testQualifierThree-1"), + Bytes.toBytes("testQualifierThree-2"), Bytes.toBytes("testQualifierThree-3") + }; + + private static final byte [][] QUALIFIERS_FOUR = { + Bytes.toBytes("testQualifierFour-0"), Bytes.toBytes("testQualifierFour-1"), + Bytes.toBytes("testQualifierFour-2"), Bytes.toBytes("testQualifierFour-3") + }; + private static final byte [][] VALUES = { Bytes.toBytes("testValueOne"), Bytes.toBytes("testValueTwo") }; + byte [][] NEW_FAMILIES = { + Bytes.toBytes("f1"), Bytes.toBytes("f2") + }; + private long numRows = ROWS_ONE.length + ROWS_TWO.length; private long colsPerRow = FAMILIES.length * QUALIFIERS_ONE.length; @@ -90,6 +123,11 @@ protected void setUp() throws Exception { HTableDescriptor htd = new HTableDescriptor(getName()); htd.addFamily(new HColumnDescriptor(FAMILIES[0])); htd.addFamily(new HColumnDescriptor(FAMILIES[1])); + htd.addFamily(new HColumnDescriptor(FAMILIES_1[0])); + htd.addFamily(new HColumnDescriptor(FAMILIES_1[1])); + htd.addFamily(new HColumnDescriptor(NEW_FAMILIES[0])); + htd.addFamily(new HColumnDescriptor(NEW_FAMILIES[1])); + htd.addFamily(new HColumnDescriptor(FAMILIES_1[1])); HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); this.region = HRegion.createHRegion(info, this.testDir, this.conf, htd); @@ -170,6 +208,73 @@ protected void tearDown() throws Exception { super.tearDown(); } + + public void testRegionScannerReseek() throws Exception { + // create new rows and column family to show how reseek works.. + for (byte[] ROW : ROWS_THREE) { + Put p = new Put(ROW); + p.setWriteToWAL(true); + for (byte[] QUALIFIER : QUALIFIERS_THREE) { + p.add(FAMILIES[0], QUALIFIER, VALUES[0]); + + } + this.region.put(p); + } + for (byte[] ROW : ROWS_FOUR) { + Put p = new Put(ROW); + p.setWriteToWAL(false); + for (byte[] QUALIFIER : QUALIFIERS_FOUR) { + p.add(FAMILIES[1], QUALIFIER, VALUES[1]); + } + this.region.put(p); + } + // Flush + this.region.flushcache(); + + // Insert second half (reverse families) + for (byte[] ROW : ROWS_THREE) { + Put p = new Put(ROW); + p.setWriteToWAL(false); + for (byte[] QUALIFIER : QUALIFIERS_THREE) { + p.add(FAMILIES[1], QUALIFIER, VALUES[0]); + } + this.region.put(p); + } + for (byte[] ROW : ROWS_FOUR) { + Put p = new Put(ROW); + p.setWriteToWAL(false); + for (byte[] QUALIFIER : QUALIFIERS_FOUR) { + p.add(FAMILIES[0], QUALIFIER, VALUES[1]); + } + this.region.put(p); + } + + Scan s = new Scan(); + // set a start row + s.setStartRow(ROWS_FOUR[1]); + RegionScanner scanner = region.getScanner(s); + + // reseek to row three. + scanner.reseek(ROWS_THREE[1]); + List results = new ArrayList(); + + // the results should belong to ROWS_THREE[1] + scanner.next(results); + for (KeyValue keyValue : results) { + assertEquals("The rows with ROWS_TWO as row key should be appearing.", + Bytes.toString(keyValue.getRow()), Bytes.toString(ROWS_THREE[1])); + } + // again try to reseek to a value before ROWS_THREE[1] + scanner.reseek(ROWS_ONE[1]); + results = new ArrayList(); + // This time no seek would have been done to ROWS_ONE[1] + scanner.next(results); + for (KeyValue keyValue : results) { + assertFalse("Cannot rewind back to a value less than previous reseek.", + Bytes.toString(keyValue.getRow()).contains("testRowOne")); + } + } + public void testNoFilter() throws Exception { // No filter long expectedRows = this.numRows; @@ -608,7 +713,7 @@ public void testFamilyFilter() throws IOException { verifyScanNoEarlyOut(s, expectedRows, expectedKeys); // Match all columns in second family - // look only in second group of rows + // look only in second group of rows expectedRows = this.numRows / 2; expectedKeys = this.colsPerRow / 2; f = new FamilyFilter(CompareOp.GREATER, @@ -1121,6 +1226,146 @@ public void testFilterListWithSingleColumnValueFilter() throws IOException { verifyScanFull(s, kvs); } + // HBASE-9747 + public void testFilterListWithPrefixFilter() throws IOException { + byte[] family = Bytes.toBytes("f1"); + byte[] qualifier = Bytes.toBytes("q1"); + HTableDescriptor htd = new HTableDescriptor(getName()); + htd.addFamily(new HColumnDescriptor(family)); + HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); + HRegion testRegion = HRegion.createHRegion(info, testDir, conf, htd); + + for(int i=0; i<5; i++) { + Put p = new Put(Bytes.toBytes((char)('a'+i) + "row")); + p.setDurability(Durability.SKIP_WAL); + p.add(family, qualifier, Bytes.toBytes(String.valueOf(111+i))); + testRegion.put(p); + } + testRegion.flushcache(); + + // rows starting with "b" + PrefixFilter pf = new PrefixFilter(new byte[] {'b'}) ; + // rows with value of column 'q1' set to '113' + SingleColumnValueFilter scvf = new SingleColumnValueFilter( + family, qualifier, CompareOp.EQUAL, Bytes.toBytes("113")); + // combine these two with OR in a FilterList + FilterList filterList = new FilterList(Operator.MUST_PASS_ONE, pf, scvf); + + Scan s1 = new Scan(); + s1.setFilter(filterList); + InternalScanner scanner = testRegion.getScanner(s1); + List results = new ArrayList(); + int resultCount = 0; + while(scanner.next(results)) { + resultCount++; + byte[] row = results.get(0).getRow(); + LOG.debug("Found row: " + Bytes.toStringBinary(row)); + Assert.assertTrue(Bytes.equals(row, Bytes.toBytes("brow")) + || Bytes.equals(row, Bytes.toBytes("crow"))); + results.clear(); + } + Assert.assertEquals(2, resultCount); + scanner.close(); + + HLog hlog = testRegion.getLog(); + testRegion.close(); + hlog.closeAndDelete(); + } + + public void testNestedFilterListWithSCVF() throws IOException { + byte[] columnStatus = Bytes.toBytes("S"); + HTableDescriptor htd = new HTableDescriptor(getName()); + htd.addFamily(new HColumnDescriptor(FAMILIES[0])); + HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); + HRegion testRegion = HRegion.createHRegion(info, testDir, conf, htd); + for(int i=0; i<10; i++) { + Put p = new Put(Bytes.toBytes("row" + i)); + p.setWriteToWAL(false); + p.add(FAMILIES[0], columnStatus, Bytes.toBytes(i%2)); + testRegion.put(p); + } + testRegion.flushcache(); + // 1. got rows > "row4" + Filter rowFilter = new RowFilter(CompareOp.GREATER, new BinaryComparator( + Bytes.toBytes("row4"))); + Scan s1 = new Scan(); + s1.setFilter(rowFilter); + InternalScanner scanner = testRegion.getScanner(s1); + List results = new ArrayList(); + int i = 5; + for (boolean done = true; done; i++) { + done = scanner.next(results); + Assert.assertTrue(Bytes.equals(results.get(0).getRow(), Bytes.toBytes("row" + i))); + Assert.assertEquals(Bytes.toInt(results.get(0).getValue()), i%2); + results.clear(); + } + // 2. got rows <= "row4" and S= + FilterList subFilterList = new FilterList(FilterList.Operator.MUST_PASS_ALL); + Filter subFilter1 = new RowFilter(CompareOp.LESS_OR_EQUAL, new BinaryComparator( + Bytes.toBytes("row4"))); + subFilterList.addFilter(subFilter1); + Filter subFilter2 = new SingleColumnValueFilter(FAMILIES[0], columnStatus, CompareOp.EQUAL, + Bytes.toBytes(0)); + subFilterList.addFilter(subFilter2); + s1 = new Scan(); + s1.setFilter(subFilterList); + scanner = testRegion.getScanner(s1); + results = new ArrayList(); + for (i=0; i<=4; i+=2) { + scanner.next(results); + Assert.assertTrue(Bytes.equals(results.get(0).getRow(), Bytes.toBytes("row" + i))); + Assert.assertEquals(Bytes.toInt(results.get(0).getValue()), i%2); + results.clear(); + } + Assert.assertFalse(scanner.next(results)); + // 3. let's begin to verify nested filter list + // 3.1 add rowFilter, then add subFilterList + FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ONE); + filterList.addFilter(rowFilter); + filterList.addFilter(subFilterList); + s1 = new Scan(); + s1.setFilter(filterList); + scanner = testRegion.getScanner(s1); + results = new ArrayList(); + for (i=0; i<=4; i+=2) { + scanner.next(results); + Assert.assertTrue(Bytes.equals(results.get(0).getRow(), Bytes.toBytes("row" + i))); + Assert.assertEquals(Bytes.toInt(results.get(0).getValue()), i%2); + results.clear(); + } + for (i=5; i<=9; i++) { + scanner.next(results); + Assert.assertTrue(Bytes.equals(results.get(0).getRow(), Bytes.toBytes("row" + i))); + Assert.assertEquals(Bytes.toInt(results.get(0).getValue()), i%2); + results.clear(); + } + Assert.assertFalse(scanner.next(results)); + // 3.2 MAGIC here! add subFilterList first, then add rowFilter + filterList = new FilterList(FilterList.Operator.MUST_PASS_ONE); + filterList.addFilter(subFilterList); + filterList.addFilter(rowFilter); + s1 = new Scan(); + s1.setFilter(filterList); + scanner = testRegion.getScanner(s1); + results = new ArrayList(); + for (i=0; i<=4; i+=2) { + scanner.next(results); + Assert.assertTrue(Bytes.equals(results.get(0).getRow(), Bytes.toBytes("row" + i))); + Assert.assertEquals(Bytes.toInt(results.get(0).getValue()), i%2); + results.clear(); + } + for (i=5; i<=9; i++) { + scanner.next(results); + Assert.assertTrue(Bytes.equals(results.get(0).getRow(), Bytes.toBytes("row" + i))); + Assert.assertEquals(Bytes.toInt(results.get(0).getValue()), i%2); + results.clear(); + } + Assert.assertFalse(scanner.next(results)); + HLog hlog = testRegion.getLog(); + testRegion.close(); + hlog.closeAndDelete(); + } + public void testSingleColumnValueFilter() throws IOException { // From HBASE-1821 @@ -1345,7 +1590,7 @@ private void verifyScanFullNoValues(Scan s, KeyValue [] kvs, boolean useLen) assertFalse("Should not have returned whole value", Bytes.equals(kv.getValue(), kvs[idx].getValue())); if (useLen) { - assertEquals("Value in result is not SIZEOF_INT", + assertEquals("Value in result is not SIZEOF_INT", kv.getValue().length, Bytes.SIZEOF_INT); LOG.info("idx = " + idx + ", len=" + kvs[idx].getValueLength() + ", actual=" + Bytes.toInt(kv.getValue())); @@ -1353,7 +1598,7 @@ private void verifyScanFullNoValues(Scan s, KeyValue [] kvs, boolean useLen) kvs[idx].getValueLength(), Bytes.toInt(kv.getValue()) ); LOG.info("good"); } else { - assertEquals("Value in result is not empty", + assertEquals("Value in result is not empty", kv.getValue().length, 0); } idx++; @@ -1367,8 +1612,14 @@ private void verifyScanFullNoValues(Scan s, KeyValue [] kvs, boolean useLen) public void testColumnPaginationFilter() throws Exception { + // Test that the filter skips multiple column versions. + Put p = new Put(ROWS_ONE[0]); + p.setWriteToWAL(false); + p.add(FAMILIES[0], QUALIFIERS_ONE[0], VALUES[0]); + this.region.put(p); + this.region.flushcache(); - // Set of KVs (page: 1; pageSize: 1) - the first set of 1 column per row + // Set of KVs (page: 1; pageSize: 1) - the first set of 1 column per row KeyValue [] expectedKVs = { // testRowOne-0 new KeyValue(ROWS_ONE[0], FAMILIES[0], QUALIFIERS_ONE[0], VALUES[0]), @@ -1516,6 +1767,41 @@ public void testKeyOnlyFilter() throws Exception { verifyScanFullNoValues(s, expectedKVs, useLen); } } + + /** + * Filter which makes sleeps for a second between each row of a scan. + * This can be useful for manual testing of bugs like HBASE-5973. For example: + * + * create 't1', 'f1' + * 1.upto(100) { |x| put 't1', 'r' + x.to_s, 'f1:q1', 'hi' } + * import org.apache.hadoop.hbase.filter.TestFilter + * scan 't1', { FILTER => TestFilter::SlowScanFilter.new(), CACHE => 50 } + * + */ + public static class SlowScanFilter extends FilterBase { + private static Thread ipcHandlerThread = null; + + @Override + public void readFields(DataInput arg0) throws IOException { + } + + @Override + public void write(DataOutput arg0) throws IOException { + } + + @Override + public boolean filterRow() { + ipcHandlerThread = Thread.currentThread(); + try { + LOG.info("Handler thread " + ipcHandlerThread + " sleeping in filter..."); + Thread.sleep(1000); + } catch (InterruptedException e) { + Throwables.propagate(e); + } + return super.filterRow(); + } + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java b/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java index 2b69a6fb0fd6..110eea422e85 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,6 +18,8 @@ */ package org.apache.hadoop.hbase.filter; +import static org.junit.Assert.assertEquals; + import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.DataInput; @@ -34,10 +35,14 @@ import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp; +import org.apache.hadoop.hbase.filter.Filter.ReturnCode; import org.apache.hadoop.hbase.filter.FilterList.Operator; import org.apache.hadoop.hbase.util.Bytes; import org.junit.experimental.categories.Category; +import com.google.common.collect.Lists; + /** * Tests filter sets * @@ -79,34 +84,31 @@ public void testMPONE() throws Exception { byte [] rowkey = Bytes.toBytes("yyyyyyyyy"); for (int i = 0; i < MAX_PAGES - 1; i++) { assertFalse(filterMPONE.filterRowKey(rowkey, 0, rowkey.length)); - assertFalse(filterMPONE.filterRow()); KeyValue kv = new KeyValue(rowkey, rowkey, Bytes.toBytes(i), Bytes.toBytes(i)); assertTrue(Filter.ReturnCode.INCLUDE == filterMPONE.filterKeyValue(kv)); + assertFalse(filterMPONE.filterRow()); } /* Only pass PageFilter */ rowkey = Bytes.toBytes("z"); assertFalse(filterMPONE.filterRowKey(rowkey, 0, rowkey.length)); - assertFalse(filterMPONE.filterRow()); KeyValue kv = new KeyValue(rowkey, rowkey, Bytes.toBytes(0), Bytes.toBytes(0)); assertTrue(Filter.ReturnCode.INCLUDE == filterMPONE.filterKeyValue(kv)); + assertFalse(filterMPONE.filterRow()); - /* PageFilter will fail now, but should pass because we match yyy */ + /* reach MAX_PAGES already, should filter any rows */ rowkey = Bytes.toBytes("yyy"); - assertFalse(filterMPONE.filterRowKey(rowkey, 0, rowkey.length)); - assertFalse(filterMPONE.filterRow()); + assertTrue(filterMPONE.filterRowKey(rowkey, 0, rowkey.length)); kv = new KeyValue(rowkey, rowkey, Bytes.toBytes(0), Bytes.toBytes(0)); - assertTrue(Filter.ReturnCode.INCLUDE == filterMPONE.filterKeyValue(kv)); + assertFalse(Filter.ReturnCode.INCLUDE == filterMPONE.filterKeyValue(kv)); /* We should filter any row */ rowkey = Bytes.toBytes("z"); assertTrue(filterMPONE.filterRowKey(rowkey, 0, rowkey.length)); - assertTrue(filterMPONE.filterRow()); assertTrue(filterMPONE.filterAllRemaining()); - } /** @@ -146,9 +148,6 @@ public void testMPALL() throws Exception { // Should fail here; row should be filtered out. KeyValue kv = new KeyValue(rowkey, rowkey, rowkey, rowkey); assertTrue(Filter.ReturnCode.NEXT_ROW == filterMPALL.filterKeyValue(kv)); - - // Both filters in Set should be satisfied by now - assertTrue(filterMPALL.filterRow()); } /** @@ -207,6 +206,75 @@ public void testOrdering() throws Exception { } } + private static class AlwaysNextColFilter extends FilterBase { + public AlwaysNextColFilter() { + super(); + } + @Override + public ReturnCode filterKeyValue(KeyValue v) { + return ReturnCode.NEXT_COL; + } + @Override + public void readFields(DataInput arg0) throws IOException {} + + @Override + public void write(DataOutput arg0) throws IOException {} + } + + /** + * When we do a "MUST_PASS_ONE" (a logical 'OR') of the two filters + * we expect to get the same result as the inclusive stop result. + * @throws Exception + */ + public void testFilterListWithInclusiveStopFilteMustPassOne() throws Exception { + byte[] r1 = Bytes.toBytes("Row1"); + byte[] r11 = Bytes.toBytes("Row11"); + byte[] r2 = Bytes.toBytes("Row2"); + + FilterList flist = new FilterList(FilterList.Operator.MUST_PASS_ONE); + flist.addFilter(new AlwaysNextColFilter()); + flist.addFilter(new InclusiveStopFilter(r1)); + flist.filterRowKey(r1, 0, r1.length); + assertEquals(flist.filterKeyValue(new KeyValue(r1,r1,r1)), ReturnCode.INCLUDE); + assertEquals(flist.filterKeyValue(new KeyValue(r11,r11,r11)), ReturnCode.INCLUDE); + + flist.reset(); + flist.filterRowKey(r2, 0, r2.length); + assertEquals(flist.filterKeyValue(new KeyValue(r2,r2,r2)), ReturnCode.SKIP); + } + + /** + * When we do a "MUST_PASS_ONE" (a logical 'OR') of the above two filters + * we expect to get the same result as the 'prefix' only result. + * @throws Exception + */ + public void testFilterListTwoFiltersMustPassOne() throws Exception { + byte[] r1 = Bytes.toBytes("Row1"); + byte[] r11 = Bytes.toBytes("Row11"); + byte[] r2 = Bytes.toBytes("Row2"); + + FilterList flist = new FilterList(FilterList.Operator.MUST_PASS_ONE); + flist.addFilter(new PrefixFilter(r1)); + flist.filterRowKey(r1, 0, r1.length); + assertEquals(flist.filterKeyValue(new KeyValue(r1,r1,r1)), ReturnCode.INCLUDE); + assertEquals(flist.filterKeyValue(new KeyValue(r11,r11,r11)), ReturnCode.INCLUDE); + + flist.reset(); + flist.filterRowKey(r2, 0, r2.length); + assertEquals(flist.filterKeyValue(new KeyValue(r2,r2,r2)), ReturnCode.SKIP); + + flist = new FilterList(FilterList.Operator.MUST_PASS_ONE); + flist.addFilter(new AlwaysNextColFilter()); + flist.addFilter(new PrefixFilter(r1)); + flist.filterRowKey(r1, 0, r1.length); + assertEquals(flist.filterKeyValue(new KeyValue(r1,r1,r1)), ReturnCode.INCLUDE); + assertEquals(flist.filterKeyValue(new KeyValue(r11,r11,r11)), ReturnCode.INCLUDE); + + flist.reset(); + flist.filterRowKey(r2, 0, r2.length); + assertEquals(flist.filterKeyValue(new KeyValue(r2,r2,r2)), ReturnCode.SKIP); + } + /** * Test serialization * @throws Exception @@ -233,6 +301,77 @@ public void testSerialization() throws Exception { // TODO: Run TESTS!!! } + /** + * Test filterKeyValue logic. + * @throws Exception + */ + public void testFilterKeyValue() throws Exception { + Filter includeFilter = new FilterBase() { + @Override + public Filter.ReturnCode filterKeyValue(KeyValue v) { + return Filter.ReturnCode.INCLUDE; + } + + @Override + public void readFields(DataInput arg0) throws IOException {} + + @Override + public void write(DataOutput arg0) throws IOException {} + }; + + Filter alternateFilter = new FilterBase() { + boolean returnInclude = true; + + @Override + public Filter.ReturnCode filterKeyValue(KeyValue v) { + Filter.ReturnCode returnCode = returnInclude ? Filter.ReturnCode.INCLUDE : + Filter.ReturnCode.SKIP; + returnInclude = !returnInclude; + return returnCode; + } + + @Override + public void readFields(DataInput arg0) throws IOException {} + + @Override + public void write(DataOutput arg0) throws IOException {} + }; + + Filter alternateIncludeFilter = new FilterBase() { + boolean returnIncludeOnly = false; + + @Override + public Filter.ReturnCode filterKeyValue(KeyValue v) { + Filter.ReturnCode returnCode = returnIncludeOnly ? Filter.ReturnCode.INCLUDE : + Filter.ReturnCode.INCLUDE_AND_NEXT_COL; + returnIncludeOnly = !returnIncludeOnly; + return returnCode; + } + + @Override + public void readFields(DataInput arg0) throws IOException {} + + @Override + public void write(DataOutput arg0) throws IOException {} + }; + + // Check must pass one filter. + FilterList mpOnefilterList = new FilterList(Operator.MUST_PASS_ONE, + Arrays.asList(new Filter[] { includeFilter, alternateIncludeFilter, alternateFilter })); + // INCLUDE, INCLUDE, INCLUDE_AND_NEXT_COL. + assertEquals(Filter.ReturnCode.INCLUDE_AND_NEXT_COL, mpOnefilterList.filterKeyValue(null)); + // INCLUDE, SKIP, INCLUDE. + assertEquals(Filter.ReturnCode.INCLUDE, mpOnefilterList.filterKeyValue(null)); + + // Check must pass all filter. + FilterList mpAllfilterList = new FilterList(Operator.MUST_PASS_ALL, + Arrays.asList(new Filter[] { includeFilter, alternateIncludeFilter, alternateFilter })); + // INCLUDE, INCLUDE, INCLUDE_AND_NEXT_COL. + assertEquals(Filter.ReturnCode.INCLUDE_AND_NEXT_COL, mpAllfilterList.filterKeyValue(null)); + // INCLUDE, SKIP, INCLUDE. + assertEquals(Filter.ReturnCode.SKIP, mpAllfilterList.filterKeyValue(null)); + } + /** * Test pass-thru of hints. */ @@ -251,6 +390,11 @@ public void write(DataOutput arg0) throws IOException {} }; Filter filterMinHint = new FilterBase() { + @Override + public ReturnCode filterKeyValue(KeyValue ignored) { + return ReturnCode.SEEK_NEXT_USING_HINT; + } + @Override public KeyValue getNextKeyHint(KeyValue currentKV) { return minKeyValue; @@ -264,6 +408,11 @@ public void write(DataOutput arg0) throws IOException {} }; Filter filterMaxHint = new FilterBase() { + @Override + public ReturnCode filterKeyValue(KeyValue ignored) { + return ReturnCode.SEEK_NEXT_USING_HINT; + } + @Override public KeyValue getNextKeyHint(KeyValue currentKV) { return new KeyValue(Bytes.toBytes(Long.MAX_VALUE), null, null); @@ -301,32 +450,77 @@ public void write(DataOutput arg0) throws IOException {} // MUST PASS ALL - // Should take the max if given two hints + // Should take the first hint filterList = new FilterList(Operator.MUST_PASS_ALL, Arrays.asList(new Filter [] { filterMinHint, filterMaxHint } )); + filterList.filterKeyValue(null); + assertEquals(0, KeyValue.COMPARATOR.compare(filterList.getNextKeyHint(null), + minKeyValue)); + + filterList = new FilterList(Operator.MUST_PASS_ALL, + Arrays.asList(new Filter [] { filterMaxHint, filterMinHint } )); + filterList.filterKeyValue(null); assertEquals(0, KeyValue.COMPARATOR.compare(filterList.getNextKeyHint(null), maxKeyValue)); - // Should have max hint even if a filter has no hint + // Should have first hint even if a filter has no hint filterList = new FilterList(Operator.MUST_PASS_ALL, Arrays.asList( - new Filter [] { filterMinHint, filterMaxHint, filterNoHint } )); + new Filter [] { filterNoHint, filterMinHint, filterMaxHint } )); + filterList.filterKeyValue(null); assertEquals(0, KeyValue.COMPARATOR.compare(filterList.getNextKeyHint(null), - maxKeyValue)); + minKeyValue)); filterList = new FilterList(Operator.MUST_PASS_ALL, Arrays.asList(new Filter [] { filterNoHint, filterMaxHint } )); + filterList.filterKeyValue(null); assertEquals(0, KeyValue.COMPARATOR.compare(filterList.getNextKeyHint(null), maxKeyValue)); + filterList.filterKeyValue(null); filterList = new FilterList(Operator.MUST_PASS_ALL, Arrays.asList(new Filter [] { filterNoHint, filterMinHint } )); + filterList.filterKeyValue(null); assertEquals(0, KeyValue.COMPARATOR.compare(filterList.getNextKeyHint(null), minKeyValue)); + } - // Should give min hint if its the only one - filterList = new FilterList(Operator.MUST_PASS_ALL, - Arrays.asList(new Filter [] { filterNoHint, filterMinHint } )); - assertEquals(0, KeyValue.COMPARATOR.compare(filterList.getNextKeyHint(null), - minKeyValue)); + /** + * Tests the behavior of transform() in a hierarchical filter. + * + * transform() only applies after a filterKeyValue() whose return-code includes the KeyValue. + * Lazy evaluation of AND + */ + public void testTransformMPO() throws Exception { + // Apply the following filter: + // (family=fam AND qualifier=qual1 AND KeyOnlyFilter) + // OR (family=fam AND qualifier=qual2) + final FilterList flist = new FilterList(Operator.MUST_PASS_ONE, Lists.newArrayList( + new FilterList(Operator.MUST_PASS_ALL, Lists.newArrayList( + new FamilyFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes("fam"))), + new QualifierFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes("qual1"))), + new KeyOnlyFilter())), + new FilterList(Operator.MUST_PASS_ALL, Lists.newArrayList( + new FamilyFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes("fam"))), + new QualifierFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes("qual2"))))))); + + final KeyValue kvQual1 = new KeyValue( + Bytes.toBytes("row"), Bytes.toBytes("fam"), Bytes.toBytes("qual1"), Bytes.toBytes("value")); + final KeyValue kvQual2 = new KeyValue( + Bytes.toBytes("row"), Bytes.toBytes("fam"), Bytes.toBytes("qual2"), Bytes.toBytes("value")); + final KeyValue kvQual3 = new KeyValue( + Bytes.toBytes("row"), Bytes.toBytes("fam"), Bytes.toBytes("qual3"), Bytes.toBytes("value")); + + // Value for fam:qual1 should be stripped: + assertEquals(Filter.ReturnCode.INCLUDE, flist.filterKeyValue(kvQual1)); + final KeyValue transformedQual1 = flist.transform(kvQual1); + assertEquals(0, transformedQual1.getValue().length); + + // Value for fam:qual2 should not be stripped: + assertEquals(Filter.ReturnCode.INCLUDE, flist.filterKeyValue(kvQual2)); + final KeyValue transformedQual2 = flist.transform(kvQual2); + assertEquals("value", Bytes.toString(transformedQual2.getValue())); + + // Other keys should be skipped: + assertEquals(Filter.ReturnCode.SKIP, flist.filterKeyValue(kvQual3)); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestFuzzyRowAndColumnRangeFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestFuzzyRowAndColumnRangeFilter.java new file mode 100644 index 000000000000..e1ed7b6f8ffb --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestFuzzyRowAndColumnRangeFilter.java @@ -0,0 +1,173 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.hadoop.hbase.filter; + +import com.google.common.collect.Lists; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.KeyValueTestUtil; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.*; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; +import org.jboss.netty.buffer.ChannelBuffer; +import org.jboss.netty.buffer.ChannelBuffers; +import org.junit.*; +import org.junit.experimental.categories.Category; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import static org.junit.Assert.assertEquals; + +/** + */ +@Category(MediumTests.class) +public class TestFuzzyRowAndColumnRangeFilter { + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private final Log LOG = LogFactory.getLog(this.getClass()); + + /** + * @throws java.lang.Exception + */ + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.startMiniCluster(); + } + + /** + * @throws java.lang.Exception + */ + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * @throws java.lang.Exception + */ + @Before + public void setUp() throws Exception { + // Nothing to do. + } + + /** + * @throws java.lang.Exception + */ + @After + public void tearDown() throws Exception { + // Nothing to do. + } + + @Test + public void Test() throws Exception { + String cf = "f"; + String table = "TestFuzzyAndColumnRangeFilterClient"; + HTable ht = TEST_UTIL.createTable(Bytes.toBytes(table), + Bytes.toBytes(cf), Integer.MAX_VALUE); + + // 10 byte row key - (2 bytes 4 bytes 4 bytes) + // 4 byte qualifier + // 4 byte value + + for (int i1 = 0; i1 < 2; i1++) { + for (int i2 = 0; i2 < 5; i2++) { + byte[] rk = new byte[10]; + + ChannelBuffer buf = ChannelBuffers.wrappedBuffer(rk); + buf.clear(); + buf.writeShort((short) 2); + buf.writeInt(i1); + buf.writeInt(i2); + + for (int c = 0; c < 5; c++) { + byte[] cq = new byte[4]; + Bytes.putBytes(cq, 0, Bytes.toBytes(c), 0, 4); + + Put p = new Put(rk); + p.setDurability(Durability.SKIP_WAL); + p.add(cf.getBytes(), cq, Bytes.toBytes(c)); + ht.put(p); + LOG.info("Inserting: rk: " + Bytes.toStringBinary(rk) + " cq: " + + Bytes.toStringBinary(cq)); + } + } + } + + TEST_UTIL.flush(); + + // test passes + runTest(ht, 0, 10); + + // test fails + runTest(ht, 1, 8); + } + + private void runTest(HTable hTable, int cqStart, int expectedSize) throws IOException { + // [0, 2, ?, ?, ?, ?, 0, 0, 0, 1] + byte[] fuzzyKey = new byte[10]; + ChannelBuffer buf = ChannelBuffers.wrappedBuffer(fuzzyKey); + buf.clear(); + buf.writeShort((short) 2); + for (int i = 0; i < 4; i++) + buf.writeByte((short)63); + buf.writeInt((short)1); + + byte[] mask = new byte[] {0 , 0, 1, 1, 1, 1, 0, 0, 0, 0}; + + Pair pair = new Pair(fuzzyKey, mask); + FuzzyRowFilter fuzzyRowFilter = new FuzzyRowFilter(Lists.newArrayList(pair)); + ColumnRangeFilter columnRangeFilter = new ColumnRangeFilter(Bytes.toBytes(cqStart), true + , Bytes.toBytes(4), true); + //regular test + runScanner(hTable, expectedSize, fuzzyRowFilter, columnRangeFilter); + //reverse filter order test + runScanner(hTable, expectedSize, columnRangeFilter, fuzzyRowFilter); + } + + private void runScanner(HTable hTable, int expectedSize, Filter... filters) throws IOException { + String cf = "f"; + Scan scan = new Scan(); + scan.addFamily(cf.getBytes()); + FilterList filterList = new FilterList(filters); + scan.setFilter(filterList); + + ResultScanner scanner = hTable.getScanner(scan); + List results = new ArrayList(); + Result result; + long timeBeforeScan = System.currentTimeMillis(); + while ((result = scanner.next()) != null) { + for (KeyValue kv : result.list()) { + LOG.info("Got rk: " + Bytes.toStringBinary(kv.getRow()) + " cq: " + + Bytes.toStringBinary(kv.getQualifier())); + results.add(kv); + } + } + long scanTime = System.currentTimeMillis() - timeBeforeScan; + scanner.close(); + + LOG.info("scan time = " + scanTime + "ms"); + LOG.info("found " + results.size() + " results"); + + assertEquals(expectedSize, results.size()); + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestFuzzyRowFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestFuzzyRowFilter.java new file mode 100644 index 000000000000..4faca8216508 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestFuzzyRowFilter.java @@ -0,0 +1,204 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.filter; + +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Assert; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(SmallTests.class) +public class TestFuzzyRowFilter { + @Test + public void testSatisfies() { + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.NEXT_EXISTS, + FuzzyRowFilter.satisfies(new byte[]{1, (byte) -128, 0, 0, 1}, // row to check + new byte[]{1, 0, 1}, // fuzzy row + new byte[]{0, 1, 0})); // mask + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.YES, + FuzzyRowFilter.satisfies(new byte[]{1, (byte) -128, 1, 0, 1}, + new byte[]{1, 0, 1}, + new byte[]{0, 1, 0})); + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.NEXT_EXISTS, + FuzzyRowFilter.satisfies(new byte[]{1, (byte) -128, 2, 0, 1}, + new byte[]{1, 0, 1}, + new byte[]{0, 1, 0})); + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.NO_NEXT, + FuzzyRowFilter.satisfies(new byte[]{2, 3, 1, 1, 1}, + new byte[]{1, 0, 1}, + new byte[]{0, 1, 0})); + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.YES, + FuzzyRowFilter.satisfies(new byte[]{1, 2, 1, 3, 3}, + new byte[]{1, 2, 0, 3}, + new byte[]{0, 0, 1, 0})); + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.NEXT_EXISTS, + FuzzyRowFilter.satisfies(new byte[]{1, 1, 1, 3, 0}, // row to check + new byte[]{1, 2, 0, 3}, // fuzzy row + new byte[]{0, 0, 1, 0})); // mask + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.NEXT_EXISTS, + FuzzyRowFilter.satisfies(new byte[]{1, 1, 1, 3, 0}, + new byte[]{1, (byte) 245, 0, 3}, + new byte[]{0, 0, 1, 0})); + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.NO_NEXT, + FuzzyRowFilter.satisfies(new byte[]{1, (byte) 245, 1, 3, 0}, + new byte[]{1, 1, 0, 3}, + new byte[]{0, 0, 1, 0})); + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.NO_NEXT, + FuzzyRowFilter.satisfies(new byte[]{1, 3, 1, 3, 0}, + new byte[]{1, 2, 0, 3}, + new byte[]{0, 0, 1, 0})); + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.NO_NEXT, + FuzzyRowFilter.satisfies(new byte[]{2, 1, 1, 1, 0}, + new byte[]{1, 2, 0, 3}, + new byte[]{0, 0, 1, 0})); + + Assert.assertEquals(FuzzyRowFilter.SatisfiesCode.NEXT_EXISTS, + FuzzyRowFilter.satisfies(new byte[]{1, 2, 1, 0, 1}, + new byte[]{0, 1, 2}, + new byte[]{1, 0, 0})); + } + + @Test + public void testGetNextForFuzzyRule() { + assertNext( + new byte[]{0, 1, 2}, // fuzzy row + new byte[]{1, 0, 0}, // mask + new byte[]{1, 2, 1, 0, 1}, // current + new byte[]{2, 1, 2, 0, 0}); // expected next + + assertNext( + new byte[]{0, 1, 2}, // fuzzy row + new byte[]{1, 0, 0}, // mask + new byte[]{1, 1, 2, 0, 1}, // current + new byte[]{1, 1, 2, 0, 2}); // expected next + + assertNext( + new byte[]{0, 1, 0, 2, 0}, // fuzzy row + new byte[]{1, 0, 1, 0, 1}, // mask + new byte[]{1, 0, 2, 0, 1}, // current + new byte[]{1, 1, 0, 2, 0}); // expected next + + assertNext( + new byte[]{1, 0, 1}, + new byte[]{0, 1, 0}, + new byte[]{1, (byte) 128, 2, 0, 1}, + new byte[]{1, (byte) 129, 1, 0, 0}); + + assertNext( + new byte[]{0, 1, 0, 1}, + new byte[]{1, 0, 1, 0}, + new byte[]{5, 1, 0, 1}, + new byte[]{5, 1, 1, 1}); + + assertNext( + new byte[]{0, 1, 0, 1}, + new byte[]{1, 0, 1, 0}, + new byte[]{5, 1, 0, 1, 1}, + new byte[]{5, 1, 0, 1, 2}); + + assertNext( + new byte[]{0, 1, 0, 0}, // fuzzy row + new byte[]{1, 0, 1, 1}, // mask + new byte[]{5, 1, (byte) 255, 1}, // current + new byte[]{5, 1, (byte) 255, 2}); // expected next + + assertNext( + new byte[]{0, 1, 0, 1}, // fuzzy row + new byte[]{1, 0, 1, 0}, // mask + new byte[]{5, 1, (byte) 255, 1}, // current + new byte[]{6, 1, 0, 1}); // expected next + + assertNext( + new byte[]{0, 1, 0, 1}, // fuzzy row + new byte[]{1, 0, 1, 0}, // mask + new byte[]{5, 1, (byte) 255, 0}, // current + new byte[]{5, 1, (byte) 255, 1}); // expected next + + assertNext( + new byte[]{5, 1, 1, 0}, + new byte[]{0, 0, 1, 1}, + new byte[]{5, 1, (byte) 255, 1}, + new byte[]{5, 1, (byte) 255, 2}); + + assertNext( + new byte[]{1, 1, 1, 1}, + new byte[]{0, 0, 1, 1}, + new byte[]{1, 1, 2, 2}, + new byte[]{1, 1, 2, 3}); + + assertNext( + new byte[]{1, 1, 1, 1}, + new byte[]{0, 0, 1, 1}, + new byte[]{1, 1, 3, 2}, + new byte[]{1, 1, 3, 3}); + + assertNext( + new byte[]{1, 1, 1, 1}, + new byte[]{1, 1, 1, 1}, + new byte[]{1, 1, 2, 3}, + new byte[]{1, 1, 2, 4}); + + assertNext( + new byte[]{1, 1, 1, 1}, + new byte[]{1, 1, 1, 1}, + new byte[]{1, 1, 3, 2}, + new byte[]{1, 1, 3, 3}); + + assertNext( + new byte[]{1, 1, 0, 0}, + new byte[]{0, 0, 1, 1}, + new byte[]{0, 1, 3, 2}, + new byte[]{1, 1, 0, 0}); + + // No next for this one + Assert.assertNull(FuzzyRowFilter.getNextForFuzzyRule( + new byte[]{2, 3, 1, 1, 1}, // row to check + new byte[]{1, 0, 1}, // fuzzy row + new byte[]{0, 1, 0})); // mask + Assert.assertNull(FuzzyRowFilter.getNextForFuzzyRule( + new byte[]{1, (byte) 245, 1, 3, 0}, + new byte[]{1, 1, 0, 3}, + new byte[]{0, 0, 1, 0})); + Assert.assertNull(FuzzyRowFilter.getNextForFuzzyRule( + new byte[]{1, 3, 1, 3, 0}, + new byte[]{1, 2, 0, 3}, + new byte[]{0, 0, 1, 0})); + Assert.assertNull(FuzzyRowFilter.getNextForFuzzyRule( + new byte[]{2, 1, 1, 1, 0}, + new byte[]{1, 2, 0, 3}, + new byte[]{0, 0, 1, 0})); + } + + private void assertNext(byte[] fuzzyRow, byte[] mask, byte[] current, byte[] expected) { + byte[] nextForFuzzyRule = FuzzyRowFilter.getNextForFuzzyRule(current, fuzzyRow, mask); + Assert.assertArrayEquals(expected, nextForFuzzyRule); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestInclusiveStopFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestInclusiveStopFilter.java index a12bd8eaa16a..50395cf6f01b 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestInclusiveStopFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestInclusiveStopFilter.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestInvocationRecordFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestInvocationRecordFilter.java new file mode 100644 index 000000000000..ee2eb8c3b6da --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestInvocationRecordFilter.java @@ -0,0 +1,192 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.filter; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.InternalScanner; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test the invocation logic of the filters. A filter must be invoked only for + * the columns that are requested for. + */ +@Category(SmallTests.class) +public class TestInvocationRecordFilter { + + private static final byte[] TABLE_NAME_BYTES = Bytes + .toBytes("invocationrecord"); + private static final byte[] FAMILY_NAME_BYTES = Bytes.toBytes("mycf"); + + private static final byte[] ROW_BYTES = Bytes.toBytes("row"); + private static final String QUALIFIER_PREFIX = "qualifier"; + private static final String VALUE_PREFIX = "value"; + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private HRegion region; + + @Before + public void setUp() throws Exception { + HTableDescriptor htd = new HTableDescriptor(TABLE_NAME_BYTES); + htd.addFamily(new HColumnDescriptor(FAMILY_NAME_BYTES)); + HRegionInfo info = new HRegionInfo(TABLE_NAME_BYTES, null, null, false); + this.region = HRegion.createHRegion(info, TEST_UTIL.getDataTestDir(), + TEST_UTIL.getConfiguration(), htd); + + Put put = new Put(ROW_BYTES); + for (int i = 0; i < 10; i += 2) { + // puts 0, 2, 4, 6 and 8 + put.add(FAMILY_NAME_BYTES, Bytes.toBytes(QUALIFIER_PREFIX + i), i, + Bytes.toBytes(VALUE_PREFIX + i)); + } + this.region.put(put); + this.region.flushcache(); + } + + @Test + public void testFilterInvocation() throws Exception { + List selectQualifiers = new ArrayList(); + List expectedQualifiers = new ArrayList(); + + selectQualifiers.add(-1); + verifyInvocationResults(selectQualifiers.toArray(new Integer[0]), + expectedQualifiers.toArray(new Integer[0])); + + selectQualifiers.clear(); + + selectQualifiers.add(0); + expectedQualifiers.add(0); + verifyInvocationResults(selectQualifiers.toArray(new Integer[0]), + expectedQualifiers.toArray(new Integer[0])); + + selectQualifiers.add(3); + verifyInvocationResults(selectQualifiers.toArray(new Integer[0]), + expectedQualifiers.toArray(new Integer[0])); + + selectQualifiers.add(4); + expectedQualifiers.add(4); + verifyInvocationResults(selectQualifiers.toArray(new Integer[0]), + expectedQualifiers.toArray(new Integer[0])); + + selectQualifiers.add(5); + verifyInvocationResults(selectQualifiers.toArray(new Integer[0]), + expectedQualifiers.toArray(new Integer[0])); + + selectQualifiers.add(8); + expectedQualifiers.add(8); + verifyInvocationResults(selectQualifiers.toArray(new Integer[0]), + expectedQualifiers.toArray(new Integer[0])); + } + + public void verifyInvocationResults(Integer[] selectQualifiers, + Integer[] expectedQualifiers) throws Exception { + Get get = new Get(ROW_BYTES); + for (int i = 0; i < selectQualifiers.length; i++) { + get.addColumn(FAMILY_NAME_BYTES, + Bytes.toBytes(QUALIFIER_PREFIX + selectQualifiers[i])); + } + + get.setFilter(new InvocationRecordFilter()); + + List expectedValues = new ArrayList(); + for (int i = 0; i < expectedQualifiers.length; i++) { + expectedValues.add(new KeyValue(ROW_BYTES, FAMILY_NAME_BYTES, Bytes + .toBytes(QUALIFIER_PREFIX + expectedQualifiers[i]), + expectedQualifiers[i], Bytes.toBytes(VALUE_PREFIX + + expectedQualifiers[i]))); + } + + Scan scan = new Scan(get); + List actualValues = new ArrayList(); + List temp = new ArrayList(); + InternalScanner scanner = this.region.getScanner(scan); + while (scanner.next(temp)) { + actualValues.addAll(temp); + temp.clear(); + } + actualValues.addAll(temp); + Assert.assertTrue("Actual values " + actualValues + + " differ from the expected values:" + expectedValues, + expectedValues.equals(actualValues)); + } + + @After + public void tearDown() throws Exception { + HLog hlog = region.getLog(); + region.close(); + hlog.closeAndDelete(); + } + + /** + * Filter which gives the list of keyvalues for which the filter is invoked. + */ + private static class InvocationRecordFilter extends FilterBase { + + private List visitedKeyValues = new ArrayList(); + + public void reset() { + visitedKeyValues.clear(); + } + + public ReturnCode filterKeyValue(KeyValue ignored) { + visitedKeyValues.add(ignored); + return ReturnCode.INCLUDE; + } + + public void filterRow(List kvs) { + kvs.clear(); + kvs.addAll(visitedKeyValues); + } + + public boolean hasFilterRow() { + return true; + } + + @Override + public void readFields(DataInput arg0) throws IOException { + //do nothing + } + + @Override + public void write(DataOutput arg0) throws IOException { + //do nothing + } + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestNullComparator.java b/src/test/java/org/apache/hadoop/hbase/filter/TestNullComparator.java new file mode 100644 index 000000000000..37dbbe9dccdd --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestNullComparator.java @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.hadoop.hbase.filter; + +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Assert; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(SmallTests.class) +public class TestNullComparator { + + @Test + public void testNullValue() + { + // given + byte[] value = null; + NullComparator comparator = new NullComparator(); + + // when + int comp1 = comparator.compareTo(value); + int comp2 = comparator.compareTo(value, 5, 15); + + // then + Assert.assertEquals(0, comp1); + Assert.assertEquals(0, comp2); + } + + @Test + public void testNonNullValue() { + // given + byte[] value = new byte[] { 0, 1, 2, 3, 4, 5 }; + NullComparator comparator = new NullComparator(); + + // when + int comp1 = comparator.compareTo(value); + int comp2 = comparator.compareTo(value, 1, 3); + + // then + Assert.assertEquals(1, comp1); + Assert.assertEquals(1, comp2); + } + + @Test + public void testEmptyValue() { + // given + byte[] value = new byte[] { 0 }; + NullComparator comparator = new NullComparator(); + + // when + int comp1 = comparator.compareTo(value); + int comp2 = comparator.compareTo(value, 1, 3); + + // then + Assert.assertEquals(1, comp1); + Assert.assertEquals(1, comp2); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestPageFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestPageFilter.java index ee9a9611a5dc..8bc03ae7039c 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestPageFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestPageFilter.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java index 922a6c1d332b..3323fbd6e22f 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestPrefixFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestPrefixFilter.java index 24a999e51d60..48d2a0654dc5 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestPrefixFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestPrefixFilter.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestRandomRowFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestRandomRowFilter.java index 7731cd9ccbc9..9662c9ec8fee 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestRandomRowFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestRandomRowFilter.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueExcludeFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueExcludeFilter.java index b7cb6b5211b7..4e348ba2a51e 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueExcludeFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueExcludeFilter.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -27,6 +26,9 @@ import org.apache.hadoop.hbase.util.Bytes; import org.junit.experimental.categories.Category; +import java.util.List; +import java.util.ArrayList; + /** * Tests for {@link SingleColumnValueExcludeFilter}. Because this filter * extends {@link SingleColumnValueFilter}, only the added functionality is @@ -53,16 +55,18 @@ public void testFilterKeyValue() throws Exception { CompareOp.EQUAL, VAL_1); // A 'match' situation - KeyValue kv; - kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER_2, VAL_1); - // INCLUDE expected because test column has not yet passed - assertTrue("otherColumn", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE); - kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_1); - // Test column will pass (will match), will SKIP because test columns are excluded - assertTrue("testedMatch", filter.filterKeyValue(kv) == Filter.ReturnCode.SKIP); - // Test column has already passed and matched, all subsequent columns are INCLUDE - kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER_2, VAL_1); - assertTrue("otherColumn", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE); + List kvs = new ArrayList(); + KeyValue kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER_2, VAL_1); + + kvs.add (new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER_2, VAL_1)); + kvs.add (new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_1)); + kvs.add (new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER_2, VAL_1)); + + filter.filterRow(kvs); + + assertEquals("resultSize", kvs.size(), 2); + assertTrue("leftKV1", KeyValue.COMPARATOR.compare(kvs.get(0), kv) == 0); + assertTrue("leftKV2", KeyValue.COMPARATOR.compare(kvs.get(1), kv) == 0); assertFalse("allRemainingWhenMatch", filter.filterAllRemaining()); // A 'mismatch' situation diff --git a/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueFilter.java b/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueFilter.java index 2a0751c587ee..2726b347e277 100644 --- a/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueFilter.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -52,6 +51,7 @@ public class TestSingleColumnValueFilter extends TestCase { private static final String QUICK_REGEX = ".+quick.+"; Filter basicFilter; + Filter nullFilter; Filter substrFilter; Filter regexFilter; @@ -59,6 +59,7 @@ public class TestSingleColumnValueFilter extends TestCase { protected void setUp() throws Exception { super.setUp(); basicFilter = basicFilterNew(); + nullFilter = nullFilterNew(); substrFilter = substrFilterNew(); regexFilter = regexFilterNew(); } @@ -68,6 +69,11 @@ private Filter basicFilterNew() { CompareOp.GREATER_OR_EQUAL, VAL_2); } + private Filter nullFilterNew() { + return new SingleColumnValueFilter(COLUMN_FAMILY, COLUMN_QUALIFIER, CompareOp.NOT_EQUAL, + new NullComparator()); + } + private Filter substrFilterNew() { return new SingleColumnValueFilter(COLUMN_FAMILY, COLUMN_QUALIFIER, CompareOp.EQUAL, @@ -105,6 +111,17 @@ private void basicFilterTests(SingleColumnValueFilter filter) assertFalse("basicFilterNotNull", filter.filterRow()); } + private void nullFilterTests(Filter filter) throws Exception { + ((SingleColumnValueFilter) filter).setFilterIfMissing(true); + KeyValue kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, FULLSTRING_1); + assertTrue("null1", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE); + assertFalse("null1FilterRow", filter.filterRow()); + filter.reset(); + kv = new KeyValue(ROW, COLUMN_FAMILY, Bytes.toBytes("qual2"), FULLSTRING_2); + assertTrue("null2", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE); + assertTrue("null2FilterRow", filter.filterRow()); + } + private void substrFilterTests(Filter filter) throws Exception { KeyValue kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, @@ -154,7 +171,8 @@ private Filter serializationTest(Filter filter) * @throws Exception */ public void testStop() throws Exception { - basicFilterTests((SingleColumnValueFilter)basicFilter); + basicFilterTests((SingleColumnValueFilter) basicFilter); + nullFilterTests(nullFilter); substrFilterTests(substrFilter); regexFilterTests(regexFilter); } @@ -166,6 +184,8 @@ public void testStop() throws Exception { public void testSerialization() throws Exception { Filter newFilter = serializationTest(basicFilter); basicFilterTests((SingleColumnValueFilter)newFilter); + newFilter = serializationTest(nullFilter); + nullFilterTests(newFilter); newFilter = serializationTest(substrFilter); substrFilterTests(newFilter); newFilter = serializationTest(regexFilter); diff --git a/src/test/java/org/apache/hadoop/hbase/io/TestFileLink.java b/src/test/java/org/apache/hadoop/hbase/io/TestFileLink.java new file mode 100644 index 000000000000..88310ef900f6 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/io/TestFileLink.java @@ -0,0 +1,244 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.io; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import org.junit.Test; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import junit.framework.TestCase; +import org.junit.experimental.categories.Category; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.FSDataInputStream; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.io.FileLink; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +/** + * Test that FileLink switches between alternate locations + * when the current location moves or gets deleted. + */ +@Category(MediumTests.class) +public class TestFileLink { + /** + * Test, on HDFS, that the FileLink is still readable + * even when the current file gets renamed. + */ + @Test + public void testHDFSLinkReadDuringRename() throws Exception { + HBaseTestingUtility testUtil = new HBaseTestingUtility(); + Configuration conf = testUtil.getConfiguration(); + conf.setInt("dfs.blocksize", 1024 * 1024); + conf.setInt("dfs.client.read.prefetch.size", 2 * 1024 * 1024); + + testUtil.startMiniDFSCluster(1); + MiniDFSCluster cluster = testUtil.getDFSCluster(); + FileSystem fs = cluster.getFileSystem(); + assertEquals("hdfs", fs.getUri().getScheme()); + + try { + testLinkReadDuringRename(fs, testUtil.getDefaultRootDirPath()); + } finally { + testUtil.shutdownMiniCluster(); + } + } + + /** + * Test, on a local filesystem, that the FileLink is still readable + * even when the current file gets renamed. + */ + @Test + public void testLocalLinkReadDuringRename() throws IOException { + HBaseTestingUtility testUtil = new HBaseTestingUtility(); + FileSystem fs = testUtil.getTestFileSystem(); + assertEquals("file", fs.getUri().getScheme()); + testLinkReadDuringRename(fs, testUtil.getDataTestDir()); + } + + /** + * Test that link is still readable even when the current file gets renamed. + */ + private void testLinkReadDuringRename(FileSystem fs, Path rootDir) throws IOException { + Path originalPath = new Path(rootDir, "test.file"); + Path archivedPath = new Path(rootDir, "archived.file"); + + writeSomeData(fs, originalPath, 256 << 20, (byte)2); + + List files = new ArrayList(); + files.add(originalPath); + files.add(archivedPath); + + FileLink link = new FileLink(files); + FSDataInputStream in = link.open(fs); + try { + byte[] data = new byte[8192]; + long size = 0; + + // Read from origin + int n = in.read(data); + dataVerify(data, n, (byte)2); + size += n; + + // Move origin to archive + assertFalse(fs.exists(archivedPath)); + fs.rename(originalPath, archivedPath); + assertFalse(fs.exists(originalPath)); + assertTrue(fs.exists(archivedPath)); + + // Try to read to the end + while ((n = in.read(data)) > 0) { + dataVerify(data, n, (byte)2); + size += n; + } + + assertEquals(256 << 20, size); + } finally { + in.close(); + if (fs.exists(originalPath)) fs.delete(originalPath); + if (fs.exists(archivedPath)) fs.delete(archivedPath); + } + } + + /** + * Test that link is still readable even when the current file gets deleted. + * + * NOTE: This test is valid only on HDFS. + * When a file is deleted from a local file-system, it is simply 'unlinked'. + * The inode, which contains the file's data, is not deleted until all + * processes have finished with it. + * In HDFS when the request exceed the cached block locations, + * a query to the namenode is performed, using the filename, + * and the deleted file doesn't exists anymore (FileNotFoundException). + */ + @Test + public void testHDFSLinkReadDuringDelete() throws Exception { + HBaseTestingUtility testUtil = new HBaseTestingUtility(); + Configuration conf = testUtil.getConfiguration(); + conf.setInt("dfs.blocksize", 1024 * 1024); + conf.setInt("dfs.client.read.prefetch.size", 2 * 1024 * 1024); + + testUtil.startMiniDFSCluster(1); + MiniDFSCluster cluster = testUtil.getDFSCluster(); + FileSystem fs = cluster.getFileSystem(); + assertEquals("hdfs", fs.getUri().getScheme()); + + try { + List files = new ArrayList(); + for (int i = 0; i < 3; i++) { + Path path = new Path(String.format("test-data-%d", i)); + writeSomeData(fs, path, 1 << 20, (byte)i); + files.add(path); + } + + FileLink link = new FileLink(files); + FSDataInputStream in = link.open(fs); + try { + byte[] data = new byte[8192]; + int n; + + // Switch to file 1 + n = in.read(data); + dataVerify(data, n, (byte)0); + fs.delete(files.get(0)); + skipBuffer(in, (byte)0); + + // Switch to file 2 + n = in.read(data); + dataVerify(data, n, (byte)1); + fs.delete(files.get(1)); + skipBuffer(in, (byte)1); + + // Switch to file 3 + n = in.read(data); + dataVerify(data, n, (byte)2); + fs.delete(files.get(2)); + skipBuffer(in, (byte)2); + + // No more files available + try { + n = in.read(data); + assert(n <= 0); + } catch (FileNotFoundException e) { + assertTrue(true); + } + } finally { + in.close(); + } + } finally { + testUtil.shutdownMiniCluster(); + } + } + + /** + * Write up to 'size' bytes with value 'v' into a new file called 'path'. + */ + private void writeSomeData (FileSystem fs, Path path, long size, byte v) throws IOException { + byte[] data = new byte[4096]; + for (int i = 0; i < data.length; i++) { + data[i] = v; + } + + FSDataOutputStream stream = fs.create(path); + try { + long written = 0; + while (written < size) { + stream.write(data, 0, data.length); + written += data.length; + } + } finally { + stream.close(); + } + } + + /** + * Verify that all bytes in 'data' have 'v' as value. + */ + private static void dataVerify(byte[] data, int n, byte v) { + for (int i = 0; i < n; ++i) { + assertEquals(v, data[i]); + } + } + + private static void skipBuffer(FSDataInputStream in, byte v) throws IOException { + byte[] data = new byte[8192]; + try { + int n; + while ((n = in.read(data)) == data.length) { + for (int i = 0; i < data.length; ++i) { + if (data[i] != v) + throw new Exception("File changed"); + } + } + } catch (Exception e) { + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java b/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java index 717432faebc8..d655a6be48fb 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java +++ b/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,6 +19,8 @@ package org.apache.hadoop.hbase.io; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; import java.io.IOException; @@ -123,6 +124,92 @@ private void doTestOfScanAndReseek(Path p, FileSystem fs, Reference bottom, halfreader.close(true); } + + // Tests the scanner on an HFile that is backed by HalfStoreFiles + @Test + public void testHalfScanner() throws IOException { + HBaseTestingUtility test_util = new HBaseTestingUtility(); + String root_dir = test_util.getDataTestDir("TestHalfStoreFileScanBefore").toString(); + Path p = new Path(root_dir, "test"); + Configuration conf = test_util.getConfiguration(); + FileSystem fs = FileSystem.get(conf); + CacheConfig cacheConf = new CacheConfig(conf); + + HFile.Writer w = HFile.getWriterFactory(conf, cacheConf) + .withPath(fs, p) + .withBlockSize(1024) + .withComparator(KeyValue.KEY_COMPARATOR) + .create(); + + // write some things. + List items = genSomeKeys(); + for (KeyValue kv : items) { + w.append(kv); + } + w.close(); + + + HFile.Reader r = HFile.createReader(fs, p, cacheConf); + r.loadFileInfo(); + byte[] midkey = r.midkey(); + KeyValue midKV = KeyValue.createKeyValueFromKey(midkey); + midkey = midKV.getRow(); + + Reference bottom = new Reference(midkey, Reference.Range.bottom); + Reference top = new Reference(midkey, Reference.Range.top); + + // Ugly code to get the item before the midkey + KeyValue beforeMidKey = null; + for (KeyValue item : items) { + if (item.equals(midKV)) { + break; + } + beforeMidKey = item; + } + + + // Seek on the splitKey, should be in top, not in bottom + KeyValue foundKeyValue = doTestOfSeekBefore(p, fs, bottom, midKV, cacheConf); + assertEquals(beforeMidKey, foundKeyValue); + + // Seek tot the last thing should be the penultimate on the top, the one before the midkey on the bottom. + foundKeyValue = doTestOfSeekBefore(p, fs, top, items.get(items.size() - 1), cacheConf); + assertEquals(items.get(items.size() - 2), foundKeyValue); + + foundKeyValue = doTestOfSeekBefore(p, fs, bottom, items.get(items.size() - 1), cacheConf); + assertEquals(beforeMidKey, foundKeyValue); + + // Try and seek before something that is in the bottom. + foundKeyValue = doTestOfSeekBefore(p, fs, top, items.get(0), cacheConf); + assertNull(foundKeyValue); + + // Try and seek before the first thing. + foundKeyValue = doTestOfSeekBefore(p, fs, bottom, items.get(0), cacheConf); + assertNull(foundKeyValue); + + // Try and seek before the second thing in the top and bottom. + foundKeyValue = doTestOfSeekBefore(p, fs, top, items.get(1), cacheConf); + assertNull(foundKeyValue); + + foundKeyValue = doTestOfSeekBefore(p, fs, bottom, items.get(1), cacheConf); + assertEquals(items.get(0), foundKeyValue); + + // Try to seek before the splitKey in the top file + foundKeyValue = doTestOfSeekBefore(p, fs, top, midKV, cacheConf); + assertNull(foundKeyValue); + } + + private KeyValue doTestOfSeekBefore(Path p, FileSystem fs, Reference bottom, KeyValue seekBefore, + CacheConfig cacheConfig) + throws IOException { + final HalfStoreFileReader halfreader = new HalfStoreFileReader(fs, p, + cacheConfig, bottom, DataBlockEncoding.NONE); + halfreader.loadFileInfo(); + final HFileScanner scanner = halfreader.getScanner(false, false); + scanner.seekBefore(seekBefore.getKey()); + return scanner.getKeyValue(); + } + private KeyValue getLastOnCol(KeyValue curr) { return KeyValue.createLastOnRow( curr.getBuffer(), curr.getRowOffset(), curr.getRowLength(), diff --git a/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java b/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java index f2f8ee38b6e2..239e1484c876 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java +++ b/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -197,6 +196,14 @@ public void testReadObjectDataInputConfiguration() throws IOException { obj = doType(conf, list, List.class); assertTrue(obj instanceof List); Assert.assertArrayEquals(list.toArray(), ((List)obj).toArray() ); + //List.class with null values + List listWithNulls = new ArrayList(); + listWithNulls.add("hello"); + listWithNulls.add("world"); + listWithNulls.add(null); + obj = doType(conf, listWithNulls, List.class); + assertTrue(obj instanceof List); + Assert.assertArrayEquals(listWithNulls.toArray(), ((List)obj).toArray() ); //ArrayList.class ArrayList arr = new ArrayList(); arr.add("hello"); @@ -531,7 +538,7 @@ public void testGetClassCode() throws IOException{ * note on the test above. */ public void testGetNextObjectCode(){ - assertEquals(82,HbaseObjectWritable.getNextClassCode()); + assertEquals(83,HbaseObjectWritable.getNextClassCode()); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java b/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java index a3c9ae9f5cde..cb29cacc69ff 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java +++ b/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -44,8 +43,10 @@ import org.apache.hadoop.hbase.io.hfile.CachedBlock; import org.apache.hadoop.hbase.io.hfile.LruBlockCache; import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet; import org.apache.hadoop.hbase.regionserver.MemStore; import org.apache.hadoop.hbase.regionserver.Store; +import org.apache.hadoop.hbase.regionserver.TimeRangeTracker; import org.apache.hadoop.hbase.regionserver.metrics.SchemaConfigured; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.ClassSize; @@ -207,7 +208,23 @@ public void testNativeSizes() throws IOException { assertEquals(expected, actual); } + // TimeRangeTracker + cl = TimeRangeTracker.class; + expected = ClassSize.estimateBase(cl, false); + actual = ClassSize.TIMERANGE_TRACKER; + if (expected != actual) { + ClassSize.estimateBase(cl, true); + assertEquals(expected, actual); + } + // KeyValueSkipListSet + cl = KeyValueSkipListSet.class; + expected = ClassSize.estimateBase(cl, false); + actual = ClassSize.KEYVALUE_SKIPLIST_SET; + if (expected != actual) { + ClassSize.estimateBase(cl, true); + assertEquals(expected, actual); + } } /** @@ -280,19 +297,19 @@ public void testSizes() throws IOException { // MemStore Deep Overhead actual = MemStore.DEEP_OVERHEAD; expected = ClassSize.estimateBase(cl, false); - expected += ClassSize.estimateBase(ReentrantReadWriteLock.class, false); expected += ClassSize.estimateBase(AtomicLong.class, false); - expected += ClassSize.estimateBase(ConcurrentSkipListMap.class, false); - expected += ClassSize.estimateBase(ConcurrentSkipListMap.class, false); - expected += ClassSize.estimateBase(CopyOnWriteArraySet.class, false); - expected += ClassSize.estimateBase(CopyOnWriteArrayList.class, false); + expected += (2 * ClassSize.estimateBase(KeyValueSkipListSet.class, false)); + expected += (2 * ClassSize.estimateBase(ConcurrentSkipListMap.class, false)); + expected += (2 * ClassSize.estimateBase(TimeRangeTracker.class, false)); if(expected != actual) { ClassSize.estimateBase(cl, true); - ClassSize.estimateBase(ReentrantReadWriteLock.class, true); ClassSize.estimateBase(AtomicLong.class, true); + ClassSize.estimateBase(KeyValueSkipListSet.class, true); + ClassSize.estimateBase(KeyValueSkipListSet.class, true); + ClassSize.estimateBase(ConcurrentSkipListMap.class, true); ClassSize.estimateBase(ConcurrentSkipListMap.class, true); - ClassSize.estimateBase(CopyOnWriteArraySet.class, true); - ClassSize.estimateBase(CopyOnWriteArrayList.class, true); + ClassSize.estimateBase(TimeRangeTracker.class, true); + ClassSize.estimateBase(TimeRangeTracker.class, true); assertEquals(expected, actual); } diff --git a/src/test/java/org/apache/hadoop/hbase/io/TestImmutableBytesWritable.java b/src/test/java/org/apache/hadoop/hbase/io/TestImmutableBytesWritable.java index 6217dec3da36..39ea85782002 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/TestImmutableBytesWritable.java +++ b/src/test/java/org/apache/hadoop/hbase/io/TestImmutableBytesWritable.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java b/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java index d3825025ae2c..7742b4eb8f57 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java +++ b/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java @@ -34,6 +34,7 @@ import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.Durability; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; @@ -67,7 +68,7 @@ public class TestChangingEncoding { new HBaseTestingUtility(); private static final Configuration conf = TEST_UTIL.getConfiguration(); - private static final int TIMEOUT_MS = 120000; + private static final int TIMEOUT_MS = 600000; private HBaseAdmin admin; private HColumnDescriptor hcd; @@ -134,14 +135,17 @@ static void writeTestDataBatch(Configuration conf, String tableName, int batchId) throws Exception { LOG.debug("Writing test data batch " + batchId); HTable table = new HTable(conf, tableName); + table.setAutoFlush(false); for (int i = 0; i < NUM_ROWS_PER_BATCH; ++i) { Put put = new Put(getRowKey(batchId, i)); for (int j = 0; j < NUM_COLS_PER_ROW; ++j) { put.add(CF_BYTES, getQualifier(j), getValue(batchId, i, j)); - table.put(put); } + put.setDurability(Durability.SKIP_WAL); + table.put(put); } + table.flushCommits(); table.close(); } @@ -238,10 +242,18 @@ public void testFlippingEncodeOnDisk() throws Exception { private void compactAndWait() throws IOException, InterruptedException { LOG.debug("Compacting table " + tableName); admin.majorCompact(tableName); - Threads.sleepWithoutInterrupt(500); HRegionServer rs = TEST_UTIL.getMiniHBaseCluster().getRegionServer(0); + + // Waiting for the compaction to start, at least .5s. + final long maxWaitime = System.currentTimeMillis() + 500; + boolean cont; + do { + cont = rs.compactSplitThread.getCompactionQueueSize() == 0; + Threads.sleep(1); + } while (cont && System.currentTimeMillis() < maxWaitime); + while (rs.compactSplitThread.getCompactionQueueSize() > 0) { - Threads.sleep(50); + Threads.sleep(5); } LOG.debug("Compaction queue size reached 0, continuing"); } diff --git a/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java b/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java index f7b73977518e..607a19246f39 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java +++ b/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java @@ -120,6 +120,26 @@ public void testNegativeTimestamps() throws IOException { includesMemstoreTS)); } + /** + * Test KeyValues with negative timestamp. + * @throws IOException On test failure. + */ + @Test + public void testZeroByte() throws IOException { + List kvList = new ArrayList(); + byte[] row = Bytes.toBytes("abcd"); + byte[] family = new byte[] { 'f' }; + byte[] qualifier0 = new byte[] { 'b' }; + byte[] qualifier1 = new byte[] { 'c' }; + byte[] value0 = new byte[] { 'd' }; + byte[] value1 = new byte[] { 0x00 }; + kvList.add(new KeyValue(row, family, qualifier0, 0, Type.Put, value0)); + kvList.add(new KeyValue(row, family, qualifier1, 0, Type.Put, value1)); + testEncodersOnDataset( + RedundantKVGenerator.convertKvToByteBuffer(kvList, + includesMemstoreTS)); + } + /** * Test whether compression -> decompression gives the consistent results on * pseudorandom sample. diff --git a/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java b/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java index f40afe48b7be..46b55620dc0c 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java +++ b/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java @@ -32,14 +32,11 @@ import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.io.hfile.CacheConfig; -import org.apache.hadoop.hbase.io.hfile.Compression.Algorithm; -import org.apache.hadoop.hbase.io.hfile.HFile; import org.apache.hadoop.hbase.io.hfile.LruBlockCache; import org.apache.hadoop.hbase.regionserver.HRegion; -import org.apache.hadoop.hbase.regionserver.StoreFile.BloomType; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.LoadTestKVGenerator; -import org.apache.hadoop.hbase.util.MultiThreadedWriter; + import org.junit.Test; import org.junit.experimental.categories.Category; import org.junit.runner.RunWith; @@ -101,17 +98,18 @@ TABLE_NAME, new HColumnDescriptor(CF_NAME) .setDataBlockEncoding(encoding) .setEncodeOnDisk(encodeOnDisk) ); + LoadTestKVGenerator dataGenerator = new LoadTestKVGenerator( MIN_VALUE_SIZE, MAX_VALUE_SIZE); // Write for (int i = 0; i < NUM_ROWS; ++i) { - byte[] key = MultiThreadedWriter.longToByteArrayKey(i); + byte[] key = LoadTestKVGenerator.md5PrefixedKey(i).getBytes(); for (int j = 0; j < NUM_COLS_PER_ROW; ++j) { Put put = new Put(key); - String colAsStr = String.valueOf(j); - byte[] value = dataGenerator.generateRandomSizeValue(i, colAsStr); - put.add(CF_BYTES, Bytes.toBytes(colAsStr), value); + byte[] col = Bytes.toBytes(String.valueOf(j)); + byte[] value = dataGenerator.generateRandomSizeValue(key, col); + put.add(CF_BYTES, col, value); region.put(put); } if (i % NUM_ROWS_PER_FLUSH == 0) { @@ -122,7 +120,7 @@ TABLE_NAME, new HColumnDescriptor(CF_NAME) for (int doneCompaction = 0; doneCompaction <= 1; ++doneCompaction) { // Read for (int i = 0; i < NUM_ROWS; ++i) { - final byte[] rowKey = MultiThreadedWriter.longToByteArrayKey(i); + byte[] rowKey = LoadTestKVGenerator.md5PrefixedKey(i).getBytes(); for (int j = 0; j < NUM_COLS_PER_ROW; ++j) { if (VERBOSE) { System.err.println("Reading row " + i + ", column " + j); @@ -131,10 +129,10 @@ TABLE_NAME, new HColumnDescriptor(CF_NAME) final byte[] qualBytes = Bytes.toBytes(qualStr); Get get = new Get(rowKey); get.addColumn(CF_BYTES, qualBytes); - Result result = region.get(get, null); + Result result = region.get(get); assertEquals(1, result.size()); - assertTrue(LoadTestKVGenerator.verify(Bytes.toString(rowKey), qualStr, - result.getValue(CF_BYTES, qualBytes))); + byte[] value = result.getValue(CF_BYTES, qualBytes); + assertTrue(LoadTestKVGenerator.verify(value, rowKey, qualBytes)); } } diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java index 61ce0775cab6..537ecd5700d6 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -37,6 +36,7 @@ import org.apache.hadoop.hbase.MultithreadedTestUtil; import org.apache.hadoop.hbase.MultithreadedTestUtil.TestThread; import org.apache.hadoop.hbase.io.HeapSize; +import org.apache.hadoop.hbase.util.ChecksumType; import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; public class CacheTestUtils { @@ -91,12 +91,12 @@ public void doAnAction() throws Exception { } toBeTested.cacheBlock(ourBlock.blockName, ourBlock.block); Cacheable retrievedBlock = toBeTested.getBlock(ourBlock.blockName, - false); + false, false); if (retrievedBlock != null) { assertEquals(ourBlock.block, retrievedBlock); toBeTested.evictBlock(ourBlock.blockName); hits.incrementAndGet(); - assertNull(toBeTested.getBlock(ourBlock.blockName, false)); + assertNull(toBeTested.getBlock(ourBlock.blockName, false, false)); } else { miss.incrementAndGet(); } @@ -124,7 +124,7 @@ public static void testCacheSimple(BlockCache toBeTested, int blockSize, HFileBlockPair[] blocks = generateHFileBlocks(numBlocks, blockSize); // Confirm empty for (HFileBlockPair block : blocks) { - assertNull(toBeTested.getBlock(block.blockName, true)); + assertNull(toBeTested.getBlock(block.blockName, true, false)); } // Add blocks @@ -137,7 +137,7 @@ public static void testCacheSimple(BlockCache toBeTested, int blockSize, // MapMaker makes no guarantees when it will evict, so neither can we. for (HFileBlockPair block : blocks) { - HFileBlock buf = (HFileBlock) toBeTested.getBlock(block.blockName, true); + HFileBlock buf = (HFileBlock) toBeTested.getBlock(block.blockName, true, false); if (buf != null) { assertEquals(block.block, buf); } @@ -148,7 +148,7 @@ public static void testCacheSimple(BlockCache toBeTested, int blockSize, for (HFileBlockPair block : blocks) { try { - if (toBeTested.getBlock(block.blockName, true) != null) { + if (toBeTested.getBlock(block.blockName, true, false) != null) { toBeTested.cacheBlock(block.blockName, block.block); fail("Cache should not allow re-caching a block"); } @@ -178,7 +178,7 @@ public static void hammerSingleKey(final BlockCache toBeTested, @Override public void doAnAction() throws Exception { ByteArrayCacheable returned = (ByteArrayCacheable) toBeTested - .getBlock(key, false); + .getBlock(key, false, false); assertArrayEquals(buf, returned.buf); totalQueries.incrementAndGet(); } @@ -217,7 +217,7 @@ public void doAnAction() throws Exception { final ByteArrayCacheable bac = new ByteArrayCacheable(buf); ByteArrayCacheable gotBack = (ByteArrayCacheable) toBeTested - .getBlock(key, true); + .getBlock(key, true, false); if (gotBack != null) { assertArrayEquals(gotBack.buf, bac.buf); } else { @@ -323,7 +323,9 @@ private static HFileBlockPair[] generateHFileBlocks(int blockSize, HFileBlock generated = new HFileBlock(BlockType.DATA, onDiskSizeWithoutHeader, uncompressedSizeWithoutHeader, prevBlockOffset, cachedBuffer, HFileBlock.DONT_FILL_HEADER, - blockSize, includesMemstoreTS); + blockSize, includesMemstoreTS, HFileBlock.MINOR_VERSION_NO_CHECKSUM, + 0, ChecksumType.NULL.getCode(), + onDiskSizeWithoutHeader + HFileBlock.HEADER_SIZE_WITH_CHECKSUMS); String strKey; /* No conflicting keys */ diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java index a48a69f4212b..5bb53a664741 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java index 76738063721a..e6a957ef748a 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java index 6007d5a1e46a..7c503c0100e8 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -42,6 +41,7 @@ import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.fs.HFileSystem; import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.StoreFile; @@ -49,6 +49,7 @@ import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; import org.apache.hadoop.hbase.util.BloomFilterFactory; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.ChecksumType; import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; import org.junit.After; import org.junit.Before; @@ -88,6 +89,8 @@ public class TestCacheOnWrite { private static final int INDEX_BLOCK_SIZE = 512; private static final int BLOOM_BLOCK_SIZE = 4096; private static final BloomType BLOOM_TYPE = StoreFile.BloomType.ROWCOL; + private static final ChecksumType CKTYPE = ChecksumType.CRC32; + private static final int CKBYTES = 512; /** The number of valid key types possible in a store file */ private static final int NUM_VALID_KEY_TYPES = @@ -192,7 +195,7 @@ public void setUp() throws IOException { conf.setBoolean(CacheConfig.CACHE_BLOOM_BLOCKS_ON_WRITE_KEY, cowType.shouldBeCached(BlockType.BLOOM_CHUNK)); cowType.modifyConf(conf); - fs = FileSystem.get(conf); + fs = HFileSystem.get(conf); cacheConf = new CacheConfig(conf); blockCache = cacheConf.getBlockCache(); } @@ -236,7 +239,7 @@ private void readStoreFile() throws IOException { false, null); BlockCacheKey blockCacheKey = new BlockCacheKey(reader.getName(), offset, encodingInCache, block.getBlockType()); - boolean isCached = blockCache.getBlock(blockCacheKey, true) != null; + boolean isCached = blockCache.getBlock(blockCacheKey, true, false) != null; boolean shouldBeCached = cowType.shouldBeCached(block.getBlockType()); if (shouldBeCached != isCached) { throw new AssertionError( @@ -292,6 +295,8 @@ public void writeStoreFile() throws IOException { .withComparator(KeyValue.COMPARATOR) .withBloomType(BLOOM_TYPE) .withMaxKeyCount(NUM_KV) + .withChecksumType(CKTYPE) + .withBytesPerChecksum(CKBYTES) .build(); final int rowLen = 32; diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java index 0dbe3e3ef74c..8769ad99ffe5 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java new file mode 100644 index 000000000000..168fc67204fa --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java @@ -0,0 +1,275 @@ +/* + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.io.hfile; + +import static org.junit.Assert.*; + +import java.io.ByteArrayInputStream; +import java.io.DataOutputStream; +import java.io.DataInputStream; +import java.io.IOException; +import java.nio.ByteBuffer; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.FSDataInputStream; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.fs.HFileSystem; +import org.apache.hadoop.hbase.io.hfile.Compression.Algorithm; +import org.apache.hadoop.hbase.util.ChecksumType; + +import static org.apache.hadoop.hbase.io.hfile.Compression.Algorithm.*; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(MediumTests.class) +public class TestChecksum { + // change this value to activate more logs + private static final boolean detailedLogging = true; + private static final boolean[] BOOLEAN_VALUES = new boolean[] { false, true }; + + private static final Log LOG = LogFactory.getLog(TestHFileBlock.class); + + static final Compression.Algorithm[] COMPRESSION_ALGORITHMS = { + NONE, GZ }; + + static final int[] BYTES_PER_CHECKSUM = { + 50, 500, 688, 16*1024, (16*1024+980), 64 * 1024}; + + private static final HBaseTestingUtility TEST_UTIL = + new HBaseTestingUtility(); + private FileSystem fs; + private HFileSystem hfs; + + @Before + public void setUp() throws Exception { + fs = HFileSystem.get(TEST_UTIL.getConfiguration()); + hfs = (HFileSystem)fs; + } + + /** + * Introduce checksum failures and check that we can still read + * the data + */ + @Test + public void testChecksumCorruption() throws IOException { + for (Compression.Algorithm algo : COMPRESSION_ALGORITHMS) { + for (boolean pread : new boolean[] { false, true }) { + LOG.info("testChecksumCorruption: Compression algorithm: " + algo + + ", pread=" + pread); + Path path = new Path(TEST_UTIL.getDataTestDir(), "blocks_v2_" + + algo); + FSDataOutputStream os = fs.create(path); + HFileBlock.Writer hbw = new HFileBlock.Writer(algo, null, + true, 1, HFile.DEFAULT_CHECKSUM_TYPE, + HFile.DEFAULT_BYTES_PER_CHECKSUM); + long totalSize = 0; + for (int blockId = 0; blockId < 2; ++blockId) { + DataOutputStream dos = hbw.startWriting(BlockType.DATA); + for (int i = 0; i < 1234; ++i) + dos.writeInt(i); + hbw.writeHeaderAndData(os); + totalSize += hbw.getOnDiskSizeWithHeader(); + } + os.close(); + + // Use hbase checksums. + assertEquals(true, hfs.useHBaseChecksum()); + + // Do a read that purposely introduces checksum verification failures. + FSDataInputStream is = fs.open(path); + HFileBlock.FSReader hbr = new FSReaderV2Test(is, algo, + totalSize, HFile.MAX_FORMAT_VERSION, fs, path); + HFileBlock b = hbr.readBlockData(0, -1, -1, pread); + b.sanityCheck(); + assertEquals(4936, b.getUncompressedSizeWithoutHeader()); + assertEquals(algo == GZ ? 2173 : 4936, + b.getOnDiskSizeWithoutHeader() - b.totalChecksumBytes()); + // read data back from the hfile, exclude header and checksum + ByteBuffer bb = b.getBufferWithoutHeader(); // read back data + DataInputStream in = new DataInputStream( + new ByteArrayInputStream( + bb.array(), bb.arrayOffset(), bb.limit())); + + // assert that we encountered hbase checksum verification failures + // but still used hdfs checksums and read data successfully. + assertEquals(1, HFile.getChecksumFailuresCount()); + validateData(in); + + // A single instance of hbase checksum failure causes the reader to + // switch off hbase checksum verification for the next 100 read + // requests. Verify that this is correct. + for (int i = 0; i < + HFileBlock.CHECKSUM_VERIFICATION_NUM_IO_THRESHOLD + 1; i++) { + b = hbr.readBlockData(0, -1, -1, pread); + assertEquals(0, HFile.getChecksumFailuresCount()); + } + // The next read should have hbase checksum verification reanabled, + // we verify this by assertng that there was a hbase-checksum failure. + b = hbr.readBlockData(0, -1, -1, pread); + assertEquals(1, HFile.getChecksumFailuresCount()); + + // Since the above encountered a checksum failure, we switch + // back to not checking hbase checksums. + b = hbr.readBlockData(0, -1, -1, pread); + assertEquals(0, HFile.getChecksumFailuresCount()); + is.close(); + + // Now, use a completely new reader. Switch off hbase checksums in + // the configuration. In this case, we should not detect + // any retries within hbase. + HFileSystem newfs = new HFileSystem(TEST_UTIL.getConfiguration(), false); + assertEquals(false, newfs.useHBaseChecksum()); + is = newfs.open(path); + hbr = new FSReaderV2Test(is, algo, + totalSize, HFile.MAX_FORMAT_VERSION, newfs, path); + b = hbr.readBlockData(0, -1, -1, pread); + is.close(); + b.sanityCheck(); + assertEquals(4936, b.getUncompressedSizeWithoutHeader()); + assertEquals(algo == GZ ? 2173 : 4936, + b.getOnDiskSizeWithoutHeader() - b.totalChecksumBytes()); + // read data back from the hfile, exclude header and checksum + bb = b.getBufferWithoutHeader(); // read back data + in = new DataInputStream(new ByteArrayInputStream( + bb.array(), bb.arrayOffset(), bb.limit())); + + // assert that we did not encounter hbase checksum verification failures + // but still used hdfs checksums and read data successfully. + assertEquals(0, HFile.getChecksumFailuresCount()); + validateData(in); + } + } + } + + /** + * Test different values of bytesPerChecksum + */ + @Test + public void testChecksumChunks() throws IOException { + Compression.Algorithm algo = NONE; + for (boolean pread : new boolean[] { false, true }) { + for (int bytesPerChecksum : BYTES_PER_CHECKSUM) { + Path path = new Path(TEST_UTIL.getDataTestDir(), "checksumChunk_" + + algo + bytesPerChecksum); + FSDataOutputStream os = fs.create(path); + HFileBlock.Writer hbw = new HFileBlock.Writer(algo, null, + true, 1,HFile.DEFAULT_CHECKSUM_TYPE, bytesPerChecksum); + + // write one block. The block has data + // that is at least 6 times more than the checksum chunk size + long dataSize = 0; + DataOutputStream dos = hbw.startWriting(BlockType.DATA); + for (; dataSize < 6 * bytesPerChecksum;) { + for (int i = 0; i < 1234; ++i) { + dos.writeInt(i); + dataSize += 4; + } + } + hbw.writeHeaderAndData(os); + long totalSize = hbw.getOnDiskSizeWithHeader(); + os.close(); + + long expectedChunks = ChecksumUtil.numChunks( + dataSize + HFileBlock.HEADER_SIZE_WITH_CHECKSUMS, + bytesPerChecksum); + LOG.info("testChecksumChunks: pread=" + pread + + ", bytesPerChecksum=" + bytesPerChecksum + + ", fileSize=" + totalSize + + ", dataSize=" + dataSize + + ", expectedChunks=" + expectedChunks); + + // Verify hbase checksums. + assertEquals(true, hfs.useHBaseChecksum()); + + // Read data back from file. + FSDataInputStream is = fs.open(path); + FSDataInputStream nochecksum = hfs.getNoChecksumFs().open(path); + HFileBlock.FSReader hbr = new HFileBlock.FSReaderV2(is, nochecksum, + algo, totalSize, HFile.MAX_FORMAT_VERSION, hfs, path); + HFileBlock b = hbr.readBlockData(0, -1, -1, pread); + is.close(); + b.sanityCheck(); + assertEquals(dataSize, b.getUncompressedSizeWithoutHeader()); + + // verify that we have the expected number of checksum chunks + assertEquals(totalSize, HFileBlock.HEADER_SIZE_WITH_CHECKSUMS + dataSize + + expectedChunks * HFileBlock.CHECKSUM_SIZE); + + // assert that we did not encounter hbase checksum verification failures + assertEquals(0, HFile.getChecksumFailuresCount()); + } + } + } + + /** + * Test to ensure that these is at least one valid checksum implementation + */ + @Test + public void testChecksumAlgorithm() throws IOException { + ChecksumType type = ChecksumType.CRC32; + assertEquals(ChecksumType.nameToType(type.getName()), type); + assertEquals(ChecksumType.valueOf(type.toString()), type); + } + + private void validateData(DataInputStream in) throws IOException { + // validate data + for (int i = 0; i < 1234; i++) { + int val = in.readInt(); + if (val != i) { + String msg = "testChecksumCorruption: data mismatch at index " + + i + " expected " + i + " found " + val; + LOG.warn(msg); + assertEquals(i, val); + } + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); + + /** + * A class that introduces hbase-checksum failures while + * reading data from hfiles. This should trigger the hdfs level + * checksum validations. + */ + static private class FSReaderV2Test extends HFileBlock.FSReaderV2 { + + FSReaderV2Test(FSDataInputStream istream, Algorithm algo, + long fileSize, int minorVersion, FileSystem fs, + Path path) throws IOException { + super(istream, istream, algo, fileSize, minorVersion, + (HFileSystem)fs, path); + } + + @Override + protected boolean validateBlockChecksum(HFileBlock block, + byte[] data, int hdrSize) throws IOException { + return false; // checksum validation failure + } + } +} + diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java index 1d0f2cffd7c9..7f875828e025 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -52,7 +51,7 @@ public class TestFixedFileTrailer { private static final Log LOG = LogFactory.getLog(TestFixedFileTrailer.class); /** The number of used fields by version. Indexed by version minus one. */ - private static final int[] NUM_FIELDS_BY_VERSION = new int[] { 8, 13 }; + private static final int[] NUM_FIELDS_BY_VERSION = new int[] { 9, 14 }; private HBaseTestingUtility util = new HBaseTestingUtility(); private FileSystem fs; @@ -83,7 +82,8 @@ public void setUp() throws IOException { @Test public void testTrailer() throws IOException { - FixedFileTrailer t = new FixedFileTrailer(version); + FixedFileTrailer t = new FixedFileTrailer(version, + HFileBlock.MINOR_VERSION_NO_CHECKSUM); t.setDataIndexCount(3); t.setEntryCount(((long) Integer.MAX_VALUE) + 1); @@ -121,7 +121,8 @@ public void testTrailer() throws IOException { // Finished writing, trying to read. { DataInputStream dis = new DataInputStream(bais); - FixedFileTrailer t2 = new FixedFileTrailer(version); + FixedFileTrailer t2 = new FixedFileTrailer(version, + HFileBlock.MINOR_VERSION_NO_CHECKSUM); t2.deserialize(dis); assertEquals(-1, bais.read()); // Ensure we have read everything. checkLoadedTrailer(version, t, t2); @@ -139,7 +140,7 @@ public void testTrailer() throws IOException { try { readTrailer(trailerPath); fail("Exception expected"); - } catch (IOException ex) { + } catch (IllegalArgumentException ex) { // Make it easy to debug this. String msg = ex.getMessage(); String cleanMsg = msg.replaceAll( @@ -191,7 +192,7 @@ private void writeTrailer(Path trailerPath, FixedFileTrailer t, private void checkLoadedTrailer(int version, FixedFileTrailer expected, FixedFileTrailer loaded) throws IOException { - assertEquals(version, loaded.getVersion()); + assertEquals(version, loaded.getMajorVersion()); assertEquals(expected.getDataIndexCount(), loaded.getDataIndexCount()); assertEquals(Math.min(expected.getEntryCount(), diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java index 5f8214e0db1e..15b7d13367e0 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java @@ -70,28 +70,30 @@ public class TestForceCacheImportantBlocks { /** Extremely small block size, so that we can get some index blocks */ private static final int BLOCK_SIZE = 256; - private static final Algorithm COMPRESSION_ALGORITHM = - Compression.Algorithm.GZ; private static final BloomType BLOOM_TYPE = BloomType.ROW; private final int hfileVersion; private final boolean cfCacheEnabled; + private final Algorithm compressionAlgorithm; @Parameters public static Collection parameters() { // HFile versions return Arrays.asList(new Object[][] { - new Object[] { new Integer(1), false }, - new Object[] { new Integer(1), true }, - new Object[] { new Integer(2), false }, - new Object[] { new Integer(2), true } + new Object[] { new Integer(1), false, Compression.Algorithm.NONE }, + new Object[] { new Integer(1), true, Compression.Algorithm.NONE }, + new Object[] { new Integer(2), false, Compression.Algorithm.NONE }, + new Object[] { new Integer(2), true, Compression.Algorithm.NONE }, + new Object[] { new Integer(2), false, Compression.Algorithm.GZ }, + new Object[] { new Integer(2), true, Compression.Algorithm.GZ } }); } public TestForceCacheImportantBlocks(int hfileVersion, - boolean cfCacheEnabled) { + boolean cfCacheEnabled, Algorithm compression) { this.hfileVersion = hfileVersion; this.cfCacheEnabled = cfCacheEnabled; + this.compressionAlgorithm = compression; TEST_UTIL.getConfiguration().setInt(HFile.FORMAT_VERSION_KEY, hfileVersion); } @@ -106,7 +108,7 @@ public void testCacheBlocks() throws IOException { HColumnDescriptor hcd = new HColumnDescriptor(Bytes.toBytes(CF)) .setMaxVersions(MAX_VERSIONS) - .setCompressionType(COMPRESSION_ALGORITHM) + .setCompressionType(compressionAlgorithm) .setBloomFilterType(BLOOM_TYPE); hcd.setBlocksize(BLOCK_SIZE); hcd.setBlockCacheEnabled(cfCacheEnabled); diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java index bb992b8a66d0..da9b5ac09396 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -30,12 +29,15 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseTestCase; +import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.KeyValue.KeyComparator; +import org.apache.hadoop.hbase.SmallTests; import org.apache.hadoop.hbase.io.hfile.HFile.Reader; import org.apache.hadoop.hbase.io.hfile.HFile.Writer; -import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.io.Writable; import org.junit.experimental.categories.Category; @@ -56,11 +58,9 @@ public class TestHFile extends HBaseTestCase { private final int minBlockSize = 512; private static String localFormatter = "%010d"; private static CacheConfig cacheConf = null; - private Map startingMetrics; @Override public void setUp() throws Exception { - startingMetrics = SchemaMetrics.getMetricsSnapshot(); ROOT_DIR = this.getUnitTestdir("TestHFile").toString(); super.setUp(); } @@ -68,7 +68,6 @@ public void setUp() throws Exception { @Override public void tearDown() throws Exception { super.tearDown(); - SchemaMetrics.validateMetricChanges(startingMetrics); } @@ -89,6 +88,61 @@ public void testEmptyHFile() throws IOException { assertNull(r.getLastKey()); } + /** + * Create 0-length hfile and show that it fails + */ + public void testCorrupt0LengthHFile() throws IOException { + if (cacheConf == null) cacheConf = new CacheConfig(conf); + Path f = new Path(ROOT_DIR, getName()); + FSDataOutputStream fsos = fs.create(f); + fsos.close(); + + try { + Reader r = HFile.createReader(fs, f, cacheConf); + } catch (CorruptHFileException che) { + // Expected failure + return; + } + fail("Should have thrown exception"); + } + + public static void truncateFile(FileSystem fs, Path src, Path dst) throws IOException { + FileStatus fst = fs.getFileStatus(src); + long len = fst.getLen(); + len = len / 2 ; + + // create a truncated hfile + FSDataOutputStream fdos = fs.create(dst); + byte[] buf = new byte[(int)len]; + FSDataInputStream fdis = fs.open(src); + fdis.read(buf); + fdos.write(buf); + fdis.close(); + fdos.close(); + } + + /** + * Create a truncated hfile and verify that exception thrown. + */ + public void testCorruptTruncatedHFile() throws IOException { + if (cacheConf == null) cacheConf = new CacheConfig(conf); + Path f = new Path(ROOT_DIR, getName()); + Writer w = HFile.getWriterFactory(conf, cacheConf).withPath(this.fs, f).create(); + writeSomeRecords(w, 0, 100); + w.close(); + + Path trunc = new Path(f.getParent(), "trucated"); + truncateFile(fs, w.getPath(), trunc); + + try { + Reader r = HFile.createReader(fs, trunc, cacheConf); + } catch (CorruptHFileException che) { + // Expected failure + return; + } + fail("Should have thrown exception"); + } + // write some records into the tfile // write them twice private int writeSomeRecords(Writer writer, int start, int n) @@ -280,12 +334,14 @@ public void testNullMetaBlocks() throws Exception { } /** - * Make sure the orginals for our compression libs doesn't change on us. + * Make sure the ordinals for our compression algorithms do not change on us. */ public void testCompressionOrdinance() { assertTrue(Compression.Algorithm.LZO.ordinal() == 0); assertTrue(Compression.Algorithm.GZ.ordinal() == 1); assertTrue(Compression.Algorithm.NONE.ordinal() == 2); + assertTrue(Compression.Algorithm.SNAPPY.ordinal() == 3); + assertTrue(Compression.Algorithm.LZ4.ordinal() == 4); } public void testComparator() throws IOException { diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java index e8b7df07591c..5d00aa2e074e 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -48,9 +47,11 @@ import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.fs.HFileSystem; import org.apache.hadoop.hbase.io.DoubleOutputStream; import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.ChecksumType; import org.apache.hadoop.hbase.util.ClassSize; import org.apache.hadoop.io.WritableUtils; import org.apache.hadoop.io.compress.CompressionOutputStream; @@ -102,16 +103,16 @@ public static Collection parameters() { @Before public void setUp() throws IOException { - fs = FileSystem.get(TEST_UTIL.getConfiguration()); + fs = HFileSystem.get(TEST_UTIL.getConfiguration()); } - public void writeTestBlockContents(DataOutputStream dos) throws IOException { + static void writeTestBlockContents(DataOutputStream dos) throws IOException { // This compresses really well. for (int i = 0; i < 1000; ++i) dos.writeInt(i / 100); } - private int writeTestKeyValues(OutputStream dos, int seed) + static int writeTestKeyValues(OutputStream dos, int seed, boolean includesMemstoreTS) throws IOException { List keyValues = new ArrayList(); Random randomizer = new Random(42l + seed); // just any fixed number @@ -191,23 +192,27 @@ public byte[] createTestV1Block(Compression.Algorithm algo) return baos.toByteArray(); } - private byte[] createTestV2Block(Compression.Algorithm algo) - throws IOException { + static HFileBlock.Writer createTestV2Block(Compression.Algorithm algo, + boolean includesMemstoreTS) throws IOException { final BlockType blockType = BlockType.DATA; HFileBlock.Writer hbw = new HFileBlock.Writer(algo, null, - includesMemstoreTS); + includesMemstoreTS, HFileReaderV2.MAX_MINOR_VERSION, + HFile.DEFAULT_CHECKSUM_TYPE, + HFile.DEFAULT_BYTES_PER_CHECKSUM); DataOutputStream dos = hbw.startWriting(blockType); writeTestBlockContents(dos); - byte[] headerAndData = hbw.getHeaderAndData(); + dos.flush(); + byte[] headerAndData = hbw.getHeaderAndDataForTest(); assertEquals(1000 * 4, hbw.getUncompressedSizeWithoutHeader()); hbw.releaseCompressor(); - return headerAndData; + return hbw; } public String createTestBlockStr(Compression.Algorithm algo, int correctLength) throws IOException { - byte[] testV2Block = createTestV2Block(algo); - int osOffset = HFileBlock.HEADER_SIZE + 9; + HFileBlock.Writer hbw = createTestV2Block(algo, includesMemstoreTS); + byte[] testV2Block = hbw.getHeaderAndDataForTest(); + int osOffset = HFileBlock.HEADER_SIZE_WITH_CHECKSUMS + 9; if (testV2Block.length == correctLength) { // Force-set the "OS" field of the gzip header to 3 (Unix) to avoid // variations across operating systems. @@ -221,14 +226,16 @@ public String createTestBlockStr(Compression.Algorithm algo, @Test public void testNoCompression() throws IOException { - assertEquals(4000 + HFileBlock.HEADER_SIZE, createTestV2Block(NONE).length); + assertEquals(4000, createTestV2Block(NONE, includesMemstoreTS). + getBlockForCaching().getUncompressedSizeWithoutHeader()); } @Test public void testGzipCompression() throws IOException { final String correctTestBlockStr = - "DATABLK*\\x00\\x00\\x00:\\x00\\x00\\x0F\\xA0\\xFF\\xFF\\xFF\\xFF" + "DATABLK*\\x00\\x00\\x00>\\x00\\x00\\x0F\\xA0\\xFF\\xFF\\xFF\\xFF" + "\\xFF\\xFF\\xFF\\xFF" + + "\\x01\\x00\\x00@\\x00\\x00\\x00\\x00[" // gzip-compressed block: http://www.gzip.org/zlib/rfc-gzip.html + "\\x1F\\x8B" // gzip magic signature + "\\x08" // Compression method: 8 = "deflate" @@ -237,13 +244,19 @@ public void testGzipCompression() throws IOException { + "\\x00" // XFL (extra flags) // OS (0 = FAT filesystems, 3 = Unix). However, this field // sometimes gets set to 0 on Linux and Mac, so we reset it to 3. + // This appears to be a difference caused by the availability + // (and use) of the native GZ codec. + "\\x03" + "\\xED\\xC3\\xC1\\x11\\x00 \\x08\\xC00DD\\xDD\\x7Fa" + "\\xD6\\xE8\\xA3\\xB9K\\x84`\\x96Q\\xD3\\xA8\\xDB\\xA8e\\xD4c" - + "\\xD46\\xEA5\\xEA3\\xEA7\\xE7\\x00LI\\s\\xA0\\x0F\\x00\\x00"; - final int correctGzipBlockLength = 82; - assertEquals(correctTestBlockStr, createTestBlockStr(GZ, - correctGzipBlockLength)); + + "\\xD46\\xEA5\\xEA3\\xEA7\\xE7\\x00LI\\x5Cs\\xA0\\x0F\\x00\\x00" + + "\\x00\\x00\\x00\\x00"; // 4 byte checksum (ignored) + final int correctGzipBlockLength = 95; + final String testBlockStr = createTestBlockStr(GZ, correctGzipBlockLength); + // We ignore the block checksum because createTestBlockStr can change the + // gzip header after the block is produced + assertEquals(correctTestBlockStr.substring(0, correctGzipBlockLength - 4), + testBlockStr.substring(0, correctGzipBlockLength - 4)); } @Test @@ -285,11 +298,16 @@ public void testReaderV1() throws IOException { public void testReaderV2() throws IOException { for (Compression.Algorithm algo : COMPRESSION_ALGORITHMS) { for (boolean pread : new boolean[] { false, true }) { + LOG.info("testReaderV2: Compression algorithm: " + algo + + ", pread=" + pread); Path path = new Path(TEST_UTIL.getDataTestDir(), "blocks_v2_" + algo); FSDataOutputStream os = fs.create(path); HFileBlock.Writer hbw = new HFileBlock.Writer(algo, null, - includesMemstoreTS); + includesMemstoreTS, + HFileReaderV2.MAX_MINOR_VERSION, + HFile.DEFAULT_CHECKSUM_TYPE, + HFile.DEFAULT_BYTES_PER_CHECKSUM); long totalSize = 0; for (int blockId = 0; blockId < 2; ++blockId) { DataOutputStream dos = hbw.startWriting(BlockType.DATA); @@ -305,21 +323,24 @@ public void testReaderV2() throws IOException { totalSize); HFileBlock b = hbr.readBlockData(0, -1, -1, pread); is.close(); + assertEquals(0, HFile.getChecksumFailuresCount()); b.sanityCheck(); assertEquals(4936, b.getUncompressedSizeWithoutHeader()); - assertEquals(algo == GZ ? 2173 : 4936, b.getOnDiskSizeWithoutHeader()); + assertEquals(algo == GZ ? 2173 : 4936, + b.getOnDiskSizeWithoutHeader() - b.totalChecksumBytes()); String blockStr = b.toString(); if (algo == GZ) { is = fs.open(path); hbr = new HFileBlock.FSReaderV2(is, algo, totalSize); - b = hbr.readBlockData(0, 2173 + HFileBlock.HEADER_SIZE, -1, pread); + b = hbr.readBlockData(0, 2173 + HFileBlock.HEADER_SIZE_WITH_CHECKSUMS + + b.totalChecksumBytes(), -1, pread); assertEquals(blockStr, b.toString()); int wrongCompressedSize = 2172; try { b = hbr.readBlockData(0, wrongCompressedSize - + HFileBlock.HEADER_SIZE, -1, pread); + + HFileBlock.HEADER_SIZE_WITH_CHECKSUMS, -1, pread); fail("Exception expected"); } catch (IOException ex) { String expectedPrefix = "On-disk size without header provided is " @@ -351,13 +372,17 @@ public void testDataBlockEncoding() throws IOException { HFileDataBlockEncoder dataBlockEncoder = new HFileDataBlockEncoderImpl(encoding); HFileBlock.Writer hbw = new HFileBlock.Writer(algo, dataBlockEncoder, - includesMemstoreTS); + includesMemstoreTS, + HFileReaderV2.MAX_MINOR_VERSION, + HFile.DEFAULT_CHECKSUM_TYPE, + HFile.DEFAULT_BYTES_PER_CHECKSUM); long totalSize = 0; final List encodedSizes = new ArrayList(); final List encodedBlocks = new ArrayList(); for (int blockId = 0; blockId < numBlocks; ++blockId) { - writeEncodedBlock(encoding, hbw, encodedSizes, encodedBlocks, - blockId); + DataOutputStream dos = hbw.startWriting(BlockType.DATA); + writeEncodedBlock(encoding, dos, encodedSizes, encodedBlocks, + blockId, includesMemstoreTS); hbw.writeHeaderAndData(os); totalSize += hbw.getOnDiskSizeWithHeader(); @@ -374,6 +399,7 @@ public void testDataBlockEncoding() throws IOException { int pos = 0; for (int blockId = 0; blockId < numBlocks; ++blockId) { b = hbr.readBlockData(pos, -1, -1, pread); + assertEquals(0, HFile.getChecksumFailuresCount()); b.sanityCheck(); pos += b.getOnDiskSizeWithHeader(); @@ -401,16 +427,16 @@ public void testDataBlockEncoding() throws IOException { } } - private void writeEncodedBlock(DataBlockEncoding encoding, - HFileBlock.Writer hbw, final List encodedSizes, - final List encodedBlocks, int blockId) throws IOException { - DataOutputStream dos = hbw.startWriting(BlockType.DATA); + static void writeEncodedBlock(DataBlockEncoding encoding, + DataOutputStream dos, final List encodedSizes, + final List encodedBlocks, int blockId, + boolean includesMemstoreTS) throws IOException { ByteArrayOutputStream baos = new ByteArrayOutputStream(); DoubleOutputStream doubleOutputStream = new DoubleOutputStream(dos, baos); final int rawBlockSize = writeTestKeyValues(doubleOutputStream, - blockId); + blockId, includesMemstoreTS); ByteBuffer rawBuf = ByteBuffer.wrap(baos.toByteArray()); rawBuf.rewind(); @@ -434,7 +460,7 @@ private void writeEncodedBlock(DataBlockEncoding encoding, encodedBlocks.add(encodedBuf); } - private void assertBuffersEqual(ByteBuffer expectedBuffer, + static void assertBuffersEqual(ByteBuffer expectedBuffer, ByteBuffer actualBuffer, Compression.Algorithm compression, DataBlockEncoding encoding, boolean pread) { if (!actualBuffer.equals(expectedBuffer)) { @@ -471,7 +497,9 @@ public void testPreviousOffset() throws IOException { for (boolean pread : BOOLEAN_VALUES) { for (boolean cacheOnWrite : BOOLEAN_VALUES) { Random rand = defaultRandom(); - LOG.info("Compression algorithm: " + algo + ", pread=" + pread); + LOG.info("testPreviousOffset:Compression algorithm: " + algo + + ", pread=" + pread + + ", cacheOnWrite=" + cacheOnWrite); Path path = new Path(TEST_UTIL.getDataTestDir(), "prev_offset"); List expectedOffsets = new ArrayList(); List expectedPrevOffsets = new ArrayList(); @@ -488,7 +516,7 @@ public void testPreviousOffset() throws IOException { for (int i = 0; i < NUM_TEST_BLOCKS; ++i) { if (!pread) { assertEquals(is.getPos(), curOffset + (i == 0 ? 0 : - HFileBlock.HEADER_SIZE)); + HFileBlock.HEADER_SIZE_WITH_CHECKSUMS)); } assertEquals(expectedOffsets.get(i).longValue(), curOffset); @@ -522,17 +550,23 @@ public void testPreviousOffset() throws IOException { b2.getUncompressedSizeWithoutHeader()); assertEquals(b.getPrevBlockOffset(), b2.getPrevBlockOffset()); assertEquals(curOffset, b2.getOffset()); + assertEquals(b.getBytesPerChecksum(), b2.getBytesPerChecksum()); + assertEquals(b.getOnDiskDataSizeWithHeader(), + b2.getOnDiskDataSizeWithHeader()); + assertEquals(0, HFile.getChecksumFailuresCount()); curOffset += b.getOnDiskSizeWithHeader(); if (cacheOnWrite) { // In the cache-on-write mode we store uncompressed bytes so we // can compare them to what was read by the block reader. - + // b's buffer has header + data + checksum while + // expectedContents have header + data only ByteBuffer bufRead = b.getBufferWithHeader(); ByteBuffer bufExpected = expectedContents.get(i); boolean bytesAreCorrect = Bytes.compareTo(bufRead.array(), - bufRead.arrayOffset(), bufRead.limit(), + bufRead.arrayOffset(), + bufRead.limit() - b.totalChecksumBytes(), bufExpected.array(), bufExpected.arrayOffset(), bufExpected.limit()) == 0; String wrongBytesMsg = ""; @@ -541,15 +575,26 @@ public void testPreviousOffset() throws IOException { // Optimization: only construct an error message in case we // will need it. wrongBytesMsg = "Expected bytes in block #" + i + " (algo=" - + algo + ", pread=" + pread + "):\n"; + + algo + ", pread=" + pread + + ", cacheOnWrite=" + cacheOnWrite + "):\n"; wrongBytesMsg += Bytes.toStringBinary(bufExpected.array(), bufExpected.arrayOffset(), Math.min(32, bufExpected.limit())) + ", actual:\n" + Bytes.toStringBinary(bufRead.array(), bufRead.arrayOffset(), Math.min(32, bufRead.limit())); + if (detailedLogging) { + LOG.warn("expected header" + + HFileBlock.toStringHeader(bufExpected) + + "\nfound header" + + HFileBlock.toStringHeader(bufRead)); + LOG.warn("bufread offset " + bufRead.arrayOffset() + + " limit " + bufRead.limit() + + " expected offset " + bufExpected.arrayOffset() + + " limit " + bufExpected.limit()); + LOG.warn(wrongBytesMsg); + } } - assertTrue(wrongBytesMsg, bytesAreCorrect); } } @@ -672,17 +717,22 @@ private long writeBlocks(Random rand, Compression.Algorithm compressAlgo, boolean cacheOnWrite = expectedContents != null; FSDataOutputStream os = fs.create(path); HFileBlock.Writer hbw = new HFileBlock.Writer(compressAlgo, null, - includesMemstoreTS); + includesMemstoreTS, + HFileReaderV2.MAX_MINOR_VERSION, + HFile.DEFAULT_CHECKSUM_TYPE, + HFile.DEFAULT_BYTES_PER_CHECKSUM); Map prevOffsetByType = new HashMap(); long totalSize = 0; for (int i = 0; i < NUM_TEST_BLOCKS; ++i) { + long pos = os.getPos(); int blockTypeOrdinal = rand.nextInt(BlockType.values().length); if (blockTypeOrdinal == BlockType.ENCODED_DATA.ordinal()) { blockTypeOrdinal = BlockType.DATA.ordinal(); } BlockType bt = BlockType.values()[blockTypeOrdinal]; DataOutputStream dos = hbw.startWriting(bt); - for (int j = 0; j < rand.nextInt(500); ++j) { + int size = rand.nextInt(500); + for (int j = 0; j < size; ++j) { // This might compress well. dos.writeShort(i + 1); dos.writeInt(j + 1); @@ -706,9 +756,9 @@ private long writeBlocks(Random rand, Compression.Algorithm compressAlgo, expectedContents.add(hbw.getUncompressedBufferWithHeader()); if (detailedLogging) { - LOG.info("Writing block #" + i + " of type " + bt + LOG.info("Written block #" + i + " of type " + bt + ", uncompressed size " + hbw.getUncompressedSizeWithoutHeader() - + " at offset " + os.getPos()); + + " at offset " + pos); } } os.close(); @@ -727,13 +777,15 @@ public void testBlockHeapSize() { } for (int size : new int[] { 100, 256, 12345 }) { - byte[] byteArr = new byte[HFileBlock.HEADER_SIZE + size]; + byte[] byteArr = new byte[HFileBlock.HEADER_SIZE_WITH_CHECKSUMS + size]; ByteBuffer buf = ByteBuffer.wrap(byteArr, 0, size); HFileBlock block = new HFileBlock(BlockType.DATA, size, size, -1, buf, - HFileBlock.FILL_HEADER, -1, includesMemstoreTS); + HFileBlock.FILL_HEADER, -1, includesMemstoreTS, + HFileBlock.MINOR_VERSION_NO_CHECKSUM, 0, ChecksumType.NULL.getCode(), + 0); long byteBufferExpectedSize = ClassSize.align(ClassSize.estimateBase(buf.getClass(), true) - + HFileBlock.HEADER_SIZE + size); + + HFileBlock.HEADER_SIZE_WITH_CHECKSUMS + size); long hfileBlockExpectedSize = ClassSize.align(ClassSize.estimateBase(HFileBlock.class, true)); long expected = hfileBlockExpectedSize + byteBufferExpectedSize; diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java new file mode 100644 index 000000000000..f5013e201ae6 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java @@ -0,0 +1,791 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.io.hfile; + +import static org.junit.Assert.*; + +import java.io.ByteArrayOutputStream; +import java.io.DataOutputStream; +import java.io.IOException; +import java.io.OutputStream; +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.FSDataInputStream; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.fs.HFileSystem; +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.io.compress.CompressionOutputStream; +import org.apache.hadoop.io.compress.Compressor; +import org.apache.hadoop.hbase.io.hfile.HFileBlock.BlockWritable; +import org.apache.hadoop.hbase.util.ChecksumType; +import org.apache.hadoop.hbase.util.Pair; +import com.google.common.base.Preconditions; + +import static org.apache.hadoop.hbase.io.hfile.Compression.Algorithm.*; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.junit.runners.Parameterized.Parameters; + +/** + * This class has unit tests to prove that older versions of + * HFiles (without checksums) are compatible with current readers. + */ +@Category(MediumTests.class) +@RunWith(Parameterized.class) +public class TestHFileBlockCompatibility { + // change this value to activate more logs + private static final boolean[] BOOLEAN_VALUES = new boolean[] { false, true }; + + private static final Log LOG = LogFactory.getLog(TestHFileBlockCompatibility.class); + + private static final Compression.Algorithm[] COMPRESSION_ALGORITHMS = { + NONE, GZ }; + + // The mnior version for pre-checksum files + private static int MINOR_VERSION = 0; + + private static final HBaseTestingUtility TEST_UTIL = + new HBaseTestingUtility(); + private HFileSystem fs; + private int uncompressedSizeV1; + + private final boolean includesMemstoreTS; + + public TestHFileBlockCompatibility(boolean includesMemstoreTS) { + this.includesMemstoreTS = includesMemstoreTS; + } + + @Parameters + public static Collection parameters() { + return HBaseTestingUtility.BOOLEAN_PARAMETERIZED; + } + + @Before + public void setUp() throws IOException { + fs = (HFileSystem)HFileSystem.get(TEST_UTIL.getConfiguration()); + } + + public byte[] createTestV1Block(Compression.Algorithm algo) + throws IOException { + Compressor compressor = algo.getCompressor(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + OutputStream os = algo.createCompressionStream(baos, compressor, 0); + DataOutputStream dos = new DataOutputStream(os); + BlockType.META.write(dos); // Let's make this a meta block. + TestHFileBlock.writeTestBlockContents(dos); + uncompressedSizeV1 = dos.size(); + dos.flush(); + algo.returnCompressor(compressor); + return baos.toByteArray(); + } + + private Writer createTestV2Block(Compression.Algorithm algo) + throws IOException { + final BlockType blockType = BlockType.DATA; + Writer hbw = new Writer(algo, null, + includesMemstoreTS); + DataOutputStream dos = hbw.startWriting(blockType); + TestHFileBlock.writeTestBlockContents(dos); + byte[] headerAndData = hbw.getHeaderAndData(); + assertEquals(1000 * 4, hbw.getUncompressedSizeWithoutHeader()); + hbw.releaseCompressor(); + return hbw; + } + + private String createTestBlockStr(Compression.Algorithm algo, + int correctLength) throws IOException { + Writer hbw = createTestV2Block(algo); + byte[] testV2Block = hbw.getHeaderAndData(); + int osOffset = HFileBlock.HEADER_SIZE_NO_CHECKSUM + 9; + if (testV2Block.length == correctLength) { + // Force-set the "OS" field of the gzip header to 3 (Unix) to avoid + // variations across operating systems. + // See http://www.gzip.org/zlib/rfc-gzip.html for gzip format. + testV2Block[osOffset] = 3; + } + return Bytes.toStringBinary(testV2Block); + } + + @Test + public void testNoCompression() throws IOException { + assertEquals(4000, createTestV2Block(NONE).getBlockForCaching(). + getUncompressedSizeWithoutHeader()); + } + + @Test + public void testGzipCompression() throws IOException { + final String correctTestBlockStr = + "DATABLK*\\x00\\x00\\x00:\\x00\\x00\\x0F\\xA0\\xFF\\xFF\\xFF\\xFF" + + "\\xFF\\xFF\\xFF\\xFF" + // gzip-compressed block: http://www.gzip.org/zlib/rfc-gzip.html + + "\\x1F\\x8B" // gzip magic signature + + "\\x08" // Compression method: 8 = "deflate" + + "\\x00" // Flags + + "\\x00\\x00\\x00\\x00" // mtime + + "\\x00" // XFL (extra flags) + // OS (0 = FAT filesystems, 3 = Unix). However, this field + // sometimes gets set to 0 on Linux and Mac, so we reset it to 3. + + "\\x03" + + "\\xED\\xC3\\xC1\\x11\\x00 \\x08\\xC00DD\\xDD\\x7Fa" + + "\\xD6\\xE8\\xA3\\xB9K\\x84`\\x96Q\\xD3\\xA8\\xDB\\xA8e\\xD4c" + + "\\xD46\\xEA5\\xEA3\\xEA7\\xE7\\x00LI\\x5Cs\\xA0\\x0F\\x00\\x00"; + final int correctGzipBlockLength = 82; + assertEquals(correctTestBlockStr, createTestBlockStr(GZ, + correctGzipBlockLength)); + } + + @Test + public void testReaderV1() throws IOException { + for (Compression.Algorithm algo : COMPRESSION_ALGORITHMS) { + for (boolean pread : new boolean[] { false, true }) { + byte[] block = createTestV1Block(algo); + Path path = new Path(TEST_UTIL.getDataTestDir(), + "blocks_v1_"+ algo); + LOG.info("Creating temporary file at " + path); + FSDataOutputStream os = fs.create(path); + int totalSize = 0; + int numBlocks = 50; + for (int i = 0; i < numBlocks; ++i) { + os.write(block); + totalSize += block.length; + } + os.close(); + + FSDataInputStream is = fs.open(path); + HFileBlock.FSReader hbr = new HFileBlock.FSReaderV1(is, algo, + totalSize); + HFileBlock b; + int numBlocksRead = 0; + long pos = 0; + while (pos < totalSize) { + b = hbr.readBlockData(pos, block.length, uncompressedSizeV1, pread); + b.sanityCheck(); + pos += block.length; + numBlocksRead++; + } + assertEquals(numBlocks, numBlocksRead); + is.close(); + } + } + } + + @Test + public void testReaderV2() throws IOException { + for (Compression.Algorithm algo : COMPRESSION_ALGORITHMS) { + for (boolean pread : new boolean[] { false, true }) { + LOG.info("testReaderV2: Compression algorithm: " + algo + + ", pread=" + pread); + Path path = new Path(TEST_UTIL.getDataTestDir(), "blocks_v2_" + + algo); + FSDataOutputStream os = fs.create(path); + Writer hbw = new Writer(algo, null, + includesMemstoreTS); + long totalSize = 0; + for (int blockId = 0; blockId < 2; ++blockId) { + DataOutputStream dos = hbw.startWriting(BlockType.DATA); + for (int i = 0; i < 1234; ++i) + dos.writeInt(i); + hbw.writeHeaderAndData(os); + totalSize += hbw.getOnDiskSizeWithHeader(); + } + os.close(); + + FSDataInputStream is = fs.open(path); + HFileBlock.FSReader hbr = new HFileBlock.FSReaderV2(is, is, algo, + totalSize, MINOR_VERSION, fs, path); + HFileBlock b = hbr.readBlockData(0, -1, -1, pread); + is.close(); + + b.sanityCheck(); + assertEquals(4936, b.getUncompressedSizeWithoutHeader()); + assertEquals(algo == GZ ? 2173 : 4936, + b.getOnDiskSizeWithoutHeader() - b.totalChecksumBytes()); + String blockStr = b.toString(); + + if (algo == GZ) { + is = fs.open(path); + hbr = new HFileBlock.FSReaderV2(is, is, algo, totalSize, MINOR_VERSION, + fs, path); + b = hbr.readBlockData(0, 2173 + HFileBlock.HEADER_SIZE_NO_CHECKSUM + + b.totalChecksumBytes(), -1, pread); + assertEquals(blockStr, b.toString()); + int wrongCompressedSize = 2172; + try { + b = hbr.readBlockData(0, wrongCompressedSize + + HFileBlock.HEADER_SIZE_NO_CHECKSUM, -1, pread); + fail("Exception expected"); + } catch (IOException ex) { + String expectedPrefix = "On-disk size without header provided is " + + wrongCompressedSize + ", but block header contains " + + b.getOnDiskSizeWithoutHeader() + "."; + assertTrue("Invalid exception message: '" + ex.getMessage() + + "'.\nMessage is expected to start with: '" + expectedPrefix + + "'", ex.getMessage().startsWith(expectedPrefix)); + } + is.close(); + } + } + } + } + + /** + * Test encoding/decoding data blocks. + * @throws IOException a bug or a problem with temporary files. + */ + @Test + public void testDataBlockEncoding() throws IOException { + final int numBlocks = 5; + for (Compression.Algorithm algo : COMPRESSION_ALGORITHMS) { + for (boolean pread : new boolean[] { false, true }) { + for (DataBlockEncoding encoding : DataBlockEncoding.values()) { + LOG.info("testDataBlockEncoding algo " + algo + + " pread = " + pread + + " encoding " + encoding); + Path path = new Path(TEST_UTIL.getDataTestDir(), "blocks_v2_" + + algo + "_" + encoding.toString()); + FSDataOutputStream os = fs.create(path); + HFileDataBlockEncoder dataBlockEncoder = + new HFileDataBlockEncoderImpl(encoding); + Writer hbw = new Writer(algo, dataBlockEncoder, + includesMemstoreTS); + long totalSize = 0; + final List encodedSizes = new ArrayList(); + final List encodedBlocks = new ArrayList(); + for (int blockId = 0; blockId < numBlocks; ++blockId) { + DataOutputStream dos = hbw.startWriting(BlockType.DATA); + TestHFileBlock.writeEncodedBlock(encoding, dos, encodedSizes, encodedBlocks, + blockId, includesMemstoreTS); + + hbw.writeHeaderAndData(os); + totalSize += hbw.getOnDiskSizeWithHeader(); + } + os.close(); + + FSDataInputStream is = fs.open(path); + HFileBlock.FSReaderV2 hbr = new HFileBlock.FSReaderV2(is, is, algo, + totalSize, MINOR_VERSION, fs, path); + hbr.setDataBlockEncoder(dataBlockEncoder); + hbr.setIncludesMemstoreTS(includesMemstoreTS); + + HFileBlock b; + int pos = 0; + for (int blockId = 0; blockId < numBlocks; ++blockId) { + b = hbr.readBlockData(pos, -1, -1, pread); + b.sanityCheck(); + pos += b.getOnDiskSizeWithHeader(); + + assertEquals((int) encodedSizes.get(blockId), + b.getUncompressedSizeWithoutHeader()); + ByteBuffer actualBuffer = b.getBufferWithoutHeader(); + if (encoding != DataBlockEncoding.NONE) { + // We expect a two-byte big-endian encoding id. + assertEquals(0, actualBuffer.get(0)); + assertEquals(encoding.getId(), actualBuffer.get(1)); + actualBuffer.position(2); + actualBuffer = actualBuffer.slice(); + } + + ByteBuffer expectedBuffer = encodedBlocks.get(blockId); + expectedBuffer.rewind(); + + // test if content matches, produce nice message + TestHFileBlock.assertBuffersEqual(expectedBuffer, actualBuffer, algo, encoding, + pread); + } + is.close(); + } + } + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); + + + /** + * This is the version of the HFileBlock.Writer that is used to + * create V2 blocks with minor version 0. These blocks do not + * have hbase-level checksums. The code is here to test + * backward compatibility. The reason we do not inherit from + * HFileBlock.Writer is because we never ever want to change the code + * in this class but the code in HFileBlock.Writer will continually + * evolve. + */ + public static final class Writer { + + // These constants are as they were in minorVersion 0. + private static final int HEADER_SIZE = HFileBlock.HEADER_SIZE_NO_CHECKSUM; + private static final boolean DONT_FILL_HEADER = HFileBlock.DONT_FILL_HEADER; + private static final byte[] DUMMY_HEADER = + HFileBlock.DUMMY_HEADER_NO_CHECKSUM; + + private enum State { + INIT, + WRITING, + BLOCK_READY + }; + + /** Writer state. Used to ensure the correct usage protocol. */ + private State state = State.INIT; + + /** Compression algorithm for all blocks this instance writes. */ + private final Compression.Algorithm compressAlgo; + + /** Data block encoder used for data blocks */ + private final HFileDataBlockEncoder dataBlockEncoder; + + /** + * The stream we use to accumulate data in uncompressed format for each + * block. We reset this stream at the end of each block and reuse it. The + * header is written as the first {@link #HEADER_SIZE} bytes into this + * stream. + */ + private ByteArrayOutputStream baosInMemory; + + /** Compressor, which is also reused between consecutive blocks. */ + private Compressor compressor; + + /** Compression output stream */ + private CompressionOutputStream compressionStream; + + /** Underlying stream to write compressed bytes to */ + private ByteArrayOutputStream compressedByteStream; + + /** + * Current block type. Set in {@link #startWriting(BlockType)}. Could be + * changed in {@link #encodeDataBlockForDisk()} from {@link BlockType#DATA} + * to {@link BlockType#ENCODED_DATA}. + */ + private BlockType blockType; + + /** + * A stream that we write uncompressed bytes to, which compresses them and + * writes them to {@link #baosInMemory}. + */ + private DataOutputStream userDataStream; + + /** + * Bytes to be written to the file system, including the header. Compressed + * if compression is turned on. + */ + private byte[] onDiskBytesWithHeader; + + /** + * Valid in the READY state. Contains the header and the uncompressed (but + * potentially encoded, if this is a data block) bytes, so the length is + * {@link #uncompressedSizeWithoutHeader} + {@link HFileBlock#HEADER_SIZE_WITH_CHECKSUMS}. + */ + private byte[] uncompressedBytesWithHeader; + + /** + * Current block's start offset in the {@link HFile}. Set in + * {@link #writeHeaderAndData(FSDataOutputStream)}. + */ + private long startOffset; + + /** + * Offset of previous block by block type. Updated when the next block is + * started. + */ + private long[] prevOffsetByType; + + /** The offset of the previous block of the same type */ + private long prevOffset; + + /** Whether we are including memstore timestamp after every key/value */ + private boolean includesMemstoreTS; + + /** + * @param compressionAlgorithm compression algorithm to use + * @param dataBlockEncoderAlgo data block encoding algorithm to use + */ + public Writer(Compression.Algorithm compressionAlgorithm, + HFileDataBlockEncoder dataBlockEncoder, boolean includesMemstoreTS) { + compressAlgo = compressionAlgorithm == null ? NONE : compressionAlgorithm; + this.dataBlockEncoder = dataBlockEncoder != null + ? dataBlockEncoder : NoOpDataBlockEncoder.INSTANCE; + + baosInMemory = new ByteArrayOutputStream(); + if (compressAlgo != NONE) { + compressor = compressionAlgorithm.getCompressor(); + compressedByteStream = new ByteArrayOutputStream(); + try { + compressionStream = + compressionAlgorithm.createPlainCompressionStream( + compressedByteStream, compressor); + } catch (IOException e) { + throw new RuntimeException("Could not create compression stream " + + "for algorithm " + compressionAlgorithm, e); + } + } + + prevOffsetByType = new long[BlockType.values().length]; + for (int i = 0; i < prevOffsetByType.length; ++i) + prevOffsetByType[i] = -1; + + this.includesMemstoreTS = includesMemstoreTS; + } + + /** + * Starts writing into the block. The previous block's data is discarded. + * + * @return the stream the user can write their data into + * @throws IOException + */ + public DataOutputStream startWriting(BlockType newBlockType) + throws IOException { + if (state == State.BLOCK_READY && startOffset != -1) { + // We had a previous block that was written to a stream at a specific + // offset. Save that offset as the last offset of a block of that type. + prevOffsetByType[blockType.getId()] = startOffset; + } + + startOffset = -1; + blockType = newBlockType; + + baosInMemory.reset(); + baosInMemory.write(DUMMY_HEADER); + + state = State.WRITING; + + // We will compress it later in finishBlock() + userDataStream = new DataOutputStream(baosInMemory); + return userDataStream; + } + + /** + * Returns the stream for the user to write to. The block writer takes care + * of handling compression and buffering for caching on write. Can only be + * called in the "writing" state. + * + * @return the data output stream for the user to write to + */ + DataOutputStream getUserDataStream() { + expectState(State.WRITING); + return userDataStream; + } + + /** + * Transitions the block writer from the "writing" state to the "block + * ready" state. Does nothing if a block is already finished. + */ + private void ensureBlockReady() throws IOException { + Preconditions.checkState(state != State.INIT, + "Unexpected state: " + state); + + if (state == State.BLOCK_READY) + return; + + // This will set state to BLOCK_READY. + finishBlock(); + } + + /** + * An internal method that flushes the compressing stream (if using + * compression), serializes the header, and takes care of the separate + * uncompressed stream for caching on write, if applicable. Sets block + * write state to "block ready". + */ + private void finishBlock() throws IOException { + userDataStream.flush(); + + // This does an array copy, so it is safe to cache this byte array. + uncompressedBytesWithHeader = baosInMemory.toByteArray(); + LOG.warn("Writer.finishBlock user data size with header before compression " + + uncompressedBytesWithHeader.length); + prevOffset = prevOffsetByType[blockType.getId()]; + + // We need to set state before we can package the block up for + // cache-on-write. In a way, the block is ready, but not yet encoded or + // compressed. + state = State.BLOCK_READY; + encodeDataBlockForDisk(); + + doCompression(); + putHeader(uncompressedBytesWithHeader, 0, onDiskBytesWithHeader.length, + uncompressedBytesWithHeader.length); + } + + /** + * Do compression if it is enabled, or re-use the uncompressed buffer if + * it is not. Fills in the compressed block's header if doing compression. + */ + private void doCompression() throws IOException { + // do the compression + if (compressAlgo != NONE) { + compressedByteStream.reset(); + compressedByteStream.write(DUMMY_HEADER); + + compressionStream.resetState(); + + compressionStream.write(uncompressedBytesWithHeader, HEADER_SIZE, + uncompressedBytesWithHeader.length - HEADER_SIZE); + + compressionStream.flush(); + compressionStream.finish(); + + onDiskBytesWithHeader = compressedByteStream.toByteArray(); + putHeader(onDiskBytesWithHeader, 0, onDiskBytesWithHeader.length, + uncompressedBytesWithHeader.length); + } else { + onDiskBytesWithHeader = uncompressedBytesWithHeader; + } + } + + /** + * Encodes this block if it is a data block and encoding is turned on in + * {@link #dataBlockEncoder}. + */ + private void encodeDataBlockForDisk() throws IOException { + if (blockType != BlockType.DATA) { + return; // skip any non-data block + } + + // do data block encoding, if data block encoder is set + ByteBuffer rawKeyValues = ByteBuffer.wrap(uncompressedBytesWithHeader, + HEADER_SIZE, uncompressedBytesWithHeader.length - + HEADER_SIZE).slice(); + Pair encodingResult = + dataBlockEncoder.beforeWriteToDisk(rawKeyValues, + includesMemstoreTS, DUMMY_HEADER); + + BlockType encodedBlockType = encodingResult.getSecond(); + if (encodedBlockType == BlockType.ENCODED_DATA) { + uncompressedBytesWithHeader = encodingResult.getFirst().array(); + blockType = BlockType.ENCODED_DATA; + } else { + // There is no encoding configured. Do some extra sanity-checking. + if (encodedBlockType != BlockType.DATA) { + throw new IOException("Unexpected block type coming out of data " + + "block encoder: " + encodedBlockType); + } + if (userDataStream.size() != + uncompressedBytesWithHeader.length - HEADER_SIZE) { + throw new IOException("Uncompressed size mismatch: " + + userDataStream.size() + " vs. " + + (uncompressedBytesWithHeader.length - HEADER_SIZE)); + } + } + } + + /** + * Put the header into the given byte array at the given offset. + * @param onDiskSize size of the block on disk + * @param uncompressedSize size of the block after decompression (but + * before optional data block decoding) + */ + private void putHeader(byte[] dest, int offset, int onDiskSize, + int uncompressedSize) { + offset = blockType.put(dest, offset); + offset = Bytes.putInt(dest, offset, onDiskSize - HEADER_SIZE); + offset = Bytes.putInt(dest, offset, uncompressedSize - HEADER_SIZE); + Bytes.putLong(dest, offset, prevOffset); + } + + /** + * Similar to {@link #writeHeaderAndData(FSDataOutputStream)}, but records + * the offset of this block so that it can be referenced in the next block + * of the same type. + * + * @param out + * @throws IOException + */ + public void writeHeaderAndData(FSDataOutputStream out) throws IOException { + long offset = out.getPos(); + if (startOffset != -1 && offset != startOffset) { + throw new IOException("A " + blockType + " block written to a " + + "stream twice, first at offset " + startOffset + ", then at " + + offset); + } + startOffset = offset; + + writeHeaderAndData((DataOutputStream) out); + } + + /** + * Writes the header and the compressed data of this block (or uncompressed + * data when not using compression) into the given stream. Can be called in + * the "writing" state or in the "block ready" state. If called in the + * "writing" state, transitions the writer to the "block ready" state. + * + * @param out the output stream to write the + * @throws IOException + */ + private void writeHeaderAndData(DataOutputStream out) throws IOException { + ensureBlockReady(); + out.write(onDiskBytesWithHeader); + } + + /** + * Returns the header or the compressed data (or uncompressed data when not + * using compression) as a byte array. Can be called in the "writing" state + * or in the "block ready" state. If called in the "writing" state, + * transitions the writer to the "block ready" state. + * + * @return header and data as they would be stored on disk in a byte array + * @throws IOException + */ + public byte[] getHeaderAndData() throws IOException { + ensureBlockReady(); + return onDiskBytesWithHeader; + } + + /** + * Releases the compressor this writer uses to compress blocks into the + * compressor pool. Needs to be called before the writer is discarded. + */ + public void releaseCompressor() { + if (compressor != null) { + compressAlgo.returnCompressor(compressor); + compressor = null; + } + } + + /** + * Returns the on-disk size of the data portion of the block. This is the + * compressed size if compression is enabled. Can only be called in the + * "block ready" state. Header is not compressed, and its size is not + * included in the return value. + * + * @return the on-disk size of the block, not including the header. + */ + public int getOnDiskSizeWithoutHeader() { + expectState(State.BLOCK_READY); + return onDiskBytesWithHeader.length - HEADER_SIZE; + } + + /** + * Returns the on-disk size of the block. Can only be called in the + * "block ready" state. + * + * @return the on-disk size of the block ready to be written, including the + * header size + */ + public int getOnDiskSizeWithHeader() { + expectState(State.BLOCK_READY); + return onDiskBytesWithHeader.length; + } + + /** + * The uncompressed size of the block data. Does not include header size. + */ + public int getUncompressedSizeWithoutHeader() { + expectState(State.BLOCK_READY); + return uncompressedBytesWithHeader.length - HEADER_SIZE; + } + + /** + * The uncompressed size of the block data, including header size. + */ + public int getUncompressedSizeWithHeader() { + expectState(State.BLOCK_READY); + return uncompressedBytesWithHeader.length; + } + + /** @return true if a block is being written */ + public boolean isWriting() { + return state == State.WRITING; + } + + /** + * Returns the number of bytes written into the current block so far, or + * zero if not writing the block at the moment. Note that this will return + * zero in the "block ready" state as well. + * + * @return the number of bytes written + */ + public int blockSizeWritten() { + if (state != State.WRITING) + return 0; + return userDataStream.size(); + } + + /** + * Returns the header followed by the uncompressed data, even if using + * compression. This is needed for storing uncompressed blocks in the block + * cache. Can be called in the "writing" state or the "block ready" state. + * + * @return uncompressed block bytes for caching on write + */ + private byte[] getUncompressedDataWithHeader() { + expectState(State.BLOCK_READY); + + return uncompressedBytesWithHeader; + } + + private void expectState(State expectedState) { + if (state != expectedState) { + throw new IllegalStateException("Expected state: " + expectedState + + ", actual state: " + state); + } + } + + /** + * Similar to {@link #getUncompressedBufferWithHeader()} but returns a byte + * buffer. + * + * @return uncompressed block for caching on write in the form of a buffer + */ + public ByteBuffer getUncompressedBufferWithHeader() { + byte[] b = getUncompressedDataWithHeader(); + return ByteBuffer.wrap(b, 0, b.length); + } + + /** + * Takes the given {@link BlockWritable} instance, creates a new block of + * its appropriate type, writes the writable into this block, and flushes + * the block into the output stream. The writer is instructed not to buffer + * uncompressed bytes for cache-on-write. + * + * @param bw the block-writable object to write as a block + * @param out the file system output stream + * @throws IOException + */ + public void writeBlock(BlockWritable bw, FSDataOutputStream out) + throws IOException { + bw.writeToBlock(startWriting(bw.getBlockType())); + writeHeaderAndData(out); + } + + /** + * Creates a new HFileBlock. + */ + public HFileBlock getBlockForCaching() { + return new HFileBlock(blockType, getOnDiskSizeWithoutHeader(), + getUncompressedSizeWithoutHeader(), prevOffset, + getUncompressedBufferWithHeader(), DONT_FILL_HEADER, startOffset, + includesMemstoreTS, MINOR_VERSION, 0, ChecksumType.NULL.getCode(), + getOnDiskSizeWithoutHeader()); + } + } +} + diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java index b7d0665f225f..22c9b696c669 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -44,6 +43,7 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.fs.HFileSystem; import org.apache.hadoop.hbase.io.hfile.HFileBlockIndex.BlockIndexReader; import org.apache.hadoop.hbase.io.hfile.HFileBlockIndex.BlockIndexChunk; import org.apache.hadoop.hbase.util.Bytes; @@ -110,7 +110,7 @@ public void setUp() throws IOException { // This test requires at least HFile format version 2. conf.setInt(HFile.FORMAT_VERSION_KEY, HFile.MAX_FORMAT_VERSION); - fs = FileSystem.get(conf); + fs = HFileSystem.get(conf); } @Test @@ -175,7 +175,7 @@ public void readIndex() throws IOException { Bytes.BYTES_RAWCOMPARATOR, numLevels, brw); indexReader.readRootIndex(blockReader.blockRange(rootIndexOffset, - fileSize).nextBlockAsStream(BlockType.ROOT_INDEX), numRootEntries); + fileSize).nextBlockWithBlockType(BlockType.ROOT_INDEX), numRootEntries); long prevOffset = -1; int i = 0; @@ -215,7 +215,10 @@ public void readIndex() throws IOException { private void writeWholeIndex() throws IOException { assertEquals(0, keys.size()); HFileBlock.Writer hbw = new HFileBlock.Writer(compr, null, - includesMemstoreTS); + includesMemstoreTS, + 1, + HFile.DEFAULT_CHECKSUM_TYPE, + HFile.DEFAULT_BYTES_PER_CHECKSUM); FSDataOutputStream outputStream = fs.create(path); HFileBlockIndex.BlockIndexWriter biw = new HFileBlockIndex.BlockIndexWriter(hbw, null, null); @@ -378,8 +381,8 @@ public void testSecondaryIndexBinarySearch() throws IOException { // Now test we can get the offset and the on-disk-size using a // higher-level API function.s boolean locateBlockResult = - BlockIndexReader.locateNonRootIndexEntry(nonRootIndex, arrayHoldingKey, - searchKey.length / 2, searchKey.length, Bytes.BYTES_RAWCOMPARATOR); + (BlockIndexReader.locateNonRootIndexEntry(nonRootIndex, arrayHoldingKey, + searchKey.length / 2, searchKey.length, Bytes.BYTES_RAWCOMPARATOR) != -1); if (i == 0) { assertFalse(locateBlockResult); diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java index e1a57e52d225..0c076f267ed6 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.io.encoding.RedundantKVGenerator; import org.apache.hadoop.hbase.regionserver.metrics.SchemaConfigured; import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; +import org.apache.hadoop.hbase.util.ChecksumType; import org.apache.hadoop.hbase.util.Pair; import org.junit.After; import org.junit.Before; @@ -92,12 +93,12 @@ public void tearDown() throws IOException { public void testEncodingWithCache() { HFileBlock block = getSampleHFileBlock(); LruBlockCache blockCache = - new LruBlockCache(8 * 1024 * 1024, 32 * 1024); + new LruBlockCache(8 * 1024 * 1024, 32 * 1024, TEST_UTIL.getConfiguration()); HFileBlock cacheBlock = blockEncoder.diskToCacheFormat(block, false); BlockCacheKey cacheKey = new BlockCacheKey("test", 0); blockCache.cacheBlock(cacheKey, cacheBlock); - HeapSize heapSize = blockCache.getBlock(cacheKey, false); + HeapSize heapSize = blockCache.getBlock(cacheKey, false, false); assertTrue(heapSize instanceof HFileBlock); HFileBlock returnedBlock = (HFileBlock) heapSize;; @@ -123,12 +124,14 @@ public void testEncodingWritePath() { HFileBlock block = getSampleHFileBlock(); Pair result = blockEncoder.beforeWriteToDisk(block.getBufferWithoutHeader(), - includesMemstoreTS); + includesMemstoreTS, HFileBlock.DUMMY_HEADER_WITH_CHECKSUM); - int size = result.getFirst().limit() - HFileBlock.HEADER_SIZE; + int size = result.getFirst().limit() - HFileBlock.HEADER_SIZE_WITH_CHECKSUMS; HFileBlock blockOnDisk = new HFileBlock(result.getSecond(), size, size, -1, result.getFirst(), HFileBlock.FILL_HEADER, 0, - includesMemstoreTS); + includesMemstoreTS, block.getMinorVersion(), + block.getBytesPerChecksum(), block.getChecksumType(), + block.getOnDiskDataSizeWithHeader()); if (blockEncoder.getEncodingOnDisk() != DataBlockEncoding.NONE) { @@ -153,12 +156,13 @@ private HFileBlock getSampleHFileBlock() { ByteBuffer keyValues = RedundantKVGenerator.convertKvToByteBuffer( generator.generateTestKeyValues(60), includesMemstoreTS); int size = keyValues.limit(); - ByteBuffer buf = ByteBuffer.allocate(size + HFileBlock.HEADER_SIZE); - buf.position(HFileBlock.HEADER_SIZE); + ByteBuffer buf = ByteBuffer.allocate(size + HFileBlock.HEADER_SIZE_WITH_CHECKSUMS); + buf.position(HFileBlock.HEADER_SIZE_WITH_CHECKSUMS); keyValues.rewind(); buf.put(keyValues); HFileBlock b = new HFileBlock(BlockType.DATA, size, size, -1, buf, - HFileBlock.FILL_HEADER, 0, includesMemstoreTS); + HFileBlock.FILL_HEADER, 0, includesMemstoreTS, + HFileReaderV2.MAX_MINOR_VERSION, 0, ChecksumType.NULL.getCode(), 0); UNKNOWN_TABLE_AND_CF.passSchemaMetricsTo(b); return b; } diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java new file mode 100644 index 000000000000..419fc0b39cb2 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations + * under the License. + */ +package org.apache.hadoop.hbase.io.hfile; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.junit.experimental.categories.Category; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.Test; + +/** + * Test a case when an inline index chunk is converted to a root one. This reproduces the bug in + * HBASE-6871. We write a carefully selected number of relatively large keys so that we accumulate + * a leaf index chunk that only goes over the configured index chunk size after adding the last + * key/value. The bug is in that when we close the file, we convert that inline (leaf-level) chunk + * into a root chunk, but then look at the size of that root chunk, find that it is greater than + * the configured chunk size, and split it into a number of intermediate index blocks that should + * really be leaf-level blocks. If more keys were added, we would flush the leaf-level block, add + * another entry to the root-level block, and that would prevent us from upgrading the leaf-level + * chunk to the root chunk, thus not triggering the bug. + */ +@Category(SmallTests.class) +public class TestHFileInlineToRootChunkConversion { + private final HBaseTestingUtility testUtil = new HBaseTestingUtility(); + private final Configuration conf = testUtil.getConfiguration(); + + @Test + public void testWriteHFile() throws Exception { + Path hfPath = new Path(testUtil.getDataTestDir(), + TestHFileInlineToRootChunkConversion.class.getSimpleName() + ".hfile"); + int maxChunkSize = 1024; + FileSystem fs = FileSystem.get(conf); + CacheConfig cacheConf = new CacheConfig(conf); + conf.setInt(HFileBlockIndex.MAX_CHUNK_SIZE_KEY, maxChunkSize); + HFileWriterV2 hfw = + (HFileWriterV2) new HFileWriterV2.WriterFactoryV2(conf, cacheConf) + .withBlockSize(16) + .withPath(fs, hfPath).create(); + List keys = new ArrayList(); + StringBuilder sb = new StringBuilder(); + + for (int i = 0; i < 4; ++i) { + sb.append("key" + String.format("%05d", i)); + sb.append("_"); + for (int j = 0; j < 100; ++j) { + sb.append('0' + j); + } + String keyStr = sb.toString(); + sb.setLength(0); + + byte[] k = Bytes.toBytes(keyStr); + System.out.println("Key: " + Bytes.toString(k)); + keys.add(k); + byte[] v = Bytes.toBytes("value" + i); + hfw.append(k, v); + } + hfw.close(); + + HFileReaderV2 reader = (HFileReaderV2) HFile.createReader(fs, hfPath, cacheConf); + HFileScanner scanner = reader.getScanner(true, true); + for (int i = 0; i < keys.size(); ++i) { + scanner.seekTo(keys.get(i)); + } + reader.close(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java index c745ffcb9e1f..91bb456561f3 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -76,7 +75,7 @@ public void testReadingExistingVersion1HFile() throws IOException { assertEquals(N, reader.getEntries()); assertEquals(N, trailer.getEntryCount()); - assertEquals(1, trailer.getVersion()); + assertEquals(1, trailer.getMajorVersion()); assertEquals(Compression.Algorithm.GZ, trailer.getCompressionCodec()); for (boolean pread : new boolean[] { false, true }) { diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java index 8798999b2216..2160e3e5b275 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java @@ -32,6 +32,8 @@ import org.apache.commons.cli.OptionBuilder; import org.apache.commons.cli.Options; import org.apache.commons.cli.ParseException; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FSDataOutputStream; @@ -44,7 +46,6 @@ import org.apache.hadoop.hbase.io.hfile.HFile.Writer; import org.apache.hadoop.io.BytesWritable; import org.junit.experimental.categories.Category; -import org.mortbay.log.Log; /** * test the performance for seek. @@ -56,6 +57,7 @@ */ @Category(MediumTests.class) public class TestHFileSeek extends TestCase { + private static final Log LOG = LogFactory.getLog(TestHFileSeek.class); private static final boolean USE_PREAD = true; private MyOptions options; private Configuration conf; @@ -529,7 +531,7 @@ public static void main(String[] argv) throws IOException { testCase.options = options; for (int i = 0; i < options.trialCount; i++) { - Log.info("Beginning trial " + (i+1)); + LOG.info("Beginning trial " + (i+1)); testCase.setUp(); testCase.testSeeks(); testCase.tearDown(); diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java index 365d9fa471b8..b1f49bb20f1d 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -27,6 +26,7 @@ import java.io.IOException; import java.nio.ByteBuffer; import java.util.ArrayList; +import java.util.Collection; import java.util.List; import java.util.Random; @@ -37,8 +37,7 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.HBaseTestingUtility; -import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.io.hfile.Compression.Algorithm; import org.apache.hadoop.hbase.io.hfile.HFile.FileInfo; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.io.RawComparator; @@ -47,14 +46,24 @@ import org.junit.Before; import org.junit.Test; import org.junit.experimental.categories.Category; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; /** * Testing writing a version 2 {@link HFile}. This is a low-level test written * during the development of {@link HFileWriterV2}. */ @Category(SmallTests.class) +@RunWith(Parameterized.class) public class TestHFileWriterV2 { + private final boolean useChecksums; + + @Parameterized.Parameters + public static Collection parameters() { + return HBaseTestingUtility.BOOLEAN_PARAMETERIZED; + } + private static final Log LOG = LogFactory.getLog(TestHFileWriterV2.class); private static final HBaseTestingUtility TEST_UTIL = @@ -63,23 +72,44 @@ public class TestHFileWriterV2 { private Configuration conf; private FileSystem fs; + public TestHFileWriterV2(boolean useChecksums) { + this.useChecksums = useChecksums; + } + @Before public void setUp() throws IOException { conf = TEST_UTIL.getConfiguration(); + conf.setBoolean(HConstants.HBASE_CHECKSUM_VERIFICATION, useChecksums); fs = FileSystem.get(conf); } @Test public void testHFileFormatV2() throws IOException { Path hfilePath = new Path(TEST_UTIL.getDataTestDir(), - "testHFileFormatV2"); + "testHFileFormatV2"); + final Compression.Algorithm compressAlgo = Compression.Algorithm.GZ; + final int entryCount = 10000; + writeDataAndReadFromHFile(hfilePath, compressAlgo, entryCount, false); + } + + + @Test + public void testMidKeyInHFile() throws IOException{ + Path hfilePath = new Path(TEST_UTIL.getDataTestDir(), + "testMidKeyInHFile"); + Compression.Algorithm compressAlgo = Compression.Algorithm.NONE; + int entryCount = 50000; + writeDataAndReadFromHFile(hfilePath, compressAlgo, entryCount, true); + } + + private void writeDataAndReadFromHFile(Path hfilePath, + Algorithm compressAlgo, int entryCount, boolean findMidKey) throws IOException { - final Compression.Algorithm COMPRESS_ALGO = Compression.Algorithm.GZ; HFileWriterV2 writer = (HFileWriterV2) new HFileWriterV2.WriterFactoryV2(conf, new CacheConfig(conf)) .withPath(fs, hfilePath) .withBlockSize(4096) - .withCompression(COMPRESS_ALGO) + .withCompression(compressAlgo) .withComparator(KeyValue.KEY_COMPARATOR) .create(); @@ -88,11 +118,10 @@ public void testHFileFormatV2() throws IOException { Random rand = new Random(9713312); // Just a fixed seed. - final int ENTRY_COUNT = 10000; List keys = new ArrayList(); List values = new ArrayList(); - for (int i = 0; i < ENTRY_COUNT; ++i) { + for (int i = 0; i < entryCount; ++i) { byte[] keyBytes = randomOrderedKey(rand, i); // A random-length random value. @@ -113,6 +142,7 @@ public void testHFileFormatV2() throws IOException { writer.appendMetaBlock("CAPITAL_OF_FRANCE", new Text("Paris")); writer.close(); + FSDataInputStream fsdis = fs.open(hfilePath); @@ -123,11 +153,14 @@ public void testHFileFormatV2() throws IOException { FixedFileTrailer trailer = FixedFileTrailer.readFromStream(fsdis, fileSize); - assertEquals(2, trailer.getVersion()); - assertEquals(ENTRY_COUNT, trailer.getEntryCount()); + assertEquals(2, trailer.getMajorVersion()); + assertEquals(useChecksums?1:0, trailer.getMinorVersion()); + assertEquals(entryCount, trailer.getEntryCount()); HFileBlock.FSReader blockReader = - new HFileBlock.FSReaderV2(fsdis, COMPRESS_ALGO, fileSize); + new HFileBlock.FSReaderV2(fsdis,fsdis, compressAlgo, fileSize, + this.useChecksums?HFileReaderV2.MAX_MINOR_VERSION:HFileReaderV2.MIN_MINOR_VERSION, + null, null); // Comparator class name is stored in the trailer in version 2. RawComparator comparator = trailer.createComparator(); HFileBlockIndex.BlockIndexReader dataBlockIndexReader = @@ -143,16 +176,21 @@ public void testHFileFormatV2() throws IOException { // Data index. We also read statistics about the block index written after // the root level. dataBlockIndexReader.readMultiLevelIndexRoot( - blockIter.nextBlockAsStream(BlockType.ROOT_INDEX), + blockIter.nextBlockWithBlockType(BlockType.ROOT_INDEX), trailer.getDataIndexCount()); - + + if (findMidKey) { + byte[] midkey = dataBlockIndexReader.midkey(); + assertNotNull("Midkey should not be null", midkey); + } + // Meta index. metaBlockIndexReader.readRootIndex( - blockIter.nextBlockAsStream(BlockType.ROOT_INDEX), + blockIter.nextBlockWithBlockType(BlockType.ROOT_INDEX).getByteStream(), trailer.getMetaIndexCount()); // File info FileInfo fileInfo = new FileInfo(); - fileInfo.readFields(blockIter.nextBlockAsStream(BlockType.FILE_INFO)); + fileInfo.readFields(blockIter.nextBlockWithBlockType(BlockType.FILE_INFO).getByteStream()); byte [] keyValueFormatVersion = fileInfo.get( HFileWriterV2.KEY_VALUE_VERSION); boolean includeMemstoreTS = keyValueFormatVersion != null && @@ -200,7 +238,7 @@ public void testHFileFormatV2() throws IOException { } LOG.info("Finished reading: entries=" + entriesRead + ", blocksRead=" + blocksRead); - assertEquals(ENTRY_COUNT, entriesRead); + assertEquals(entryCount, entriesRead); // Meta blocks. We can scan until the load-on-open data offset (which is // the root block index offset in version 2) because we are not testing @@ -226,6 +264,7 @@ public void testHFileFormatV2() throws IOException { fsdis.close(); } + // Static stuff used by various HFile v2 unit tests private static final String COLUMN_FAMILY_NAME = "_-myColumnFamily-_"; diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java index d980948fe250..a4415265f154 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,26 +18,29 @@ */ package org.apache.hadoop.hbase.io.hfile; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + import java.nio.ByteBuffer; import java.util.Collection; import java.util.Map; import java.util.Random; +import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.io.HeapSize; +import org.apache.hadoop.hbase.io.hfile.LruBlockCache.EvictionThread; import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; import org.apache.hadoop.hbase.regionserver.metrics.TestSchemaMetrics; import org.apache.hadoop.hbase.util.ClassSize; import org.junit.After; import org.junit.Before; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; import org.junit.runners.Parameterized.Parameters; -import org.junit.experimental.categories.Category; -import static org.junit.Assert.*; - /** * Tests the concurrent LruBlockCache.

* @@ -51,6 +53,8 @@ public class TestLruBlockCache { private Map startingMetrics; + private final HBaseTestingUtility TEST_UTIL = + new HBaseTestingUtility(); public TestLruBlockCache(boolean useTableName) { SchemaMetrics.setUseTableNameInTest(useTableName); @@ -73,14 +77,21 @@ public void tearDown() throws Exception { @Test public void testBackgroundEvictionThread() throws Exception { - long maxSize = 100000; long blockSize = calculateBlockSizeDefault(maxSize, 9); // room for 9, will evict - LruBlockCache cache = new LruBlockCache(maxSize,blockSize); + LruBlockCache cache = new LruBlockCache(maxSize, blockSize, TEST_UTIL.getConfiguration()); CachedItem [] blocks = generateFixedBlocks(10, blockSize, "block"); + EvictionThread evictionThread = cache.getEvictionThread(); + assertTrue(evictionThread != null); + + // Make sure eviction thread has entered run method + while (!evictionThread.isEnteringRun()) { + Thread.sleep(1); + } + // Add all the blocks for (CachedItem block : blocks) { cache.cacheBlock(block.cacheKey, block); @@ -90,7 +101,7 @@ public void testBackgroundEvictionThread() throws Exception { int n = 0; while(cache.getEvictionCount() == 0) { Thread.sleep(200); - assertTrue(n++ < 10); + assertTrue(n++ < 20); } System.out.println("Background Evictions run: " + cache.getEvictionCount()); @@ -104,7 +115,7 @@ public void testCacheSimple() throws Exception { long maxSize = 1000000; long blockSize = calculateBlockSizeDefault(maxSize, 101); - LruBlockCache cache = new LruBlockCache(maxSize, blockSize); + LruBlockCache cache = new LruBlockCache(maxSize, blockSize, TEST_UTIL.getConfiguration()); CachedItem [] blocks = generateRandomBlocks(100, blockSize); @@ -112,7 +123,7 @@ public void testCacheSimple() throws Exception { // Confirm empty for (CachedItem block : blocks) { - assertTrue(cache.getBlock(block.cacheKey, true) == null); + assertTrue(cache.getBlock(block.cacheKey, true, false) == null); } // Add blocks @@ -126,27 +137,17 @@ public void testCacheSimple() throws Exception { // Check if all blocks are properly cached and retrieved for (CachedItem block : blocks) { - HeapSize buf = cache.getBlock(block.cacheKey, true); + HeapSize buf = cache.getBlock(block.cacheKey, true, false); assertTrue(buf != null); assertEquals(buf.heapSize(), block.heapSize()); } - // Re-add same blocks and ensure nothing has changed - for (CachedItem block : blocks) { - try { - cache.cacheBlock(block.cacheKey, block); - assertTrue("Cache should not allow re-caching a block", false); - } catch(RuntimeException re) { - // expected - } - } - // Verify correctly calculated cache heap size assertEquals(expectedCacheSize, cache.heapSize()); // Check if all blocks are properly cached and retrieved for (CachedItem block : blocks) { - HeapSize buf = cache.getBlock(block.cacheKey, true); + HeapSize buf = cache.getBlock(block.cacheKey, true, false); assertTrue(buf != null); assertEquals(buf.heapSize(), block.heapSize()); } @@ -164,7 +165,7 @@ public void testCacheEvictionSimple() throws Exception { long maxSize = 100000; long blockSize = calculateBlockSizeDefault(maxSize, 10); - LruBlockCache cache = new LruBlockCache(maxSize,blockSize,false); + LruBlockCache cache = new LruBlockCache(maxSize, blockSize, false, TEST_UTIL.getConfiguration()); CachedItem [] blocks = generateFixedBlocks(10, blockSize, "block"); @@ -191,10 +192,10 @@ public void testCacheEvictionSimple() throws Exception { (maxSize * LruBlockCache.DEFAULT_ACCEPTABLE_FACTOR)); // All blocks except block 0 and 1 should be in the cache - assertTrue(cache.getBlock(blocks[0].cacheKey, true) == null); - assertTrue(cache.getBlock(blocks[1].cacheKey, true) == null); + assertTrue(cache.getBlock(blocks[0].cacheKey, true, false) == null); + assertTrue(cache.getBlock(blocks[1].cacheKey, true, false) == null); for(int i=2;i TYPE_COUNT = new HashMap(3); + static { + TYPE_COUNT.put(BloomType.ROWCOL, 2); + TYPE_COUNT.put(BloomType.ROW, 2); + TYPE_COUNT.put(BloomType.NONE, 2); + } + + private BloomType bloomType; + private int expectedCount; + + @Parameters + public static Collection parameters() { + List params = new ArrayList(); + for (Object type : TYPE_COUNT.keySet()) { + params.add(new Object[] { type, TYPE_COUNT.get(type) }); + } + return params; + } + + public TestScannerSelectionUsingKeyRange(Object expectedType, Object expectedCount) { + bloomType = (BloomType)expectedType; + expectedCount = expectedCount; + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.cleanupTestDir(); + } + + @Test + public void testScannerSelection() throws IOException { + Configuration conf = TEST_UTIL.getConfiguration(); + conf.setInt("hbase.hstore.compactionThreshold", 10000); + HColumnDescriptor hcd = new HColumnDescriptor(FAMILY_BYTES).setBlockCacheEnabled(true) + .setBloomFilterType(bloomType); + HTableDescriptor htd = new HTableDescriptor(TABLE); + htd.addFamily(hcd); + HRegionInfo info = new HRegionInfo(Bytes.toBytes(TABLE)); + HRegion region = HRegion.createHRegion(info, TEST_UTIL.getClusterTestDir(), conf, htd); + + for (int iFile = 0; iFile < NUM_FILES; ++iFile) { + for (int iRow = 0; iRow < NUM_ROWS; ++iRow) { + Put put = new Put(Bytes.toBytes("row" + iRow)); + for (int iCol = 0; iCol < NUM_COLS_PER_ROW; ++iCol) { + put.add(FAMILY_BYTES, Bytes.toBytes("col" + iCol), + Bytes.toBytes("value" + iFile + "_" + iRow + "_" + iCol)); + } + region.put(put); + } + region.flushcache(); + } + + Scan scan = new Scan(Bytes.toBytes("aaa"), Bytes.toBytes("aaz")); + CacheConfig cacheConf = new CacheConfig(conf); + LruBlockCache cache = (LruBlockCache) cacheConf.getBlockCache(); + cache.clearCache(); + Map metricsBefore = SchemaMetrics.getMetricsSnapshot(); + SchemaMetrics.validateMetricChanges(metricsBefore); + InternalScanner scanner = region.getScanner(scan); + List results = new ArrayList(); + while (scanner.next(results)) { + } + scanner.close(); + assertEquals(0, results.size()); + Set accessedFiles = cache.getCachedFileNamesForTest(); + assertEquals(accessedFiles.size(), 0); + //assertEquals(cache.getBlockCount(), 0); + Map diffMetrics = SchemaMetrics.diffMetrics(metricsBefore, + SchemaMetrics.getMetricsSnapshot()); + SchemaMetrics schemaMetrics = SchemaMetrics.getInstance(TABLE, FAMILY); + long dataBlockRead = SchemaMetrics.getLong(diffMetrics, + schemaMetrics.getBlockMetricName(BlockCategory.DATA, false, BlockMetricType.READ_COUNT)); + assertEquals(dataBlockRead, 0); + region.close(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java index 17284a3880aa..44aa3e422d5e 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java @@ -27,6 +27,7 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HRegionInfo; @@ -38,7 +39,6 @@ import org.apache.hadoop.hbase.io.hfile.BlockType.BlockCategory; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.InternalScanner; -import org.apache.hadoop.hbase.regionserver.StoreFile.BloomType; import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.BlockMetricType; import org.apache.hadoop.hbase.util.Bytes; @@ -98,6 +98,8 @@ public TestScannerSelectionUsingTTL(int numFreshFiles, @Test public void testScannerSelection() throws IOException { + Configuration conf = TEST_UTIL.getConfiguration(); + conf.setBoolean("hbase.store.delete.expired.storefile", false); HColumnDescriptor hcd = new HColumnDescriptor(FAMILY_BYTES) .setMaxVersions(Integer.MAX_VALUE) @@ -107,7 +109,7 @@ public void testScannerSelection() throws IOException { HRegionInfo info = new HRegionInfo(Bytes.toBytes(TABLE)); HRegion region = HRegion.createHRegion(info, TEST_UTIL.getClusterTestDir(), - TEST_UTIL.getConfiguration(), htd); + conf, htd); for (int iFile = 0; iFile < totalNumFiles; ++iFile) { if (iFile == NUM_EXPIRED_FILES) { @@ -127,7 +129,7 @@ public void testScannerSelection() throws IOException { Scan scan = new Scan(); scan.setMaxVersions(Integer.MAX_VALUE); - CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration()); + CacheConfig cacheConf = new CacheConfig(conf); LruBlockCache cache = (LruBlockCache) cacheConf.getBlockCache(); cache.clearCache(); InternalScanner scanner = region.getScanner(scan); diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java index bb7d75a8f016..6dbc503ce176 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -98,6 +97,89 @@ public void testSeekBefore() throws Exception { reader.close(); } + public void testSeekBeforeWithReSeekTo() throws Exception { + Path p = makeNewFile(); + HFile.Reader reader = HFile.createReader(fs, p, new CacheConfig(conf)); + reader.loadFileInfo(); + HFileScanner scanner = reader.getScanner(false, true); + assertEquals(false, scanner.seekBefore(toKV("a").getKey())); + assertEquals(false, scanner.seekBefore(toKV("b").getKey())); + assertEquals(false, scanner.seekBefore(toKV("c").getKey())); + + // seekBefore d, so the scanner points to c + assertEquals(true, scanner.seekBefore(toKV("d").getKey())); + assertEquals("c", toRowStr(scanner.getKeyValue())); + // reseekTo e and g + assertEquals(0, scanner.reseekTo(toKV("c").getKey())); + assertEquals("c", toRowStr(scanner.getKeyValue())); + assertEquals(0, scanner.reseekTo(toKV("g").getKey())); + assertEquals("g", toRowStr(scanner.getKeyValue())); + + // seekBefore e, so the scanner points to c + assertEquals(true, scanner.seekBefore(toKV("e").getKey())); + assertEquals("c", toRowStr(scanner.getKeyValue())); + // reseekTo e and g + assertEquals(0, scanner.reseekTo(toKV("e").getKey())); + assertEquals("e", toRowStr(scanner.getKeyValue())); + assertEquals(0, scanner.reseekTo(toKV("g").getKey())); + assertEquals("g", toRowStr(scanner.getKeyValue())); + + // seekBefore f, so the scanner points to e + assertEquals(true, scanner.seekBefore(toKV("f").getKey())); + assertEquals("e", toRowStr(scanner.getKeyValue())); + // reseekTo e and g + assertEquals(0, scanner.reseekTo(toKV("e").getKey())); + assertEquals("e", toRowStr(scanner.getKeyValue())); + assertEquals(0, scanner.reseekTo(toKV("g").getKey())); + assertEquals("g", toRowStr(scanner.getKeyValue())); + + // seekBefore g, so the scanner points to e + assertEquals(true, scanner.seekBefore(toKV("g").getKey())); + assertEquals("e", toRowStr(scanner.getKeyValue())); + // reseekTo e and g again + assertEquals(0, scanner.reseekTo(toKV("e").getKey())); + assertEquals("e", toRowStr(scanner.getKeyValue())); + assertEquals(0, scanner.reseekTo(toKV("g").getKey())); + assertEquals("g", toRowStr(scanner.getKeyValue())); + + // seekBefore h, so the scanner points to g + assertEquals(true, scanner.seekBefore(toKV("h").getKey())); + assertEquals("g", toRowStr(scanner.getKeyValue())); + // reseekTo g + assertEquals(0, scanner.reseekTo(toKV("g").getKey())); + assertEquals("g", toRowStr(scanner.getKeyValue())); + + // seekBefore i, so the scanner points to g + assertEquals(true, scanner.seekBefore(toKV("i").getKey())); + assertEquals("g", toRowStr(scanner.getKeyValue())); + // reseekTo g + assertEquals(0, scanner.reseekTo(toKV("g").getKey())); + assertEquals("g", toRowStr(scanner.getKeyValue())); + + // seekBefore j, so the scanner points to i + assertEquals(true, scanner.seekBefore(toKV("j").getKey())); + assertEquals("i", toRowStr(scanner.getKeyValue())); + // reseekTo i + assertEquals(0, scanner.reseekTo(toKV("i").getKey())); + assertEquals("i", toRowStr(scanner.getKeyValue())); + + // seekBefore k, so the scanner points to i + assertEquals(true, scanner.seekBefore(toKV("k").getKey())); + assertEquals("i", toRowStr(scanner.getKeyValue())); + // reseekTo i and k + assertEquals(0, scanner.reseekTo(toKV("i").getKey())); + assertEquals("i", toRowStr(scanner.getKeyValue())); + assertEquals(0, scanner.reseekTo(toKV("k").getKey())); + assertEquals("k", toRowStr(scanner.getKeyValue())); + + // seekBefore l, so the scanner points to k + assertEquals(true, scanner.seekBefore(toKV("l").getKey())); + assertEquals("k", toRowStr(scanner.getKeyValue())); + // reseekTo k + assertEquals(0, scanner.reseekTo(toKV("k").getKey())); + assertEquals("k", toRowStr(scanner.getKeyValue())); + } + public void testSeekTo() throws Exception { Path p = makeNewFile(); HFile.Reader reader = HFile.createReader(fs, p, new CacheConfig(conf)); diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java index 4f21fbdff638..2df5bff4e982 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java index 124d131849c7..0f09b063f995 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java b/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java index a02f109515e2..577d42339aef 100644 --- a/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java +++ b/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java b/src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java index 5229b6d0500a..c6b5d6bc10bc 100644 --- a/src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java +++ b/src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -37,6 +36,7 @@ import org.apache.log4j.AppenderSkeleton; import org.apache.log4j.Logger; import org.apache.log4j.spi.LoggingEvent; +import org.apache.log4j.Level; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -69,30 +69,38 @@ private void testDelayedRpc(boolean delayReturnValue) throws Exception { rpcServer = HBaseRPC.getServer(new TestRpcImpl(delayReturnValue), new Class[]{ TestRpcImpl.class }, isa.getHostName(), isa.getPort(), 1, 0, true, conf, 0); - rpcServer.start(); + RpcEngine rpcEngine = null; + try { + rpcServer.start(); + rpcEngine = HBaseRPC.getProtocolEngine(conf); - TestRpc client = (TestRpc) HBaseRPC.getProxy(TestRpc.class, 0, + TestRpc client = rpcEngine.getProxy(TestRpc.class, 0, rpcServer.getListenerAddress(), conf, 1000); - List results = new ArrayList(); - - TestThread th1 = new TestThread(client, true, results); - TestThread th2 = new TestThread(client, false, results); - TestThread th3 = new TestThread(client, false, results); - th1.start(); - Thread.sleep(100); - th2.start(); - Thread.sleep(200); - th3.start(); - - th1.join(); - th2.join(); - th3.join(); - - assertEquals(UNDELAYED, results.get(0).intValue()); - assertEquals(UNDELAYED, results.get(1).intValue()); - assertEquals(results.get(2).intValue(), delayReturnValue ? DELAYED : - 0xDEADBEEF); + List results = new ArrayList(); + + TestThread th1 = new TestThread(client, true, results); + TestThread th2 = new TestThread(client, false, results); + TestThread th3 = new TestThread(client, false, results); + th1.start(); + Thread.sleep(100); + th2.start(); + Thread.sleep(200); + th3.start(); + + th1.join(); + th2.join(); + th3.join(); + + assertEquals(UNDELAYED, results.get(0).intValue()); + assertEquals(UNDELAYED, results.get(1).intValue()); + assertEquals(results.get(2).intValue(), delayReturnValue ? DELAYED : + 0xDEADBEEF); + } finally { + if (rpcEngine != null) { + rpcEngine.close(); + } + } } private static class ListAppender extends AppenderSkeleton { @@ -126,38 +134,48 @@ public void testTooManyDelayedRpcs() throws Exception { ListAppender listAppender = new ListAppender(); Logger log = Logger.getLogger("org.apache.hadoop.ipc.HBaseServer"); log.addAppender(listAppender); + log.setLevel(Level.WARN); InetSocketAddress isa = new InetSocketAddress("localhost", 0); rpcServer = HBaseRPC.getServer(new TestRpcImpl(true), new Class[]{ TestRpcImpl.class }, isa.getHostName(), isa.getPort(), 1, 0, true, conf, 0); - rpcServer.start(); - TestRpc client = (TestRpc) HBaseRPC.getProxy(TestRpc.class, 0, - rpcServer.getListenerAddress(), conf, 1000); + RpcEngine rpcEngine = null; + try { + rpcServer.start(); + rpcEngine = HBaseRPC.getProtocolEngine(conf); - Thread threads[] = new Thread[MAX_DELAYED_RPC + 1]; + TestRpc client = rpcEngine.getProxy(TestRpc.class, 0, + rpcServer.getListenerAddress(), conf, 1000); - for (int i = 0; i < MAX_DELAYED_RPC; i++) { - threads[i] = new TestThread(client, true, null); - threads[i].start(); - } + Thread threads[] = new Thread[MAX_DELAYED_RPC + 1]; - /* No warnings till here. */ - assertTrue(listAppender.getMessages().isEmpty()); + for (int i = 0; i < MAX_DELAYED_RPC; i++) { + threads[i] = new TestThread(client, true, null); + threads[i].start(); + } - /* This should give a warning. */ - threads[MAX_DELAYED_RPC] = new TestThread(client, true, null); - threads[MAX_DELAYED_RPC].start(); + /* No warnings till here. */ + assertTrue(listAppender.getMessages().isEmpty()); - for (int i = 0; i < MAX_DELAYED_RPC; i++) { - threads[i].join(); - } + /* This should give a warning. */ + threads[MAX_DELAYED_RPC] = new TestThread(client, true, null); + threads[MAX_DELAYED_RPC].start(); + + for (int i = 0; i < MAX_DELAYED_RPC; i++) { + threads[i].join(); + } - assertFalse(listAppender.getMessages().isEmpty()); - assertTrue(listAppender.getMessages().get(0).startsWith( - "Too many delayed calls")); + assertFalse(listAppender.getMessages().isEmpty()); + assertTrue(listAppender.getMessages().get(0).startsWith( + "Too many delayed calls")); - log.removeAppender(listAppender); + log.removeAppender(listAppender); + } finally { + if (rpcEngine != null) { + rpcEngine.close(); + } + } } public interface TestRpc extends VersionedProtocol { @@ -175,7 +193,6 @@ private static class TestRpcImpl implements TestRpc { /** * @param delayReturnValue Should the response to the delayed call be set * at the start or the end of the delay. - * @param delay Amount of milliseconds to delay the call by */ public TestRpcImpl(boolean delayReturnValue) { this.delayReturnValue = delayReturnValue; @@ -186,7 +203,7 @@ public int test(final boolean delay) { if (!delay) { return UNDELAYED; } - final Delayable call = rpcServer.getCurrentCall(); + final Delayable call = HBaseServer.getCurrentCall(); call.startDelay(delayReturnValue); new Thread() { public void run() { @@ -254,30 +271,38 @@ public void testEndDelayThrowing() throws IOException { rpcServer = HBaseRPC.getServer(new FaultyTestRpc(), new Class[]{ TestRpcImpl.class }, isa.getHostName(), isa.getPort(), 1, 0, true, conf, 0); - rpcServer.start(); + RpcEngine rpcEngine = null; + try { + rpcServer.start(); + rpcEngine = HBaseRPC.getProtocolEngine(conf); - TestRpc client = (TestRpc) HBaseRPC.getProxy(TestRpc.class, 0, - rpcServer.getListenerAddress(), conf, 1000); + TestRpc client = rpcEngine.getProxy(TestRpc.class, 0, + rpcServer.getListenerAddress(), conf, 1000); - int result = 0xDEADBEEF; + int result = 0xDEADBEEF; - try { - result = client.test(false); - } catch (Exception e) { - fail("No exception should have been thrown."); - } - assertEquals(result, UNDELAYED); + try { + result = client.test(false); + } catch (Exception e) { + fail("No exception should have been thrown."); + } + assertEquals(result, UNDELAYED); - boolean caughtException = false; - try { - result = client.test(true); - } catch(Exception e) { - // Exception thrown by server is enclosed in a RemoteException. - if (e.getCause().getMessage().startsWith( - "java.lang.Exception: Something went wrong")) - caughtException = true; + boolean caughtException = false; + try { + result = client.test(true); + } catch(Exception e) { + // Exception thrown by server is enclosed in a RemoteException. + if (e.getCause().getMessage().startsWith( + "java.lang.Exception: Something went wrong")) + caughtException = true; + } + assertTrue(caughtException); + } finally { + if (rpcEngine != null) { + rpcEngine.close(); + } } - assertTrue(caughtException); } /** @@ -288,7 +313,7 @@ private static class FaultyTestRpc implements TestRpc { public int test(boolean delay) { if (!delay) return UNDELAYED; - Delayable call = rpcServer.getCurrentCall(); + Delayable call = HBaseServer.getCurrentCall(); call.startDelay(true); try { call.endDelayThrowing(new Exception("Something went wrong")); diff --git a/src/test/java/org/apache/hadoop/hbase/ipc/TestPBOnWritableRpc.java b/src/test/java/org/apache/hadoop/hbase/ipc/TestPBOnWritableRpc.java index d5a906850d29..d0eb78bac171 100644 --- a/src/test/java/org/apache/hadoop/hbase/ipc/TestPBOnWritableRpc.java +++ b/src/test/java/org/apache/hadoop/hbase/ipc/TestPBOnWritableRpc.java @@ -25,14 +25,17 @@ import java.net.InetSocketAddress; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.io.Text; import org.apache.hadoop.io.Writable; import org.junit.Test; +import org.junit.experimental.categories.Category; import com.google.protobuf.DescriptorProtos; import com.google.protobuf.DescriptorProtos.EnumDescriptorProto; /** Unit tests to test PB-based types on WritableRpcEngine. */ +@Category(MediumTests.class) public class TestPBOnWritableRpc { private static Configuration conf = new Configuration(); @@ -71,7 +74,7 @@ public EnumDescriptorProto exchangeProto(EnumDescriptorProto arg) { } } - @Test(timeout=10000) + @Test(timeout=60000) public void testCalls() throws Exception { testCallsInternal(conf); } @@ -80,18 +83,18 @@ private void testCallsInternal(Configuration conf) throws Exception { RpcServer rpcServer = HBaseRPC.getServer(new TestImpl(), new Class[] {TestProtocol.class}, "localhost", // BindAddress is IP we got for this server. - 9999, // port number + 0, // port number 2, // number of handlers 0, // we dont use high priority handlers in master conf.getBoolean("hbase.rpc.verbose", false), conf, 0); - TestProtocol proxy = null; + RpcEngine rpcEngine = null; try { rpcServer.start(); + rpcEngine = HBaseRPC.getProtocolEngine(conf); - InetSocketAddress isa = - new InetSocketAddress("localhost", 9999); - proxy = (TestProtocol) HBaseRPC.waitForProxy( + InetSocketAddress isa = rpcServer.getListenerAddress(); + TestProtocol proxy = HBaseRPC.waitForProxy(rpcEngine, TestProtocol.class, TestProtocol.VERSION, isa, conf, -1, 8000, 8000); @@ -115,8 +118,8 @@ private void testCallsInternal(Configuration conf) throws Exception { assertNotSame(sendProto, retProto); } finally { rpcServer.stop(); - if(proxy != null) { - HBaseRPC.stopProxy(proxy); + if (rpcEngine != null) { + rpcEngine.close(); } } } diff --git a/src/test/java/org/apache/hadoop/hbase/ipc/TestProtocolExtension.java b/src/test/java/org/apache/hadoop/hbase/ipc/TestProtocolExtension.java new file mode 100644 index 000000000000..a8a0c6cc3877 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/ipc/TestProtocolExtension.java @@ -0,0 +1,106 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.ipc; + +import java.io.IOException; +import java.net.InetSocketAddress; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** Unit test for Protocol extending common interface. */ +@Category(SmallTests.class) +public class TestProtocolExtension { + private static final String ADDRESS = "0.0.0.0"; + + public static final Log LOG = + LogFactory.getLog(TestProtocolExtension.class); + + private static Configuration conf = new Configuration(); + + public interface ProtocolExtention { + void logClassName(); + } + + public interface TestProtocol extends VersionedProtocol, ProtocolExtention { + public static final long VERSION = 7L; + + void ping() throws IOException; + + // @Override // Uncomment to make the test pass + // public void logClassName(); +} + + public static class TestImpl implements TestProtocol { + public long getProtocolVersion(String protocol, long clientVersion) { + return TestProtocol.VERSION; + } + + @Override + public void ping() {} + + @Override + public void logClassName() { + LOG.info(this.getClass().getName()); + } + + @Override + public ProtocolSignature getProtocolSignature(String protocol, + long clientVersion, int clientMethodsHash) throws IOException { + return new ProtocolSignature(VERSION, null); + } + } + + @Test + public void testCalls() throws Exception { + RpcServer server = HBaseRPC.getServer(TestProtocol.class, + new TestImpl(), + new Class[]{ProtocolExtention.class}, + ADDRESS, + 6016, + 10, 10, false, + conf, 10); + RpcEngine rpcEngine = null; + try { + server.start(); + rpcEngine = HBaseRPC.getProtocolEngine(conf); + + InetSocketAddress addr = server.getListenerAddress(); + TestProtocol proxy = rpcEngine.getProxy( + TestProtocol.class, TestProtocol.VERSION, addr, conf, 10000); + + proxy.ping(); + + proxy.logClassName(); + } finally { + server.stop(); + if (rpcEngine != null) { + rpcEngine.close(); + } + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java b/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java index 5956169c20b5..5f9351988229 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java +++ b/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -195,16 +194,16 @@ static void runTestMapreduce(HTable table) throws IOException, * * @throws IOException */ - static HTable createIOEScannerTable(byte[] name) throws IOException { + static HTable createIOEScannerTable(byte[] name, final int failCnt) + throws IOException { // build up a mock scanner stuff to fail the first time Answer a = new Answer() { - boolean first = true; + int cnt = 0; @Override public ResultScanner answer(InvocationOnMock invocation) throws Throwable { // first invocation return the busted mock scanner - if (first) { - first = false; + if (cnt++ < failCnt) { // create mock ResultScanner that always fails. Scan scan = mock(Scan.class); doReturn("bogus".getBytes()).when(scan).getStartRow(); // avoid npe @@ -230,16 +229,16 @@ public ResultScanner answer(InvocationOnMock invocation) throws Throwable { * * @throws IOException */ - static HTable createDNRIOEScannerTable(byte[] name) throws IOException { + static HTable createDNRIOEScannerTable(byte[] name, final int failCnt) + throws IOException { // build up a mock scanner stuff to fail the first time Answer a = new Answer() { - boolean first = true; + int cnt = 0; @Override public ResultScanner answer(InvocationOnMock invocation) throws Throwable { // first invocation return the busted mock scanner - if (first) { - first = false; + if (cnt++ < failCnt) { // create mock ResultScanner that always fails. Scan scan = mock(Scan.class); doReturn("bogus".getBytes()).when(scan).getStartRow(); // avoid npe @@ -280,7 +279,18 @@ public void testTableRecordReader() throws IOException { */ @Test public void testTableRecordReaderScannerFail() throws IOException { - HTable htable = createIOEScannerTable("table2".getBytes()); + HTable htable = createIOEScannerTable("table2".getBytes(), 1); + runTestMapred(htable); + } + + /** + * Run test assuming Scanner IOException failure using mapred api, + * + * @throws IOException + */ + @Test(expected = IOException.class) + public void testTableRecordReaderScannerFailTwice() throws IOException { + HTable htable = createIOEScannerTable("table3".getBytes(), 2); runTestMapred(htable); } @@ -290,9 +300,21 @@ public void testTableRecordReaderScannerFail() throws IOException { * * @throws DoNotRetryIOException */ - @Test(expected = DoNotRetryIOException.class) + @Test public void testTableRecordReaderScannerTimeout() throws IOException { - HTable htable = createDNRIOEScannerTable("table3".getBytes()); + HTable htable = createDNRIOEScannerTable("table4".getBytes(), 1); + runTestMapred(htable); + } + + /** + * Run test assuming UnknownScannerException (which is a type of + * DoNotRetryIOException) using mapred api. + * + * @throws DoNotRetryIOException + */ + @Test(expected = DoNotRetryIOException.class) + public void testTableRecordReaderScannerTimeoutTwice() throws IOException { + HTable htable = createDNRIOEScannerTable("table5".getBytes(), 2); runTestMapred(htable); } @@ -318,7 +340,20 @@ public void testTableRecordReaderMapreduce() throws IOException, @Test public void testTableRecordReaderScannerFailMapreduce() throws IOException, InterruptedException { - HTable htable = createIOEScannerTable("table2-mr".getBytes()); + HTable htable = createIOEScannerTable("table2-mr".getBytes(), 1); + runTestMapreduce(htable); + } + + /** + * Run test assuming Scanner IOException failure using newer mapreduce api + * + * @throws IOException + * @throws InterruptedException + */ + @Test(expected = IOException.class) + public void testTableRecordReaderScannerFailMapreduceTwice() throws IOException, + InterruptedException { + HTable htable = createIOEScannerTable("table3-mr".getBytes(), 2); runTestMapreduce(htable); } @@ -329,10 +364,24 @@ public void testTableRecordReaderScannerFailMapreduce() throws IOException, * @throws InterruptedException * @throws DoNotRetryIOException */ - @Test(expected = DoNotRetryIOException.class) + @Test public void testTableRecordReaderScannerTimeoutMapreduce() throws IOException, InterruptedException { - HTable htable = createDNRIOEScannerTable("table3-mr".getBytes()); + HTable htable = createDNRIOEScannerTable("table4-mr".getBytes(), 1); + runTestMapreduce(htable); + } + + /** + * Run test assuming UnknownScannerException (which is a type of + * DoNotRetryIOException) using newer mapreduce api + * + * @throws InterruptedException + * @throws DoNotRetryIOException + */ + @Test(expected = DoNotRetryIOException.class) + public void testTableRecordReaderScannerTimeoutMapreduceTwice() + throws IOException, InterruptedException { + HTable htable = createDNRIOEScannerTable("table5-mr".getBytes(), 2); runTestMapreduce(htable); } diff --git a/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java b/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java index 26a6f610f096..fad6678e8d90 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java +++ b/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -21,6 +20,7 @@ import java.io.File; import java.io.IOException; +import java.util.Iterator; import java.util.Map; import java.util.NavigableMap; @@ -28,7 +28,6 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; @@ -42,11 +41,15 @@ import org.apache.hadoop.mapred.MapReduceBase; import org.apache.hadoop.mapred.OutputCollector; import org.apache.hadoop.mapred.Reporter; +import org.apache.hadoop.mapred.RunningJob; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; +import static org.junit.Assert.fail; +import static org.junit.Assert.assertTrue; + /** * Test Map/Reduce job over HBase tables. The map/reduce process we're testing * on our tables is simple - take every row in the table, reverse the value of @@ -58,7 +61,7 @@ public class TestTableMapReduce { LogFactory.getLog(TestTableMapReduce.class.getName()); private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); - static final String MULTI_REGION_TABLE_NAME = "mrtest"; + static final byte[] MULTI_REGION_TABLE_NAME = Bytes.toBytes("mrtest"); static final byte[] INPUT_FAMILY = Bytes.toBytes("contents"); static final byte[] OUTPUT_FAMILY = Bytes.toBytes("text"); @@ -69,12 +72,10 @@ public class TestTableMapReduce { @BeforeClass public static void beforeClass() throws Exception { - HTableDescriptor desc = new HTableDescriptor(MULTI_REGION_TABLE_NAME); - desc.addFamily(new HColumnDescriptor(INPUT_FAMILY)); - desc.addFamily(new HColumnDescriptor(OUTPUT_FAMILY)); UTIL.startMiniCluster(); - HBaseAdmin admin = new HBaseAdmin(UTIL.getConfiguration()); - admin.createTable(desc, HBaseTestingUtility.KEYS); + HTable table = UTIL.createTable(MULTI_REGION_TABLE_NAME, new byte[][] {INPUT_FAMILY, OUTPUT_FAMILY}); + UTIL.createMultiRegions(table, INPUT_FAMILY); + UTIL.loadTable(table, INPUT_FAMILY); UTIL.startMiniMapReduceCluster(); } @@ -150,7 +151,8 @@ private void runTestOnTable(HTable table) throws IOException { IdentityTableReduce.class, jobConf); LOG.info("Started " + Bytes.toString(table.getTableName())); - JobClient.runJob(jobConf); + RunningJob job = JobClient.runJob(jobConf); + assertTrue(job.isSuccessful()); LOG.info("After map/reduce completion"); // verify map-reduce results @@ -184,7 +186,7 @@ private void verify(String tableName) throws IOException { // continue } } - org.junit.Assert.assertTrue(verified); + assertTrue(verified); } /** @@ -199,7 +201,10 @@ private void verifyAttempt(final HTable table) throws IOException, NullPointerEx TableInputFormat.addColumns(scan, columns); ResultScanner scanner = table.getScanner(scan); try { - for (Result r : scanner) { + Iterator itr = scanner.iterator(); + assertTrue(itr.hasNext()); + while(itr.hasNext()) { + Result r = itr.next(); if (LOG.isDebugEnabled()) { if (r.size() > 2 ) { throw new IOException("Too many results, expected 2 got " + @@ -247,7 +252,7 @@ private void verifyAttempt(final HTable table) throws IOException, NullPointerEx r.getRow() + ", first value=" + first + ", second value=" + second); } - org.junit.Assert.fail(); + fail(); } } } finally { diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java new file mode 100644 index 000000000000..74327922264e --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java @@ -0,0 +1,126 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import java.io.IOException; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.mapred.JobConf; +import org.apache.hadoop.mapred.MiniMRCluster; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapreduce.JobContext; +import org.apache.hadoop.mapreduce.JobID; + +/** + * This class provides shims for HBase to interact with the Hadoop 1.0.x and the + * Hadoop 0.23.x series. + * + * NOTE: No testing done against 0.22.x, or 0.21.x. + */ +abstract public class MapreduceTestingShim { + private static MapreduceTestingShim instance; + private static Class[] emptyParam = new Class[] {}; + + static { + try { + // This class exists in hadoop 0.22+ but not in Hadoop 20.x/1.x + Class c = Class + .forName("org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl"); + instance = new MapreduceV2Shim(); + } catch (Exception e) { + instance = new MapreduceV1Shim(); + } + } + + abstract public JobContext newJobContext(Configuration jobConf) + throws IOException; + + abstract public JobConf obtainJobConf(MiniMRCluster cluster); + + public static JobContext createJobContext(Configuration jobConf) + throws IOException { + return instance.newJobContext(jobConf); + } + + public static JobConf getJobConf(MiniMRCluster cluster) { + return instance.obtainJobConf(cluster); + } + + private static class MapreduceV1Shim extends MapreduceTestingShim { + public JobContext newJobContext(Configuration jobConf) throws IOException { + // Implementing: + // return new JobContext(jobConf, new JobID()); + JobID jobId = new JobID(); + Constructor c; + try { + c = JobContext.class.getConstructor(Configuration.class, JobID.class); + return c.newInstance(jobConf, jobId); + } catch (Exception e) { + throw new IllegalStateException( + "Failed to instantiate new JobContext(jobConf, new JobID())", e); + } + } + public JobConf obtainJobConf(MiniMRCluster cluster) { + if (cluster == null) return null; + try { + Object runner = cluster.getJobTrackerRunner(); + Method meth = runner.getClass().getDeclaredMethod("getJobTracker", emptyParam); + Object tracker = meth.invoke(runner, new Object []{}); + Method m = tracker.getClass().getDeclaredMethod("getConf", emptyParam); + return (JobConf) m.invoke(tracker, new Object []{}); + } catch (NoSuchMethodException nsme) { + return null; + } catch (InvocationTargetException ite) { + return null; + } catch (IllegalAccessException iae) { + return null; + } + } + }; + + private static class MapreduceV2Shim extends MapreduceTestingShim { + public JobContext newJobContext(Configuration jobConf) { + // Implementing: + // return Job.getInstance(jobConf); + try { + Method m = Job.class.getMethod("getInstance", Configuration.class); + return (JobContext) m.invoke(null, jobConf); // static method, then arg + } catch (Exception e) { + e.printStackTrace(); + throw new IllegalStateException( + "Failed to return from Job.getInstance(jobConf)"); + } + } + public JobConf obtainJobConf(MiniMRCluster cluster) { + try { + Method meth = MiniMRCluster.class.getMethod("getJobTrackerConf", emptyParam); + return (JobConf) meth.invoke(cluster, new Object []{}); + } catch (NoSuchMethodException nsme) { + return null; + } catch (InvocationTargetException ite) { + return null; + } catch (IllegalAccessException iae) { + return null; + } + } + }; + +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/NMapInputFormat.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/NMapInputFormat.java index f4b3f65042a7..92888ed5e0f8 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/NMapInputFormat.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/NMapInputFormat.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java new file mode 100644 index 000000000000..e93d781b02bd --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java @@ -0,0 +1,148 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Basic test for the CopyTable M/R tool + */ +@Category(LargeTests.class) +public class TestCopyTable { + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static MiniHBaseCluster cluster; + + @BeforeClass + public static void beforeClass() throws Exception { + cluster = TEST_UTIL.startMiniCluster(3); + TEST_UTIL.startMiniMapReduceCluster(); + } + + @AfterClass + public static void afterClass() throws Exception { + TEST_UTIL.shutdownMiniMapReduceCluster(); + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * Simple end-to-end test + * @throws Exception + */ + @Test + public void testCopyTable() throws Exception { + final byte[] TABLENAME1 = Bytes.toBytes("testCopyTable1"); + final byte[] TABLENAME2 = Bytes.toBytes("testCopyTable2"); + final byte[] FAMILY = Bytes.toBytes("family"); + final byte[] COLUMN1 = Bytes.toBytes("c1"); + + HTable t1 = TEST_UTIL.createTable(TABLENAME1, FAMILY); + HTable t2 = TEST_UTIL.createTable(TABLENAME2, FAMILY); + + // put rows into the first table + for (int i = 0; i < 10; i++) { + Put p = new Put(Bytes.toBytes("row" + i)); + p.add(FAMILY, COLUMN1, COLUMN1); + t1.put(p); + } + + CopyTable copy = new CopyTable(TEST_UTIL.getConfiguration()); + + assertEquals( + 0, + copy.run(new String[] { "--new.name=" + Bytes.toString(TABLENAME2), + Bytes.toString(TABLENAME1) })); + + // verify the data was copied into table 2 + for (int i = 0; i < 10; i++) { + Get g = new Get(Bytes.toBytes("row" + i)); + Result r = t2.get(g); + assertEquals(1, r.size()); + assertTrue(Bytes.equals(COLUMN1, r.raw()[0].getQualifier())); + } + + t1.close(); + t2.close(); + TEST_UTIL.deleteTable(TABLENAME1); + TEST_UTIL.deleteTable(TABLENAME2); + } + + @Test + public void testStartStopRow() throws Exception { + final byte[] TABLENAME1 = Bytes.toBytes("testStartStopRow1"); + final byte[] TABLENAME2 = Bytes.toBytes("testStartStopRow2"); + final byte[] FAMILY = Bytes.toBytes("family"); + final byte[] COLUMN1 = Bytes.toBytes("c1"); + final byte[] ROW0 = Bytes.toBytes("row0"); + final byte[] ROW1 = Bytes.toBytes("row1"); + final byte[] ROW2 = Bytes.toBytes("row2"); + + HTable t1 = TEST_UTIL.createTable(TABLENAME1, FAMILY); + HTable t2 = TEST_UTIL.createTable(TABLENAME2, FAMILY); + + // put rows into the first table + Put p = new Put(ROW0); + p.add(FAMILY, COLUMN1, COLUMN1); + t1.put(p); + p = new Put(ROW1); + p.add(FAMILY, COLUMN1, COLUMN1); + t1.put(p); + p = new Put(ROW2); + p.add(FAMILY, COLUMN1, COLUMN1); + t1.put(p); + + CopyTable copy = new CopyTable(TEST_UTIL.getConfiguration()); + assertEquals( + 0, + copy.run(new String[] { "--new.name=" + Bytes.toString(TABLENAME2), "--startrow=row1", + "--stoprow=row2", Bytes.toString(TABLENAME1) })); + + // verify the data was copied into table 2 + // row1 exist, row0, row2 do not exist + Get g = new Get(ROW1); + Result r = t2.get(g); + assertEquals(1, r.size()); + assertTrue(Bytes.equals(COLUMN1, r.raw()[0].getQualifier())); + + g = new Get(ROW0); + r = t2.get(g); + assertEquals(0, r.size()); + + g = new Get(ROW2); + r = t2.get(g); + assertEquals(0, r.size()); + + t1.close(); + t2.close(); + TEST_UTIL.deleteTable(TABLENAME1); + TEST_UTIL.deleteTable(TABLENAME2); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java index a6bcbac44954..5c4ebec337fc 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -30,9 +29,9 @@ import java.lang.reflect.Constructor; import java.util.Arrays; import java.util.HashMap; -import java.util.List; import java.util.Map; import java.util.Map.Entry; +import java.util.Set; import java.util.concurrent.Callable; import java.util.Random; @@ -51,13 +50,17 @@ import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.io.ImmutableBytesWritable; +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; import org.apache.hadoop.hbase.io.hfile.CacheConfig; import org.apache.hadoop.hbase.io.hfile.Compression; +import org.apache.hadoop.hbase.io.hfile.HFileDataBlockEncoder; import org.apache.hadoop.hbase.io.hfile.Compression.Algorithm; import org.apache.hadoop.hbase.io.hfile.HFile; import org.apache.hadoop.hbase.io.hfile.HFile.Reader; import org.apache.hadoop.hbase.regionserver.Store; +import org.apache.hadoop.hbase.regionserver.StoreFile; import org.apache.hadoop.hbase.regionserver.TimeRangeTracker; +import org.apache.hadoop.hbase.regionserver.StoreFile.BloomType; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.FSUtils; import org.apache.hadoop.hbase.util.Threads; @@ -74,8 +77,6 @@ import org.junit.experimental.categories.Category; import org.mockito.Mockito; -import com.google.common.collect.Lists; - /** * Simple test for {@link KeyValueSortReducer} and {@link HFileOutputFormat}. * Sets up and runs a mapreduce job that writes hfile output. @@ -272,7 +273,7 @@ public void test_TIMERANGE() throws Exception { // verify that the file has the proper FileInfo. writer.close(context); - // the generated file lives 1 directory down from the attempt directory + // the generated file lives 1 directory down from the attempt directory // and is the only file, e.g. // _attempt__0000_r_000000_0/b/1979617994050536795 FileSystem fs = FileSystem.get(conf); @@ -341,7 +342,8 @@ public void testWritingPEData() throws Exception { @Test public void testJobConfiguration() throws Exception { - Job job = new Job(); + Job job = new Job(util.getConfiguration()); + job.setWorkingDirectory(util.getDataTestDir("testJobConfiguration")); HTable table = Mockito.mock(HTable.class); setupMockStartKeys(table); HFileOutputFormat.configureIncrementalLoad(job, table); @@ -466,6 +468,7 @@ private void runIncrementalPELoad( Configuration conf, HTable table, Path outDir) throws Exception { Job job = new Job(conf, "testLocalMRIncrementalLoad"); + job.setWorkingDirectory(util.getDataTestDir("runIncrementalPELoad")); setupRandomGeneratorMapper(job); HFileOutputFormat.configureIncrementalLoad(job, table); FileOutputFormat.setOutputPath(job, outDir); @@ -479,36 +482,40 @@ private void runIncrementalPELoad( } /** - * Test for - * {@link HFileOutputFormat#createFamilyCompressionMap(Configuration)}. Tests - * that the compression map is correctly deserialized from configuration + * Test for {@link HFileOutputFormat#configureCompression(HTable, + * Configuration)} and {@link HFileOutputFormat#createFamilyCompressionMap + * (Configuration)}. + * Tests that the compression map is correctly serialized into + * and deserialized from configuration * * @throws IOException */ @Test - public void testCreateFamilyCompressionMap() throws IOException { + public void testSerializeDeserializeFamilyCompressionMap() throws IOException { for (int numCfs = 0; numCfs <= 3; numCfs++) { Configuration conf = new Configuration(this.util.getConfiguration()); - Map familyToCompression = getMockColumnFamilies(numCfs); + Map familyToCompression = + getMockColumnFamiliesForCompression(numCfs); HTable table = Mockito.mock(HTable.class); - setupMockColumnFamilies(table, familyToCompression); + setupMockColumnFamiliesForCompression(table, familyToCompression); HFileOutputFormat.configureCompression(table, conf); // read back family specific compression setting from the configuration - Map retrievedFamilyToCompressionMap = HFileOutputFormat.createFamilyCompressionMap(conf); + Map retrievedFamilyToCompressionMap = HFileOutputFormat + .createFamilyCompressionMap(conf); // test that we have a value for all column families that matches with the // used mock values for (Entry entry : familyToCompression.entrySet()) { - assertEquals("Compression configuration incorrect for column family:" + entry.getKey(), entry.getValue() - .getName(), retrievedFamilyToCompressionMap.get(entry.getKey().getBytes())); + assertEquals("Compression configuration incorrect for column family:" + + entry.getKey(), entry.getValue(), + retrievedFamilyToCompressionMap.get(entry.getKey().getBytes())); } } } - private void setupMockColumnFamilies(HTable table, - Map familyToCompression) throws IOException - { + private void setupMockColumnFamiliesForCompression(HTable table, + Map familyToCompression) throws IOException { HTableDescriptor mockTableDescriptor = new HTableDescriptor(TABLE_NAME); for (Entry entry : familyToCompression.entrySet()) { mockTableDescriptor.addFamily(new HColumnDescriptor(entry.getKey()) @@ -520,21 +527,11 @@ private void setupMockColumnFamilies(HTable table, Mockito.doReturn(mockTableDescriptor).when(table).getTableDescriptor(); } - private void setupMockStartKeys(HTable table) throws IOException { - byte[][] mockKeys = new byte[][] { - HConstants.EMPTY_BYTE_ARRAY, - Bytes.toBytes("aaa"), - Bytes.toBytes("ggg"), - Bytes.toBytes("zzz") - }; - Mockito.doReturn(mockKeys).when(table).getStartKeys(); - } - /** * @return a map from column family names to compression algorithms for * testing column family compression. Column family names have special characters */ - private Map getMockColumnFamilies(int numCfs) { + private Map getMockColumnFamiliesForCompression (int numCfs) { Map familyToCompression = new HashMap(); // use column family names having special characters if (numCfs-- > 0) { @@ -553,29 +550,252 @@ private Map getMockColumnFamilies(int numCfs) { } /** - * Test that {@link HFileOutputFormat} RecordWriter uses compression settings - * from the column family descriptor + * Test for {@link HFileOutputFormat#configureBloomType(HTable, + * Configuration)} and {@link HFileOutputFormat#createFamilyBloomTypeMap + * (Configuration)}. + * Tests that the compression map is correctly serialized into + * and deserialized from configuration + * + * @throws IOException + */ + @Test + public void testSerializeDeserializeFamilyBloomTypeMap() throws IOException { + for (int numCfs = 0; numCfs <= 2; numCfs++) { + Configuration conf = new Configuration(this.util.getConfiguration()); + Map familyToBloomType = + getMockColumnFamiliesForBloomType(numCfs); + HTable table = Mockito.mock(HTable.class); + setupMockColumnFamiliesForBloomType(table, + familyToBloomType); + HFileOutputFormat.configureBloomType(table, conf); + + // read back family specific bloom type settings from the configuration + Map retrievedFamilyToBloomTypeMap = + HFileOutputFormat + .createFamilyBloomTypeMap(conf); + + // test that we have a value for all column families that matches with the + // used mock values + for (Entry entry : familyToBloomType.entrySet()) { + assertEquals("BloomType configuration incorrect for column family:" + + entry.getKey(), entry.getValue(), + retrievedFamilyToBloomTypeMap.get(entry.getKey().getBytes())); + } + } + } + + private void setupMockColumnFamiliesForBloomType(HTable table, + Map familyToDataBlockEncoding) throws IOException { + HTableDescriptor mockTableDescriptor = new HTableDescriptor(TABLE_NAME); + for (Entry entry : familyToDataBlockEncoding.entrySet()) { + mockTableDescriptor.addFamily(new HColumnDescriptor(entry.getKey()) + .setMaxVersions(1) + .setBloomFilterType(entry.getValue()) + .setBlockCacheEnabled(false) + .setTimeToLive(0)); + } + Mockito.doReturn(mockTableDescriptor).when(table).getTableDescriptor(); + } + + /** + * @return a map from column family names to compression algorithms for + * testing column family compression. Column family names have special characters + */ + private Map + getMockColumnFamiliesForBloomType (int numCfs) { + Map familyToBloomType = + new HashMap(); + // use column family names having special characters + if (numCfs-- > 0) { + familyToBloomType.put("Family1!@#!@#&", BloomType.ROW); + } + if (numCfs-- > 0) { + familyToBloomType.put("Family2=asdads&!AASD", + BloomType.ROWCOL); + } + if (numCfs-- > 0) { + familyToBloomType.put("Family3", BloomType.NONE); + } + return familyToBloomType; + } + + /** + * Test for {@link HFileOutputFormat#configureBlockSize(HTable, + * Configuration)} and {@link HFileOutputFormat#createFamilyBlockSizeMap + * (Configuration)}. + * Tests that the compression map is correctly serialized into + * and deserialized from configuration + * + * @throws IOException + */ + @Test + public void testSerializeDeserializeFamilyBlockSizeMap() throws IOException { + for (int numCfs = 0; numCfs <= 3; numCfs++) { + Configuration conf = new Configuration(this.util.getConfiguration()); + Map familyToBlockSize = + getMockColumnFamiliesForBlockSize(numCfs); + HTable table = Mockito.mock(HTable.class); + setupMockColumnFamiliesForBlockSize(table, + familyToBlockSize); + HFileOutputFormat.configureBlockSize(table, conf); + + // read back family specific data block size from the configuration + Map retrievedFamilyToBlockSizeMap = + HFileOutputFormat + .createFamilyBlockSizeMap(conf); + + // test that we have a value for all column families that matches with the + // used mock values + for (Entry entry : familyToBlockSize.entrySet()) { + assertEquals("BlockSize configuration incorrect for column family:" + + entry.getKey(), entry.getValue(), + retrievedFamilyToBlockSizeMap.get(entry.getKey().getBytes())); + } + } + } + + private void setupMockColumnFamiliesForBlockSize(HTable table, + Map familyToDataBlockEncoding) throws IOException { + HTableDescriptor mockTableDescriptor = new HTableDescriptor(TABLE_NAME); + for (Entry entry : familyToDataBlockEncoding.entrySet()) { + mockTableDescriptor.addFamily(new HColumnDescriptor(entry.getKey()) + .setMaxVersions(1) + .setBlocksize(entry.getValue()) + .setBlockCacheEnabled(false) + .setTimeToLive(0)); + } + Mockito.doReturn(mockTableDescriptor).when(table).getTableDescriptor(); + } + + /** + * @return a map from column family names to compression algorithms for + * testing column family compression. Column family names have special characters + */ + private Map + getMockColumnFamiliesForBlockSize (int numCfs) { + Map familyToBlockSize = + new HashMap(); + // use column family names having special characters + if (numCfs-- > 0) { + familyToBlockSize.put("Family1!@#!@#&", 1234); + } + if (numCfs-- > 0) { + familyToBlockSize.put("Family2=asdads&!AASD", + Integer.MAX_VALUE); + } + if (numCfs-- > 0) { + familyToBlockSize.put("Family2=asdads&!AASD", + Integer.MAX_VALUE); + } + if (numCfs-- > 0) { + familyToBlockSize.put("Family3", 0); + } + return familyToBlockSize; + } + + /** + * Test for {@link HFileOutputFormat#configureDataBlockEncoding(HTable, + * Configuration)} and {@link HFileOutputFormat#createFamilyDataBlockEncodingMap + * (Configuration)}. + * Tests that the compression map is correctly serialized into + * and deserialized from configuration + * + * @throws IOException */ @Test - public void testColumnFamilyCompression() throws Exception { + public void testSerializeDeserializeFamilyDataBlockEncodingMap() throws IOException { + for (int numCfs = 0; numCfs <= 3; numCfs++) { + Configuration conf = new Configuration(this.util.getConfiguration()); + Map familyToDataBlockEncoding = + getMockColumnFamiliesForDataBlockEncoding(numCfs); + HTable table = Mockito.mock(HTable.class); + setupMockColumnFamiliesForDataBlockEncoding(table, + familyToDataBlockEncoding); + HFileOutputFormat.configureDataBlockEncoding(table, conf); + + // read back family specific data block encoding settings from the configuration + Map retrievedFamilyToDataBlockEncodingMap = + HFileOutputFormat + .createFamilyDataBlockEncodingMap(conf); + + // test that we have a value for all column families that matches with the + // used mock values + for (Entry entry : familyToDataBlockEncoding.entrySet()) { + assertEquals("DataBlockEncoding configuration incorrect for column family:" + + entry.getKey(), entry.getValue(), + retrievedFamilyToDataBlockEncodingMap.get(entry.getKey().getBytes + ()).getEncodingOnDisk()); + } + } + } + + private void setupMockColumnFamiliesForDataBlockEncoding(HTable table, + Map familyToDataBlockEncoding) throws IOException { + HTableDescriptor mockTableDescriptor = new HTableDescriptor(TABLE_NAME); + for (Entry entry : familyToDataBlockEncoding.entrySet()) { + mockTableDescriptor.addFamily(new HColumnDescriptor(entry.getKey()) + .setMaxVersions(1) + .setDataBlockEncoding(entry.getValue()) + .setBlockCacheEnabled(false) + .setTimeToLive(0)); + } + Mockito.doReturn(mockTableDescriptor).when(table).getTableDescriptor(); + } + + /** + * @return a map from column family names to compression algorithms for + * testing column family compression. Column family names have special characters + */ + private Map + getMockColumnFamiliesForDataBlockEncoding (int numCfs) { + Map familyToDataBlockEncoding = + new HashMap(); + // use column family names having special characters + if (numCfs-- > 0) { + familyToDataBlockEncoding.put("Family1!@#!@#&", DataBlockEncoding.DIFF); + } + if (numCfs-- > 0) { + familyToDataBlockEncoding.put("Family2=asdads&!AASD", + DataBlockEncoding.FAST_DIFF); + } + if (numCfs-- > 0) { + familyToDataBlockEncoding.put("Family2=asdads&!AASD", + DataBlockEncoding.PREFIX); + } + if (numCfs-- > 0) { + familyToDataBlockEncoding.put("Family3", DataBlockEncoding.NONE); + } + return familyToDataBlockEncoding; + } + + private void setupMockStartKeys(HTable table) throws IOException { + byte[][] mockKeys = new byte[][] { + HConstants.EMPTY_BYTE_ARRAY, + Bytes.toBytes("aaa"), + Bytes.toBytes("ggg"), + Bytes.toBytes("zzz") + }; + Mockito.doReturn(mockKeys).when(table).getStartKeys(); + } + + /** + * Test that {@link HFileOutputFormat} RecordWriter uses compression and + * bloom filter settings from the column family descriptor + */ + @Test + public void testColumnFamilySettings() throws Exception { Configuration conf = new Configuration(this.util.getConfiguration()); RecordWriter writer = null; TaskAttemptContext context = null; - Path dir = - util.getDataTestDir("testColumnFamilyCompression"); + Path dir = util.getDataTestDir("testColumnFamilySettings"); + // Setup table descriptor HTable table = Mockito.mock(HTable.class); - - Map configuredCompression = - new HashMap(); - Compression.Algorithm[] supportedAlgos = getSupportedCompressionAlgorithms(); - - int familyIndex = 0; - for (byte[] family : FAMILIES) { - configuredCompression.put(Bytes.toString(family), - supportedAlgos[familyIndex++ % supportedAlgos.length]); + HTableDescriptor htd = new HTableDescriptor(TABLE_NAME); + Mockito.doReturn(htd).when(table).getTableDescriptor(); + for (HColumnDescriptor hcd: this.util.generateColumnDescriptors()) { + htd.addFamily(hcd); } - setupMockColumnFamilies(table, configuredCompression); // set up the table to return some mock keys setupMockStartKeys(table); @@ -586,6 +806,7 @@ public void testColumnFamilyCompression() throws Exception { // pollutes the GZip codec pool with an incompatible compressor. conf.set("io.seqfile.compression.type", "NONE"); Job job = new Job(conf, "testLocalMRIncrementalLoad"); + job.setWorkingDirectory(util.getDataTestDir("testColumnFamilyCompression")); setupRandomGeneratorMapper(job); HFileOutputFormat.configureIncrementalLoad(job, table); FileOutputFormat.setOutputPath(job, dir); @@ -594,75 +815,45 @@ public void testColumnFamilyCompression() throws Exception { writer = hof.getRecordWriter(context); // write out random rows - writeRandomKeyValues(writer, context, ROWSPERSPLIT); + writeRandomKeyValues(writer, context, htd.getFamiliesKeys(), ROWSPERSPLIT); writer.close(context); // Make sure that a directory was created for every CF - FileSystem fileSystem = dir.getFileSystem(conf); + FileSystem fs = dir.getFileSystem(conf); // commit so that the filesystem has one directory per column family hof.getOutputCommitter(context).commitTask(context); hof.getOutputCommitter(context).commitJob(context); - for (byte[] family : FAMILIES) { - String familyStr = new String(family); - boolean found = false; - for (FileStatus f : fileSystem.listStatus(dir)) { - - if (Bytes.toString(family).equals(f.getPath().getName())) { - // we found a matching directory - found = true; - - // verify that the compression on this file matches the configured - // compression - Path dataFilePath = fileSystem.listStatus(f.getPath())[0].getPath(); - Reader reader = HFile.createReader(fileSystem, dataFilePath, - new CacheConfig(conf)); - reader.loadFileInfo(); - assertEquals("Incorrect compression used for column family " + familyStr - + "(reader: " + reader + ")", - configuredCompression.get(familyStr), reader.getCompressionAlgorithm()); - break; - } - } - - if (!found) { - fail("HFile for column family " + familyStr + " not found"); - } + FileStatus[] families = FSUtils.listStatus(fs, dir, new FSUtils.FamilyDirFilter(fs)); + assertEquals(htd.getFamilies().size(), families.length); + for (FileStatus f : families) { + String familyStr = f.getPath().getName(); + HColumnDescriptor hcd = htd.getFamily(Bytes.toBytes(familyStr)); + // verify that the compression on this file matches the configured + // compression + Path dataFilePath = fs.listStatus(f.getPath())[0].getPath(); + Reader reader = HFile.createReader(fs, dataFilePath, new CacheConfig(conf)); + Map fileInfo = reader.loadFileInfo(); + + byte[] bloomFilter = fileInfo.get(StoreFile.BLOOM_FILTER_TYPE_KEY); + if (bloomFilter == null) bloomFilter = Bytes.toBytes("NONE"); + assertEquals("Incorrect bloom filter used for column family " + familyStr + + "(reader: " + reader + ")", + hcd.getBloomFilterType(), BloomType.valueOf(Bytes.toString(bloomFilter))); + assertEquals("Incorrect compression used for column family " + familyStr + + "(reader: " + reader + ")", hcd.getCompression(), reader.getCompressionAlgorithm()); } - } finally { dir.getFileSystem(conf).delete(dir, true); } } - - /** - * @return - */ - private Compression.Algorithm[] getSupportedCompressionAlgorithms() { - String[] allAlgos = HFile.getSupportedCompressionAlgorithms(); - List supportedAlgos = Lists.newArrayList(); - - for (String algoName : allAlgos) { - try { - Compression.Algorithm algo = Compression.getCompressionAlgorithmByName(algoName); - algo.getCompressor(); - supportedAlgos.add(algo); - }catch (Exception e) { - // this algo is not available - } - } - - return supportedAlgos.toArray(new Compression.Algorithm[0]); - } - - /** * Write random values to the writer assuming a table created using * {@link #FAMILIES} as column family descriptors */ - private void writeRandomKeyValues(RecordWriter writer, TaskAttemptContext context, - int numRows) + private void writeRandomKeyValues(RecordWriter writer, + TaskAttemptContext context, Set families, int numRows) throws IOException, InterruptedException { byte keyBytes[] = new byte[Bytes.SIZEOF_INT]; int valLength = 10; @@ -678,7 +869,7 @@ private void writeRandomKeyValues(RecordWriter random.nextBytes(valBytes); ImmutableBytesWritable key = new ImmutableBytesWritable(keyBytes); - for (byte[] family : TestHFileOutputFormat.FAMILIES) { + for (byte[] family : families) { KeyValue kv = new KeyValue(keyBytes, family, PerformanceEvaluation.QUALIFIER_NAME, valBytes); writer.write(key, kv); @@ -686,12 +877,85 @@ private void writeRandomKeyValues(RecordWriter } } + /** + * This test is to test the scenario happened in HBASE-6901. + * All files are bulk loaded and excluded from minor compaction. + * Without the fix of HBASE-6901, an ArrayIndexOutOfBoundsException + * will be thrown. + */ + @Test + public void testExcludeAllFromMinorCompaction() throws Exception { + Configuration conf = util.getConfiguration(); + conf.setInt("hbase.hstore.compaction.min", 2); + generateRandomStartKeys(5); + + try { + util.startMiniCluster(); + final FileSystem fs = util.getDFSCluster().getFileSystem(); + HBaseAdmin admin = new HBaseAdmin(conf); + HTable table = util.createTable(TABLE_NAME, FAMILIES); + assertEquals("Should start with empty table", 0, util.countRows(table)); + + // deep inspection: get the StoreFile dir + final Path storePath = Store.getStoreHomedir( + HTableDescriptor.getTableDir(FSUtils.getRootDir(conf), TABLE_NAME), + admin.getTableRegions(TABLE_NAME).get(0).getEncodedName(), + FAMILIES[0]); + assertEquals(0, fs.listStatus(storePath).length); + + // Generate two bulk load files + conf.setBoolean("hbase.mapreduce.hfileoutputformat.compaction.exclude", + true); + util.startMiniMapReduceCluster(); + + for (int i = 0; i < 2; i++) { + Path testDir = util.getDataTestDir("testExcludeAllFromMinorCompaction_" + i); + runIncrementalPELoad(conf, table, testDir); + // Perform the actual load + new LoadIncrementalHFiles(conf).doBulkLoad(testDir, table); + } + + // Ensure data shows up + int expectedRows = 2 * NMapInputFormat.getNumMapTasks(conf) * ROWSPERSPLIT; + assertEquals("LoadIncrementalHFiles should put expected data in table", + expectedRows, util.countRows(table)); + + // should have a second StoreFile now + assertEquals(2, fs.listStatus(storePath).length); + + // minor compactions shouldn't get rid of the file + admin.compact(TABLE_NAME); + try { + quickPoll(new Callable() { + public Boolean call() throws Exception { + return fs.listStatus(storePath).length == 1; + } + }, 5000); + throw new IOException("SF# = " + fs.listStatus(storePath).length); + } catch (AssertionError ae) { + // this is expected behavior + } + + // a major compaction should work though + admin.majorCompact(TABLE_NAME); + quickPoll(new Callable() { + public Boolean call() throws Exception { + return fs.listStatus(storePath).length == 1; + } + }, 5000); + + } finally { + util.shutdownMiniMapReduceCluster(); + util.shutdownMiniCluster(); + } + } + @Test public void testExcludeMinorCompaction() throws Exception { Configuration conf = util.getConfiguration(); conf.setInt("hbase.hstore.compaction.min", 2); Path testDir = util.getDataTestDir("testExcludeMinorCompaction"); - byte[][] startKeys = generateRandomStartKeys(5); + generateRandomStartKeys(5); try { util.startMiniCluster(); diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java new file mode 100644 index 000000000000..f91187b0a418 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java @@ -0,0 +1,238 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.util.List; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.mapreduce.HLogInputFormat.HLogRecordReader; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.regionserver.wal.WALEdit; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.mapreduce.InputSplit; +import org.apache.hadoop.mapreduce.JobContext; +import org.apache.hadoop.mapreduce.MapReduceTestUtil; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * JUnit tests for the HLogRecordReader + */ +@Category(MediumTests.class) +public class TestHLogRecordReader { + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static Configuration conf; + private static FileSystem fs; + private static Path hbaseDir; + private static final byte [] tableName = Bytes.toBytes(getName()); + private static final byte [] rowName = tableName; + private static final HRegionInfo info = new HRegionInfo(tableName, + Bytes.toBytes(""), Bytes.toBytes(""), false); + private static final byte [] family = Bytes.toBytes("column"); + private static final byte [] value = Bytes.toBytes("value"); + private static HTableDescriptor htd; + private static Path logDir; + private static Path oldLogDir; + + private static String getName() { + return "TestHLogRecordReader"; + } + + @Before + public void setUp() throws Exception { + FileStatus[] entries = fs.listStatus(hbaseDir); + for (FileStatus dir : entries) { + fs.delete(dir.getPath(), true); + } + + } + @BeforeClass + public static void setUpBeforeClass() throws Exception { + // Make block sizes small. + conf = TEST_UTIL.getConfiguration(); + conf.setInt("dfs.blocksize", 1024 * 1024); + conf.setInt("dfs.replication", 1); + TEST_UTIL.startMiniDFSCluster(1); + + conf = TEST_UTIL.getConfiguration(); + fs = TEST_UTIL.getDFSCluster().getFileSystem(); + + hbaseDir = TEST_UTIL.createRootDir(); + logDir = new Path(hbaseDir, HConstants.HREGION_LOGDIR_NAME); + oldLogDir = new Path(hbaseDir, HConstants.HREGION_OLDLOGDIR_NAME); + htd = new HTableDescriptor(tableName); + htd.addFamily(new HColumnDescriptor(family)); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * Test partial reads from the log based on passed time range + * @throws Exception + */ + @Test + public void testPartialRead() throws Exception { + HLog log = new HLog(fs, logDir, oldLogDir, conf); + long ts = System.currentTimeMillis(); + WALEdit edit = new WALEdit(); + edit.add(new KeyValue(rowName, family, Bytes.toBytes("1"), + ts, value)); + log.append(info, tableName, edit, + ts, htd); + edit = new WALEdit(); + edit.add(new KeyValue(rowName, family, Bytes.toBytes("2"), + ts+1, value)); + log.append(info, tableName, edit, + ts+1, htd); + log.rollWriter(); + + Thread.sleep(1); + long ts1 = System.currentTimeMillis(); + + edit = new WALEdit(); + edit.add(new KeyValue(rowName, family, Bytes.toBytes("3"), + ts1+1, value)); + log.append(info, tableName, edit, + ts1+1, htd); + edit = new WALEdit(); + edit.add(new KeyValue(rowName, family, Bytes.toBytes("4"), + ts1+2, value)); + log.append(info, tableName, edit, + ts1+2, htd); + log.close(); + + HLogInputFormat input = new HLogInputFormat(); + Configuration jobConf = new Configuration(conf); + jobConf.set("mapred.input.dir", logDir.toString()); + jobConf.setLong(HLogInputFormat.END_TIME_KEY, ts); + + // only 1st file is considered, and only its 1st entry is used + List splits = input.getSplits(MapreduceTestingShim.createJobContext(jobConf)); + assertEquals(1, splits.size()); + testSplit(splits.get(0), Bytes.toBytes("1")); + + jobConf.setLong(HLogInputFormat.START_TIME_KEY, ts+1); + jobConf.setLong(HLogInputFormat.END_TIME_KEY, ts1+1); + splits = input.getSplits(MapreduceTestingShim.createJobContext(jobConf)); + // both files need to be considered + assertEquals(2, splits.size()); + // only the 2nd entry from the 1st file is used + testSplit(splits.get(0), Bytes.toBytes("2")); + // only the 1nd entry from the 2nd file is used + testSplit(splits.get(1), Bytes.toBytes("3")); + } + + /** + * Test basic functionality + * @throws Exception + */ + @Test + public void testHLogRecordReader() throws Exception { + HLog log = new HLog(fs, logDir, oldLogDir, conf); + byte [] value = Bytes.toBytes("value"); + WALEdit edit = new WALEdit(); + edit.add(new KeyValue(rowName, family, Bytes.toBytes("1"), + System.currentTimeMillis(), value)); + log.append(info, tableName, edit, + System.currentTimeMillis(), htd); + + Thread.sleep(1); // make sure 2nd log gets a later timestamp + long secondTs = System.currentTimeMillis(); + log.rollWriter(); + + edit = new WALEdit(); + edit.add(new KeyValue(rowName, family, Bytes.toBytes("2"), + System.currentTimeMillis(), value)); + log.append(info, tableName, edit, + System.currentTimeMillis(), htd); + log.close(); + long thirdTs = System.currentTimeMillis(); + + // should have 2 log files now + HLogInputFormat input = new HLogInputFormat(); + Configuration jobConf = new Configuration(conf); + jobConf.set("mapred.input.dir", logDir.toString()); + + // make sure both logs are found + List splits = input.getSplits(MapreduceTestingShim.createJobContext(jobConf)); + assertEquals(2, splits.size()); + + // should return exactly one KV + testSplit(splits.get(0), Bytes.toBytes("1")); + // same for the 2nd split + testSplit(splits.get(1), Bytes.toBytes("2")); + + // now test basic time ranges: + + // set an endtime, the 2nd log file can be ignored completely. + jobConf.setLong(HLogInputFormat.END_TIME_KEY, secondTs-1); + splits = input.getSplits(MapreduceTestingShim.createJobContext(jobConf)); + assertEquals(1, splits.size()); + testSplit(splits.get(0), Bytes.toBytes("1")); + + // now set a start time + jobConf.setLong(HLogInputFormat.END_TIME_KEY, Long.MAX_VALUE); + jobConf.setLong(HLogInputFormat.START_TIME_KEY, thirdTs); + splits = input.getSplits(MapreduceTestingShim.createJobContext(jobConf)); + // both logs need to be considered + assertEquals(2, splits.size()); + // but both readers skip all edits + testSplit(splits.get(0)); + testSplit(splits.get(1)); + } + + /** + * Create a new reader from the split, and match the edits against the passed columns. + */ + private void testSplit(InputSplit split, byte[]... columns) throws Exception { + HLogRecordReader reader = new HLogRecordReader(); + reader.initialize(split, MapReduceTestUtil.createDummyMapTaskAttemptContext(conf)); + + for (byte[] column : columns) { + assertTrue(reader.nextKeyValue()); + assertTrue(Bytes + .equals(column, reader.getCurrentValue().getKeyValues().get(0).getQualifier())); + } + assertFalse(reader.nextKeyValue()); + reader.close(); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java index a3faeb3380b1..d77b9a420e75 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java @@ -17,18 +17,18 @@ */ package org.apache.hadoop.hbase.mapreduce; -import static org.junit.Assert.assertTrue; +import static org.junit.Assert.*; + +import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HColumnDescriptor; -import org.apache.hadoop.hbase.HConstants; import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.MediumTests; -import org.apache.hadoop.hbase.MiniHBaseCluster; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HTable; @@ -36,16 +36,18 @@ import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.PrefixFilter; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.util.GenericOptionsParser; import org.junit.After; import org.junit.AfterClass; +import org.junit.Assert; import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; -import static org.junit.Assert.assertEquals; @Category(MediumTests.class) public class TestImportExport { @@ -59,13 +61,13 @@ public class TestImportExport { private static final byte[] QUAL = Bytes.toBytes("q"); private static final String OUTPUT_DIR = "outputdir"; - private static MiniHBaseCluster cluster; private static long now = System.currentTimeMillis(); @BeforeClass public static void beforeClass() throws Exception { - cluster = UTIL.startMiniCluster(); + UTIL.startMiniCluster(); UTIL.startMiniMapReduceCluster(); + UTIL.getConfiguration().set("mapred.job.tracker", "local"); } @AfterClass @@ -105,12 +107,12 @@ public void testSimpleCase() throws Exception { OUTPUT_DIR, "1000" }; - - GenericOptionsParser opts = new GenericOptionsParser(new Configuration(cluster.getConfiguration()), args); + GenericOptionsParser opts = new GenericOptionsParser(new Configuration(UTIL.getConfiguration()), args); Configuration conf = opts.getConfiguration(); args = opts.getRemainingArgs(); Job job = Export.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); job.waitForCompletion(false); assertTrue(job.isSuccessful()); @@ -123,11 +125,12 @@ public void testSimpleCase() throws Exception { OUTPUT_DIR }; - opts = new GenericOptionsParser(new Configuration(cluster.getConfiguration()), args); + opts = new GenericOptionsParser(new Configuration(UTIL.getConfiguration()), args); conf = opts.getConfiguration(); args = opts.getRemainingArgs(); job = Import.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); job.waitForCompletion(false); assertTrue(job.isSuccessful()); @@ -141,6 +144,26 @@ public void testSimpleCase() throws Exception { assertEquals(3, r.size()); } + /** + * Test export .META. table + * + * @throws Exception + */ + @Test + public void testMetaExport() throws Exception { + String EXPORT_TABLE = ".META."; + String[] args = new String[] { EXPORT_TABLE, OUTPUT_DIR, "1", "0", "0" }; + GenericOptionsParser opts = new GenericOptionsParser(new Configuration( + UTIL.getConfiguration()), args); + Configuration conf = opts.getConfiguration(); + args = opts.getRemainingArgs(); + + Job job = Export.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); + job.waitForCompletion(false); + assertTrue(job.isSuccessful()); + } + @Test public void testWithDeletes() throws Exception { String EXPORT_TABLE = "exportWithDeletes"; @@ -160,7 +183,7 @@ public void testWithDeletes() throws Exception { p.add(FAMILYA, QUAL, now+4, QUAL); t.put(p); - Delete d = new Delete(ROW1, now+3, null); + Delete d = new Delete(ROW1, now+3); t.delete(d); d = new Delete(ROW1); d.deleteColumns(FAMILYA, QUAL, now+2); @@ -173,11 +196,12 @@ public void testWithDeletes() throws Exception { "1000" }; - GenericOptionsParser opts = new GenericOptionsParser(new Configuration(cluster.getConfiguration()), args); + GenericOptionsParser opts = new GenericOptionsParser(new Configuration(UTIL.getConfiguration()), args); Configuration conf = opts.getConfiguration(); args = opts.getRemainingArgs(); Job job = Export.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); job.waitForCompletion(false); assertTrue(job.isSuccessful()); @@ -196,11 +220,12 @@ public void testWithDeletes() throws Exception { OUTPUT_DIR }; - opts = new GenericOptionsParser(new Configuration(cluster.getConfiguration()), args); + opts = new GenericOptionsParser(new Configuration(UTIL.getConfiguration()), args); conf = opts.getConfiguration(); args = opts.getRemainingArgs(); job = Import.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); job.waitForCompletion(false); assertTrue(job.isSuccessful()); @@ -219,4 +244,192 @@ public void testWithDeletes() throws Exception { assertEquals(now, res[6].getTimestamp()); t.close(); } + + @Test + public void testWithFilter() throws Exception { + String EXPORT_TABLE = "exportSimpleCase_ImportWithFilter"; + HTableDescriptor desc = new HTableDescriptor(EXPORT_TABLE); + desc.addFamily(new HColumnDescriptor(FAMILYA).setMaxVersions(5)); + UTIL.getHBaseAdmin().createTable(desc); + HTable exportTable = new HTable(UTIL.getConfiguration(), EXPORT_TABLE); + + Put p = new Put(ROW1); + p.add(FAMILYA, QUAL, now, QUAL); + p.add(FAMILYA, QUAL, now + 1, QUAL); + p.add(FAMILYA, QUAL, now + 2, QUAL); + p.add(FAMILYA, QUAL, now + 3, QUAL); + p.add(FAMILYA, QUAL, now + 4, QUAL); + exportTable.put(p); + + String[] args = new String[] { EXPORT_TABLE, OUTPUT_DIR, "1000" }; + + GenericOptionsParser opts = new GenericOptionsParser(new Configuration( + UTIL.getConfiguration()), args); + Configuration conf = opts.getConfiguration(); + args = opts.getRemainingArgs(); + + Job job = Export.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); + job.waitForCompletion(false); + assertTrue(job.isSuccessful()); + + String IMPORT_TABLE = "importWithFilter"; + desc = new HTableDescriptor(IMPORT_TABLE); + desc.addFamily(new HColumnDescriptor(FAMILYA).setMaxVersions(5)); + UTIL.getHBaseAdmin().createTable(desc); + + HTable importTable = new HTable(UTIL.getConfiguration(), IMPORT_TABLE); + args = new String[] { "-D" + Import.FILTER_CLASS_CONF_KEY + "=" + PrefixFilter.class.getName(), + "-D" + Import.FILTER_ARGS_CONF_KEY + "=" + Bytes.toString(ROW1), IMPORT_TABLE, OUTPUT_DIR, + "1000" }; + + opts = new GenericOptionsParser(new Configuration(UTIL.getConfiguration()), args); + conf = opts.getConfiguration(); + args = opts.getRemainingArgs(); + + job = Import.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); + job.waitForCompletion(false); + assertTrue(job.isSuccessful()); + + // get the count of the source table for that time range + PrefixFilter filter = new PrefixFilter(ROW1); + int count = getCount(exportTable, filter); + + Assert.assertEquals("Unexpected row count between export and import tables", count, + getCount(importTable, null)); + + // and then test that a broken command doesn't bork everything - easier here because we don't + // need to re-run the export job + + args = new String[] { "-D" + Import.FILTER_CLASS_CONF_KEY + "=" + Filter.class.getName(), + "-D" + Import.FILTER_ARGS_CONF_KEY + "=" + Bytes.toString(ROW1) + "", EXPORT_TABLE, + OUTPUT_DIR, "1000" }; + + opts = new GenericOptionsParser(new Configuration(UTIL.getConfiguration()), args); + conf = opts.getConfiguration(); + args = opts.getRemainingArgs(); + + job = Import.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); + job.waitForCompletion(false); + assertFalse("Job succeeedd, but it had a non-instantiable filter!", job.isSuccessful()); + + // cleanup + exportTable.close(); + importTable.close(); + } + + + @Test + public void testWithMultipleDeleteFamilyMarkersOfSameRowSameFamily() throws Exception { + String EXPORT_TABLE = "exportWithMultipleDeleteFamilyMarkersOfSameRowSameFamily"; + HTableDescriptor desc = new HTableDescriptor(EXPORT_TABLE); + desc.addFamily(new HColumnDescriptor(FAMILYA) + .setMaxVersions(5) + .setKeepDeletedCells(true) + ); + UTIL.getHBaseAdmin().createTable(desc); + HTable exportT = new HTable(UTIL.getConfiguration(), EXPORT_TABLE); + + //Add first version of QUAL + Put p = new Put(ROW1); + p.add(FAMILYA, QUAL, now, QUAL); + exportT.put(p); + + //Add Delete family marker + Delete d = new Delete(ROW1, now+3); + exportT.delete(d); + + //Add second version of QUAL + p = new Put(ROW1); + p.add(FAMILYA, QUAL, now+5, "s".getBytes()); + exportT.put(p); + + //Add second Delete family marker + d = new Delete(ROW1, now+7); + exportT.delete(d); + + String[] args = new String[] { + "-D" + Export.RAW_SCAN + "=true", + EXPORT_TABLE, + OUTPUT_DIR, + "1000", // max number of key versions per key to export + }; + + GenericOptionsParser opts = new GenericOptionsParser(new Configuration( + UTIL.getConfiguration()), args); + Configuration conf = opts.getConfiguration(); + args = opts.getRemainingArgs(); + + Job job = Export.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); + job.waitForCompletion(false); + assertTrue(job.isSuccessful()); + + String IMPORT_TABLE = "importWithMultipleDeleteFamilyMarkersOfSameRowSameFamily"; + desc = new HTableDescriptor(IMPORT_TABLE); + desc.addFamily(new HColumnDescriptor(FAMILYA) + .setMaxVersions(5) + .setKeepDeletedCells(true) + ); + UTIL.getHBaseAdmin().createTable(desc); + + HTable importT = new HTable(UTIL.getConfiguration(), IMPORT_TABLE); + args = new String[] { + IMPORT_TABLE, + OUTPUT_DIR + }; + + opts = new GenericOptionsParser(new Configuration(UTIL.getConfiguration()), args); + conf = opts.getConfiguration(); + args = opts.getRemainingArgs(); + + job = Import.createSubmittableJob(conf, args); + job.getConfiguration().set("mapreduce.framework.name", "yarn"); + job.waitForCompletion(false); + assertTrue(job.isSuccessful()); + + Scan s = new Scan(); + s.setMaxVersions(); + s.setRaw(true); + + ResultScanner importedTScanner = importT.getScanner(s); + Result importedTResult = importedTScanner.next(); + + ResultScanner exportedTScanner = exportT.getScanner(s); + Result exportedTResult = exportedTScanner.next(); + try + { + Result.compareResults(exportedTResult, importedTResult); + } + catch (Exception e) { + fail("Original and imported tables data comparision failed with error:"+e.getMessage()); + } + finally + { + exportT.close(); + importT.close(); + } + } + + /** + * Count the number of keyvalues in the specified table for the given timerange + * @param start + * @param end + * @param table + * @return + * @throws IOException + */ + private int getCount(HTable table, Filter filter) throws IOException { + Scan scan = new Scan(); + scan.setFilter(filter); + ResultScanner results = table.getScanner(scan); + int count = 0; + for (Result res : results) { + count += res.size(); + } + results.close(); + return count; + } } diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java index ac30a62677a1..dd0263788ad2 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -23,6 +22,8 @@ import java.util.List; import java.util.ArrayList; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; import org.apache.hadoop.hbase.*; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.fs.FSDataOutputStream; @@ -52,6 +53,7 @@ @Category(MediumTests.class) public class TestImportTsv { + private static final Log LOG = LogFactory.getLog(TestImportTsv.class); @Test public void testTsvParserSpecParsing() { @@ -61,6 +63,7 @@ public void testTsvParserSpecParsing() { assertNull(parser.getFamily(0)); assertNull(parser.getQualifier(0)); assertEquals(0, parser.getRowKeyColumnIndex()); + assertFalse(parser.hasTimestamp()); parser = new TsvParser("HBASE_ROW_KEY,col1:scol1", "\t"); assertNull(parser.getFamily(0)); @@ -68,6 +71,7 @@ public void testTsvParserSpecParsing() { assertBytesEquals(Bytes.toBytes("col1"), parser.getFamily(1)); assertBytesEquals(Bytes.toBytes("scol1"), parser.getQualifier(1)); assertEquals(0, parser.getRowKeyColumnIndex()); + assertFalse(parser.hasTimestamp()); parser = new TsvParser("HBASE_ROW_KEY,col1:scol1,col1:scol2", "\t"); assertNull(parser.getFamily(0)); @@ -77,6 +81,19 @@ public void testTsvParserSpecParsing() { assertBytesEquals(Bytes.toBytes("col1"), parser.getFamily(2)); assertBytesEquals(Bytes.toBytes("scol2"), parser.getQualifier(2)); assertEquals(0, parser.getRowKeyColumnIndex()); + assertFalse(parser.hasTimestamp()); + + parser = new TsvParser("HBASE_ROW_KEY,col1:scol1,HBASE_TS_KEY,col1:scol2", + "\t"); + assertNull(parser.getFamily(0)); + assertNull(parser.getQualifier(0)); + assertBytesEquals(Bytes.toBytes("col1"), parser.getFamily(1)); + assertBytesEquals(Bytes.toBytes("scol1"), parser.getQualifier(1)); + assertBytesEquals(Bytes.toBytes("col1"), parser.getFamily(3)); + assertBytesEquals(Bytes.toBytes("scol2"), parser.getQualifier(3)); + assertEquals(0, parser.getRowKeyColumnIndex()); + assertTrue(parser.hasTimestamp()); + assertEquals(2, parser.getTimestampKeyColumnIndex()); } @Test @@ -90,10 +107,32 @@ public void testTsvParser() throws BadTsvLineException { assertNull(parser.getQualifier(2)); assertEquals(2, parser.getRowKeyColumnIndex()); + assertEquals(TsvParser.DEFAULT_TIMESTAMP_COLUMN_INDEX, parser + .getTimestampKeyColumnIndex()); + byte[] line = Bytes.toBytes("val_a\tval_b\tval_c\tval_d"); ParsedLine parsed = parser.parse(line, line.length); checkParsing(parsed, Splitter.on("\t").split(Bytes.toString(line))); } + + + @Test + public void testTsvParserWithTimestamp() throws BadTsvLineException { + TsvParser parser = new TsvParser("HBASE_ROW_KEY,HBASE_TS_KEY,col_a,", "\t"); + assertNull(parser.getFamily(0)); + assertNull(parser.getQualifier(0)); + assertNull(parser.getFamily(1)); + assertNull(parser.getQualifier(1)); + assertBytesEquals(Bytes.toBytes("col_a"), parser.getFamily(2)); + assertBytesEquals(HConstants.EMPTY_BYTE_ARRAY, parser.getQualifier(2)); + assertEquals(0, parser.getRowKeyColumnIndex()); + assertEquals(1, parser.getTimestampKeyColumnIndex()); + + byte[] line = Bytes.toBytes("rowkey\t1234\tval_a"); + ParsedLine parsed = parser.parse(line, line.length); + assertEquals(1234l, parsed.getTimestamp(-1)); + checkParsing(parsed, Splitter.on("\t").split(Bytes.toString(line))); + } private void checkParsing(ParsedLine parsed, Iterable expected) { ArrayList parsedCols = new ArrayList(); @@ -120,29 +159,48 @@ private void assertBytesEquals(byte[] a, byte[] b) { public void testTsvParserBadTsvLineExcessiveColumns() throws BadTsvLineException { TsvParser parser = new TsvParser("HBASE_ROW_KEY,col_a", "\t"); byte[] line = Bytes.toBytes("val_a\tval_b\tval_c"); - ParsedLine parsed = parser.parse(line, line.length); + parser.parse(line, line.length); } @Test(expected=BadTsvLineException.class) public void testTsvParserBadTsvLineZeroColumn() throws BadTsvLineException { TsvParser parser = new TsvParser("HBASE_ROW_KEY,col_a", "\t"); byte[] line = Bytes.toBytes(""); - ParsedLine parsed = parser.parse(line, line.length); + parser.parse(line, line.length); } @Test(expected=BadTsvLineException.class) public void testTsvParserBadTsvLineOnlyKey() throws BadTsvLineException { TsvParser parser = new TsvParser("HBASE_ROW_KEY,col_a", "\t"); byte[] line = Bytes.toBytes("key_only"); - ParsedLine parsed = parser.parse(line, line.length); + parser.parse(line, line.length); } @Test(expected=BadTsvLineException.class) public void testTsvParserBadTsvLineNoRowKey() throws BadTsvLineException { TsvParser parser = new TsvParser("col_a,HBASE_ROW_KEY", "\t"); byte[] line = Bytes.toBytes("only_cola_data_and_no_row_key"); + parser.parse(line, line.length); + } + + @Test(expected = BadTsvLineException.class) + public void testTsvParserInvalidTimestamp() throws BadTsvLineException { + TsvParser parser = new TsvParser("HBASE_ROW_KEY,HBASE_TS_KEY,col_a,", "\t"); + assertEquals(1, parser.getTimestampKeyColumnIndex()); + byte[] line = Bytes.toBytes("rowkey\ttimestamp\tval_a"); ParsedLine parsed = parser.parse(line, line.length); + assertEquals(-1, parsed.getTimestamp(-1)); + checkParsing(parsed, Splitter.on("\t").split(Bytes.toString(line))); } + + @Test(expected = BadTsvLineException.class) + public void testTsvParserNoTimestampValue() throws BadTsvLineException { + TsvParser parser = new TsvParser("HBASE_ROW_KEY,col_a,HBASE_TS_KEY", "\t"); + assertEquals(2, parser.getTimestampKeyColumnIndex()); + byte[] line = Bytes.toBytes("rowkey\tval_a"); + parser.parse(line, line.length); + } + @Test public void testMROnTable() @@ -159,8 +217,25 @@ public void testMROnTable() INPUT_FILE }; - doMROnTableTest(INPUT_FILE, FAMILY, TABLE_NAME, args, 1); + doMROnTableTest(INPUT_FILE, FAMILY, TABLE_NAME, null, args, 1); } + + @Test + public void testMROnTableWithTimestamp() throws Exception { + String TABLE_NAME = "TestTable"; + String FAMILY = "FAM"; + String INPUT_FILE = "InputFile1.csv"; + + // Prepare the arguments required for the test. + String[] args = new String[] { + "-D" + ImportTsv.COLUMNS_CONF_KEY + + "=HBASE_ROW_KEY,HBASE_TS_KEY,FAM:A,FAM:B", + "-D" + ImportTsv.SEPARATOR_CONF_KEY + "=,", TABLE_NAME, INPUT_FILE }; + + String data = "KEY,1234,VALUE1,VALUE2\n"; + doMROnTableTest(INPUT_FILE, FAMILY, TABLE_NAME, data, args, 1); + } + @Test public void testMROnTableWithCustomMapper() @@ -176,16 +251,16 @@ public void testMROnTableWithCustomMapper() INPUT_FILE }; - doMROnTableTest(INPUT_FILE, FAMILY, TABLE_NAME, args, 3); + doMROnTableTest(INPUT_FILE, FAMILY, TABLE_NAME, null, args, 3); } private void doMROnTableTest(String inputFile, String family, String tableName, - String[] args, int valueMultiplier) throws Exception { + String data, String[] args, int valueMultiplier) throws Exception { // Cluster HBaseTestingUtility htu1 = new HBaseTestingUtility(); - MiniHBaseCluster cluster = htu1.startMiniCluster(); + htu1.startMiniCluster(); htu1.startMiniMapReduceCluster(); GenericOptionsParser opts = new GenericOptionsParser(htu1.getConfiguration(), args); @@ -193,22 +268,26 @@ private void doMROnTableTest(String inputFile, String family, String tableName, args = opts.getRemainingArgs(); try { - FileSystem fs = FileSystem.get(conf); FSDataOutputStream op = fs.create(new Path(inputFile), true); - String line = "KEY\u001bVALUE1\u001bVALUE2\n"; - op.write(line.getBytes(HConstants.UTF8_ENCODING)); + if (data == null) { + data = "KEY\u001bVALUE1\u001bVALUE2\n"; + } + op.write(Bytes.toBytes(data)); op.close(); final byte[] FAM = Bytes.toBytes(family); final byte[] TAB = Bytes.toBytes(tableName); - final byte[] QA = Bytes.toBytes("A"); - final byte[] QB = Bytes.toBytes("B"); - - HTableDescriptor desc = new HTableDescriptor(TAB); - desc.addFamily(new HColumnDescriptor(FAM)); - new HBaseAdmin(conf).createTable(desc); - + if (conf.get(ImportTsv.BULK_OUTPUT_CONF_KEY) == null) { + HTableDescriptor desc = new HTableDescriptor(TAB); + desc.addFamily(new HColumnDescriptor(FAM)); + HBaseAdmin admin = new HBaseAdmin(conf); + admin.createTable(desc); + admin.close(); + } else { // set the hbaseAdmin as we are not going through main() + LOG.info("set the hbaseAdmin"); + ImportTsv.createHbaseAdmin(conf); + } Job job = ImportTsv.createSubmittableJob(conf, args); job.waitForCompletion(false); assertTrue(job.isSuccessful()); @@ -248,6 +327,7 @@ private void doMROnTableTest(String inputFile, String family, String tableName, // continue } } + table.close(); assertTrue(verified); } finally { htu1.shutdownMiniMapReduceCluster(); @@ -255,8 +335,23 @@ private void doMROnTableTest(String inputFile, String family, String tableName, } } + @Test + public void testBulkOutputWithoutAnExistingTable() throws Exception { + String TABLE_NAME = "TestTable"; + String FAMILY = "FAM"; + String INPUT_FILE = "InputFile2.esv"; + + // Prepare the arguments required for the test. + String[] args = new String[] { + "-D" + ImportTsv.COLUMNS_CONF_KEY + "=HBASE_ROW_KEY,FAM:A,FAM:B", + "-D" + ImportTsv.SEPARATOR_CONF_KEY + "=\u001b", + "-D" + ImportTsv.BULK_OUTPUT_CONF_KEY + "=output", TABLE_NAME, + INPUT_FILE }; + doMROnTableTest(INPUT_FILE, FAMILY, TABLE_NAME, null, args, 3); + } + public static String toU8Str(byte[] bytes) throws UnsupportedEncodingException { - return new String(bytes, HConstants.UTF8_ENCODING); + return new String(bytes); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java index d0f9ef72e665..c48963414327 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -24,21 +23,30 @@ import static org.junit.Assert.assertTrue; import java.io.IOException; +import java.util.List; import java.util.TreeMap; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.UserProvider; import org.apache.hadoop.hbase.io.hfile.CacheConfig; -import org.apache.hadoop.hbase.io.hfile.Compression; import org.apache.hadoop.hbase.io.hfile.HFile; import org.apache.hadoop.hbase.io.hfile.HFileScanner; +import org.apache.hadoop.hbase.regionserver.StoreFile; import org.apache.hadoop.hbase.regionserver.StoreFile.BloomType; import org.apache.hadoop.hbase.util.Bytes; -import org.junit.*; +import org.apache.hadoop.hbase.util.HFileTestUtil; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; import org.junit.experimental.categories.Category; /** @@ -48,6 +56,7 @@ */ @Category(LargeTests.class) public class TestLoadIncrementalHFiles { + protected static boolean useSecureHBaseOverride = false; private static final byte[] QUALIFIER = Bytes.toBytes("myqual"); private static final byte[] FAMILY = Bytes.toBytes("myfam"); @@ -56,11 +65,7 @@ public class TestLoadIncrementalHFiles { Bytes.toBytes("ppp") }; - public static int BLOCKSIZE = 64*1024; - public static String COMPRESSION = - Compression.Algorithm.NONE.getName(); - - private static HBaseTestingUtility util = new HBaseTestingUtility(); + static HBaseTestingUtility util = new HBaseTestingUtility(); @BeforeClass public static void setUpBeforeClass() throws Exception { @@ -133,7 +138,7 @@ private void runTest(String testName, BloomType bloomType, for (byte[][] range : hfileRanges) { byte[] from = range[0]; byte[] to = range[1]; - createHFile(util.getConfiguration(), fs, new Path(familyDir, "hfile_" + HFileTestUtil.createHFile(util.getConfiguration(), fs, new Path(familyDir, "hfile_" + hfileIdx++), FAMILY, QUALIFIER, from, to, 1000); } int expectedRows = hfileIdx * 1000; @@ -149,20 +154,115 @@ private void runTest(String testName, BloomType bloomType, HTable table = new HTable(util.getConfiguration(), TABLE); util.waitTableAvailable(TABLE, 30000); - LoadIncrementalHFiles loader = new LoadIncrementalHFiles( - util.getConfiguration()); + + LoadIncrementalHFiles loader = + new LoadIncrementalHFiles(util.getConfiguration(), useSecureHBaseOverride); loader.doBulkLoad(dir, table); assertEquals(expectedRows, util.countRows(table)); } + private void + verifyAssignedSequenceNumber(String testName, byte[][][] hfileRanges, boolean nonZero) + throws Exception { + Path dir = util.getDataTestDir(testName); + FileSystem fs = util.getTestFileSystem(); + dir = dir.makeQualified(fs); + Path familyDir = new Path(dir, Bytes.toString(FAMILY)); + + int hfileIdx = 0; + for (byte[][] range : hfileRanges) { + byte[] from = range[0]; + byte[] to = range[1]; + HFileTestUtil.createHFile(util.getConfiguration(), fs, new Path(familyDir, + "hfile_" + hfileIdx++), FAMILY, QUALIFIER, from, to, 1000); + } + + final byte[] TABLE = Bytes.toBytes("mytable_" + testName); + + HBaseAdmin admin = new HBaseAdmin(util.getConfiguration()); + HTableDescriptor htd = new HTableDescriptor(TABLE); + HColumnDescriptor familyDesc = new HColumnDescriptor(FAMILY); + htd.addFamily(familyDesc); + admin.createTable(htd, SPLIT_KEYS); + + HTable table = new HTable(util.getConfiguration(), TABLE); + util.waitTableAvailable(TABLE, 30000); + LoadIncrementalHFiles loader = + new LoadIncrementalHFiles(util.getConfiguration(), useSecureHBaseOverride); + + // Do a dummy put to increase the hlog sequence number + Put put = new Put(Bytes.toBytes("row")); + put.add(FAMILY, QUALIFIER, Bytes.toBytes("value")); + table.put(put); + + loader.doBulkLoad(dir, table); + + // Get the store files + List files = + util.getHBaseCluster().getRegions(TABLE).get(0).getStore(FAMILY).getStorefiles(); + for (StoreFile file : files) { + // the sequenceId gets initialized during createReader + file.createReader(); + + if (nonZero) assertTrue(file.getMaxSequenceId() > 0); + else assertTrue(file.getMaxSequenceId() == -1); + } + } + + /** + * Test loading into a column family that does not exist. + */ + @Test + public void testNonexistentColumnFamilyLoad() throws Exception { + String testName = "testNonexistentColumnFamilyLoad"; + byte[][][] hfileRanges = new byte[][][] { + new byte[][]{ Bytes.toBytes("aaa"), Bytes.toBytes("ccc") }, + new byte[][]{ Bytes.toBytes("ddd"), Bytes.toBytes("ooo") }, + }; + + Path dir = util.getDataTestDir(testName); + FileSystem fs = util.getTestFileSystem(); + dir = dir.makeQualified(fs); + Path familyDir = new Path(dir, Bytes.toString(FAMILY)); + + int hfileIdx = 0; + for (byte[][] range : hfileRanges) { + byte[] from = range[0]; + byte[] to = range[1]; + HFileTestUtil.createHFile(util.getConfiguration(), fs, new Path(familyDir, "hfile_" + + hfileIdx++), FAMILY, QUALIFIER, from, to, 1000); + } + + final byte[] TABLE = Bytes.toBytes("mytable_"+testName); + + HBaseAdmin admin = new HBaseAdmin(util.getConfiguration()); + HTableDescriptor htd = new HTableDescriptor(TABLE); + admin.createTable(htd, SPLIT_KEYS); + + HTable table = new HTable(util.getConfiguration(), TABLE); + util.waitTableAvailable(TABLE, 30000); + // make sure we go back to the usual user provider + UserProvider.setUserProviderForTesting(util.getConfiguration(), UserProvider.class); + LoadIncrementalHFiles loader = + new LoadIncrementalHFiles(util.getConfiguration(), useSecureHBaseOverride); + try { + loader.doBulkLoad(dir, table); + assertTrue("Loading into table with non-existent family should have failed", false); + } catch (Exception e) { + assertTrue("IOException expected", e instanceof IOException); + } + table.close(); + admin.close(); + } + @Test public void testSplitStoreFile() throws IOException { Path dir = util.getDataTestDir("testSplitHFile"); FileSystem fs = util.getTestFileSystem(); Path testIn = new Path(dir, "testhfile"); HColumnDescriptor familyDesc = new HColumnDescriptor(FAMILY); - createHFile(util.getConfiguration(), fs, testIn, FAMILY, QUALIFIER, + HFileTestUtil.createHFile(util.getConfiguration(), fs, testIn, FAMILY, QUALIFIER, Bytes.toBytes("aaa"), Bytes.toBytes("zzz"), 1000); Path bottomOut = new Path(dir, "bottom.out"); @@ -195,36 +295,6 @@ private int verifyHFile(Path p) throws IOException { return count; } - - /** - * Create an HFile with the given number of rows between a given - * start key and end key. - * TODO put me in an HFileTestUtil or something? - */ - static void createHFile( - Configuration conf, - FileSystem fs, Path path, - byte[] family, byte[] qualifier, - byte[] startKey, byte[] endKey, int numRows) throws IOException - { - HFile.Writer writer = HFile.getWriterFactory(conf, new CacheConfig(conf)) - .withPath(fs, path) - .withBlockSize(BLOCKSIZE) - .withCompression(COMPRESSION) - .withComparator(KeyValue.KEY_COMPARATOR) - .create(); - long now = System.currentTimeMillis(); - try { - // subtract 2 since iterateOnSplits doesn't include boundary keys - for (byte[] key : Bytes.iterateOnSplits(startKey, endKey, numRows-2)) { - KeyValue kv = new KeyValue(key, family, qualifier, now, key); - writer.append(kv); - } - } finally { - writer.close(); - } - } - private void addStartEndKeysForTest(TreeMap map, byte[] first, byte[] last) { Integer value = map.containsKey(first)?(Integer)map.get(first):0; map.put(first, value+1); diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java index 301ee27f1a92..73d549af0f3b 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java @@ -37,17 +37,20 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; import org.apache.hadoop.hbase.HRegionInfo; import org.apache.hadoop.hbase.HRegionLocation; import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.LargeTests; import org.apache.hadoop.hbase.TableExistsException; +import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.catalog.MetaEditor; +import org.apache.hadoop.hbase.catalog.MetaReader; import org.apache.hadoop.hbase.client.HConnection; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; -import org.apache.hadoop.hbase.client.ServerCallable; import org.apache.hadoop.hbase.ipc.HRegionInterface; import org.apache.hadoop.hbase.regionserver.HRegionServer; import org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad; @@ -68,11 +71,13 @@ public class TestLoadIncrementalHFilesSplitRecovery { final static Log LOG = LogFactory.getLog(TestHRegionServerBulkLoad.class); - private static HBaseTestingUtility util; + static HBaseTestingUtility util; final static int NUM_CFS = 10; final static byte[] QUAL = Bytes.toBytes("qual"); final static int ROWCOUNT = 100; + + protected static boolean useSecureHBaseOverride = false; private final static byte[][] families = new byte[NUM_CFS][]; static { @@ -122,6 +127,29 @@ private void setupTable(String table, int cfs) throws IOException { } } + /** + * Creates a table with given table name,specified number of column families
+ * and splitkeys if the table does not already exist. + * @param table + * @param cfs + * @param SPLIT_KEYS + */ + private void setupTableWithSplitkeys(String table, int cfs, byte[][] SPLIT_KEYS) + throws IOException { + try { + LOG.info("Creating table " + table); + HTableDescriptor htd = new HTableDescriptor(table); + for (int i = 0; i < cfs; i++) { + htd.addFamily(new HColumnDescriptor(family(i))); + } + + util.getHBaseAdmin().createTable(htd, SPLIT_KEYS); + } catch (TableExistsException tee) { + LOG.info("Table " + table + " already exists"); + } + } + + private Path buildBulkFiles(String table, int value) throws Exception { Path dir = util.getDataTestDir(table); Path bulk1 = new Path(dir, table+value); @@ -135,8 +163,8 @@ private Path buildBulkFiles(String table, int value) throws Exception { */ private void populateTable(String table, int value) throws Exception { // create HFiles for different column families - LoadIncrementalHFiles lih = new LoadIncrementalHFiles( - util.getConfiguration()); + LoadIncrementalHFiles lih = + new LoadIncrementalHFiles(util.getConfiguration(), useSecureHBaseOverride); Path bulk1 = buildBulkFiles(table, value); HTable t = new HTable(util.getConfiguration(), Bytes.toBytes(table)); lih.doBulkLoad(bulk1, t); @@ -228,7 +256,7 @@ public void testBulkLoadPhaseFailure() throws Exception { final AtomicInteger attmptedCalls = new AtomicInteger(); final AtomicInteger failedCalls = new AtomicInteger(); LoadIncrementalHFiles lih = new LoadIncrementalHFiles( - util.getConfiguration()) { + util.getConfiguration(), useSecureHBaseOverride) { protected List tryAtomicRegionLoad(final HConnection conn, byte[] tableName, final byte[] first, Collection lqis) @@ -272,8 +300,8 @@ private HConnection getMockedConnection(final Configuration conf) Mockito.when(c.locateRegion((byte[]) Mockito.any(), (byte[]) Mockito.any())). thenReturn(loc); HRegionInterface hri = Mockito.mock(HRegionInterface.class); - Mockito.when(hri.bulkLoadHFiles(Mockito.anyList(), (byte [])Mockito.any())). - thenThrow(new IOException("injecting bulk load error")); + Mockito.when(hri.bulkLoadHFiles(Mockito.anyList(), (byte [])Mockito.any(), + Mockito.anyBoolean())).thenThrow(new IOException("injecting bulk load error")); Mockito.when(c.getHRegionConnection(Mockito.anyString(), Mockito.anyInt())). thenReturn(hri); return c; @@ -295,8 +323,8 @@ public void testSplitWhileBulkLoadPhase() throws Exception { // Now let's cause trouble. This will occur after checks and cause bulk // files to fail when attempt to atomically import. This is recoverable. final AtomicInteger attemptedCalls = new AtomicInteger(); - LoadIncrementalHFiles lih2 = new LoadIncrementalHFiles( - util.getConfiguration()) { + LoadIncrementalHFiles lih2 = + new LoadIncrementalHFiles(util.getConfiguration(), useSecureHBaseOverride) { protected void bulkLoadPhase(final HTable htable, final HConnection conn, ExecutorService pool, Deque queue, @@ -336,8 +364,8 @@ public void testGroupOrSplitPresplit() throws Exception { forceSplit(table); final AtomicInteger countedLqis= new AtomicInteger(); - LoadIncrementalHFiles lih = new LoadIncrementalHFiles( - util.getConfiguration()) { + LoadIncrementalHFiles lih = + new LoadIncrementalHFiles(util.getConfiguration(), useSecureHBaseOverride) { protected List groupOrSplit( Multimap regionGroups, final LoadQueueItem item, final HTable htable, @@ -368,8 +396,8 @@ public void testGroupOrSplitFailure() throws Exception { String table = "groupOrSplitFailure"; setupTable(table, 10); - LoadIncrementalHFiles lih = new LoadIncrementalHFiles( - util.getConfiguration()) { + LoadIncrementalHFiles lih = + new LoadIncrementalHFiles(util.getConfiguration(), useSecureHBaseOverride) { int i = 0; protected List groupOrSplit( @@ -392,6 +420,64 @@ protected List groupOrSplit( fail("doBulkLoad should have thrown an exception"); } + + @Test + public void testGroupOrSplitWhenRegionHoleExistsInMeta() throws Exception { + String tableName = "testGroupOrSplitWhenRegionHoleExistsInMeta"; + byte[][] SPLIT_KEYS = new byte[][] { Bytes.toBytes("row_00000100") }; + + setupTableWithSplitkeys(tableName, 10, SPLIT_KEYS); + HTable table = new HTable(util.getConfiguration(), Bytes.toBytes(tableName)); + Path dir = buildBulkFiles(tableName, 2); + + final AtomicInteger countedLqis = new AtomicInteger(); + LoadIncrementalHFiles loader = new LoadIncrementalHFiles( + util.getConfiguration()) { + + protected List groupOrSplit( + Multimap regionGroups, + final LoadQueueItem item, final HTable htable, + final Pair startEndKeys) throws IOException { + List lqis = super.groupOrSplit(regionGroups, item, htable, startEndKeys); + if (lqis != null) { + countedLqis.addAndGet(lqis.size()); + } + return lqis; + } + }; + + // do bulkload when there is no region hole in hbase:meta. + try { + loader.doBulkLoad(dir, table); + } catch (Exception e) { + LOG.error("exeception=", e); + } + // check if all the data are loaded into the table. + this.assertExpectedTable(tableName, ROWCOUNT, 2); + + dir = buildBulkFiles(tableName, 3); + + // Mess it up by leaving a hole in the hbase:meta + CatalogTracker ct = new CatalogTracker(util.getConfiguration()); + List regionInfos = MetaReader.getTableRegions(ct, Bytes.toBytes(tableName)); + for (HRegionInfo regionInfo : regionInfos) { + if (Bytes.equals(regionInfo.getStartKey(), HConstants.EMPTY_BYTE_ARRAY)) { + MetaEditor.deleteRegion(ct, regionInfo); + break; + } + } + + try { + loader.doBulkLoad(dir, table); + } catch (Exception e) { + LOG.error("exeception=", e); + assertTrue("IOException expected", e instanceof IOException); + } + + table.close(); + + this.assertExpectedTable(tableName, ROWCOUNT, 2); + } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultiTableInputFormat.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultiTableInputFormat.java new file mode 100644 index 000000000000..89802f249ad3 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultiTableInputFormat.java @@ -0,0 +1,254 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.NavigableMap; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileUtil; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.io.ImmutableBytesWritable; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapreduce.Reducer; +import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Tests various scan start and stop row scenarios. This is set in a scan and + * tested in a MapReduce job to see if that is handed over and done properly + * too. + */ +@Category(LargeTests.class) +public class TestMultiTableInputFormat { + + static final Log LOG = LogFactory.getLog(TestMultiTableInputFormat.class); + static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + static final String TABLE_NAME = "scantest"; + static final byte[] INPUT_FAMILY = Bytes.toBytes("contents"); + static final String KEY_STARTROW = "startRow"; + static final String KEY_LASTROW = "stpRow"; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + // switch TIF to log at DEBUG level + TEST_UTIL.enableDebug(MultiTableInputFormat.class); + TEST_UTIL.enableDebug(MultiTableInputFormatBase.class); + // start mini hbase cluster + TEST_UTIL.startMiniCluster(3); + // create and fill table + for (int i = 0; i < 3; i++) { + HTable table = + TEST_UTIL.createTable(Bytes.toBytes(TABLE_NAME + String.valueOf(i)), + INPUT_FAMILY); + TEST_UTIL.createMultiRegions(table, INPUT_FAMILY); + TEST_UTIL.loadTable(table, INPUT_FAMILY); + } + // start MR cluster + TEST_UTIL.startMiniMapReduceCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniMapReduceCluster(); + TEST_UTIL.shutdownMiniCluster(); + } + + @After + public void tearDown() throws Exception { + Configuration c = TEST_UTIL.getConfiguration(); + FileUtil.fullyDelete(new File(c.get("hadoop.tmp.dir"))); + } + + /** + * Pass the key and value to reducer. + */ + public static class ScanMapper extends + TableMapper { + /** + * Pass the key and value to reduce. + * + * @param key The key, here "aaa", "aab" etc. + * @param value The value is the same as the key. + * @param context The task context. + * @throws IOException When reading the rows fails. + */ + @Override + public void map(ImmutableBytesWritable key, Result value, Context context) + throws IOException, InterruptedException { + if (value.size() != 1) { + throw new IOException("There should only be one input column"); + } + Map>> cf = + value.getMap(); + if (!cf.containsKey(INPUT_FAMILY)) { + throw new IOException("Wrong input columns. Missing: '" + + Bytes.toString(INPUT_FAMILY) + "'."); + } + String val = Bytes.toStringBinary(value.getValue(INPUT_FAMILY, null)); + LOG.debug("map: key -> " + Bytes.toStringBinary(key.get()) + + ", value -> " + val); + context.write(key, key); + } + } + + /** + * Checks the last and first keys seen against the scanner boundaries. + */ + public static class ScanReducer + extends + Reducer { + private String first = null; + private String last = null; + + protected void reduce(ImmutableBytesWritable key, + Iterable values, Context context) + throws IOException, InterruptedException { + int count = 0; + for (ImmutableBytesWritable value : values) { + String val = Bytes.toStringBinary(value.get()); + LOG.debug("reduce: key[" + count + "] -> " + + Bytes.toStringBinary(key.get()) + ", value -> " + val); + if (first == null) first = val; + last = val; + count++; + } + assertEquals(3, count); + } + + protected void cleanup(Context context) throws IOException, + InterruptedException { + Configuration c = context.getConfiguration(); + String startRow = c.get(KEY_STARTROW); + String lastRow = c.get(KEY_LASTROW); + LOG.info("cleanup: first -> \"" + first + "\", start row -> \"" + + startRow + "\""); + LOG.info("cleanup: last -> \"" + last + "\", last row -> \"" + lastRow + + "\""); + if (startRow != null && startRow.length() > 0) { + assertEquals(startRow, first); + } + if (lastRow != null && lastRow.length() > 0) { + assertEquals(lastRow, last); + } + } + } + + @Test + public void testScanEmptyToEmpty() throws IOException, InterruptedException, + ClassNotFoundException { + testScan(null, null, null); + } + + @Test + public void testScanEmptyToAPP() throws IOException, InterruptedException, + ClassNotFoundException { + testScan(null, "app", "apo"); + } + + @Test + public void testScanOBBToOPP() throws IOException, InterruptedException, + ClassNotFoundException { + testScan("obb", "opp", "opo"); + } + + @Test + public void testScanOPPToEmpty() throws IOException, InterruptedException, + ClassNotFoundException { + testScan("opp", null, "zzz"); + } + + @Test + public void testScanYZYToEmpty() throws IOException, InterruptedException, + ClassNotFoundException { + testScan("yzy", null, "zzz"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + private void testScan(String start, String stop, String last) + throws IOException, InterruptedException, ClassNotFoundException { + String jobName = + "Scan" + (start != null ? start.toUpperCase() : "Empty") + "To" + + (stop != null ? stop.toUpperCase() : "Empty"); + LOG.info("Before map/reduce startup - job " + jobName); + Configuration c = new Configuration(TEST_UTIL.getConfiguration()); + + c.set(KEY_STARTROW, start != null ? start : ""); + c.set(KEY_LASTROW, last != null ? last : ""); + + List scans = new ArrayList(); + + for(int i=0; i<3; i++){ + Scan scan = new Scan(); + + scan.addFamily(INPUT_FAMILY); + scan.setAttribute(Scan.SCAN_ATTRIBUTES_TABLE_NAME, Bytes.toBytes(TABLE_NAME + i)); + + if (start != null) { + scan.setStartRow(Bytes.toBytes(start)); + } + if (stop != null) { + scan.setStopRow(Bytes.toBytes(stop)); + } + + scans.add(scan); + + LOG.info("scan before: " + scan); + } + + Job job = new Job(c, jobName); + + TableMapReduceUtil.initTableMapperJob(scans, ScanMapper.class, + ImmutableBytesWritable.class, ImmutableBytesWritable.class, job); + job.setReducerClass(ScanReducer.class); + job.setNumReduceTasks(1); // one to get final "first" and "last" key + FileOutputFormat.setOutputPath(job, new Path(job.getJobName())); + LOG.info("Started " + job.getJobName()); + job.waitForCompletion(true); + assertTrue(job.isSuccessful()); + LOG.info("After map/reduce completion - job " + jobName); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java similarity index 93% rename from src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java rename to src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java index cc5b1df39c60..f77b7ebe041c 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java @@ -19,6 +19,7 @@ import java.io.File; import java.io.IOException; +import java.util.Iterator; import java.util.Map; import java.util.NavigableMap; @@ -28,7 +29,6 @@ import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; @@ -52,23 +52,21 @@ * a particular cell, and write it back to the table. */ @Category(LargeTests.class) -public class TestMulitthreadedTableMapper { - private static final Log LOG = LogFactory.getLog(TestMulitthreadedTableMapper.class); +public class TestMultithreadedTableMapper { + private static final Log LOG = LogFactory.getLog(TestMultithreadedTableMapper.class); private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); - static final String MULTI_REGION_TABLE_NAME = "mrtest"; + static final byte[] MULTI_REGION_TABLE_NAME = Bytes.toBytes("mrtest"); static final byte[] INPUT_FAMILY = Bytes.toBytes("contents"); static final byte[] OUTPUT_FAMILY = Bytes.toBytes("text"); static final int NUMBER_OF_THREADS = 10; @BeforeClass public static void beforeClass() throws Exception { - HTableDescriptor desc = new HTableDescriptor(MULTI_REGION_TABLE_NAME); - desc.addFamily(new HColumnDescriptor(INPUT_FAMILY)); - desc.addFamily(new HColumnDescriptor(OUTPUT_FAMILY)); UTIL.startMiniCluster(); - HBaseAdmin admin = new HBaseAdmin(UTIL.getConfiguration()); - admin.createTable(desc, HBaseTestingUtility.KEYS); + HTable table = UTIL.createTable(MULTI_REGION_TABLE_NAME, new byte[][] {INPUT_FAMILY, OUTPUT_FAMILY}); + UTIL.createMultiRegions(table, INPUT_FAMILY); + UTIL.loadTable(table, INPUT_FAMILY); UTIL.startMiniMapReduceCluster(); } @@ -149,7 +147,7 @@ private void runTestOnTable(HTable table) IdentityTableReducer.class, job); FileOutputFormat.setOutputPath(job, new Path("test")); LOG.info("Started " + Bytes.toString(table.getTableName())); - job.waitForCompletion(true); + assertTrue(job.waitForCompletion(true)); LOG.info("After map/reduce completion"); // verify map-reduce results verify(Bytes.toString(table.getTableName())); @@ -203,7 +201,10 @@ private void verifyAttempt(final HTable table) scan.addFamily(OUTPUT_FAMILY); ResultScanner scanner = table.getScanner(scan); try { - for (Result r : scanner) { + Iterator itr = scanner.iterator(); + assertTrue(itr.hasNext()); + while(itr.hasNext()) { + Result r = itr.next(); if (LOG.isDebugEnabled()) { if (r.size() > 2 ) { throw new IOException("Too many results, expected 2 got " + diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSimpleTotalOrderPartitioner.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSimpleTotalOrderPartitioner.java index 7ec759a391c0..ac65c6f3ef77 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSimpleTotalOrderPartitioner.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSimpleTotalOrderPartitioner.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan1.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan1.java new file mode 100644 index 000000000000..77ea47a290e5 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan1.java @@ -0,0 +1,99 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import java.io.IOException; + +import org.apache.hadoop.hbase.LargeTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * TestTableInputFormatScan part 1. + * @see TestTableInputFormatScanBase + */ +@Category(LargeTests.class) +public class TestTableInputFormatScan1 extends TestTableInputFormatScanBase { + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToEmpty() + throws IOException, InterruptedException, ClassNotFoundException { + testScan(null, null, null); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToAPP() + throws IOException, InterruptedException, ClassNotFoundException { + testScan(null, "app", "apo"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToBBA() + throws IOException, InterruptedException, ClassNotFoundException { + testScan(null, "bba", "baz"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToBBB() + throws IOException, InterruptedException, ClassNotFoundException { + testScan(null, "bbb", "bba"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToOPP() + throws IOException, InterruptedException, ClassNotFoundException { + testScan(null, "opp", "opo"); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan2.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan2.java new file mode 100644 index 000000000000..f35bbd112083 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan2.java @@ -0,0 +1,117 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import java.io.IOException; + +import org.apache.hadoop.hbase.LargeTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * TestTableInputFormatScan part 2. + * @see TestTableInputFormatScanBase + */ +@Category(LargeTests.class) +public class TestTableInputFormatScan2 extends TestTableInputFormatScanBase { + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanOBBToOPP() + throws IOException, InterruptedException, ClassNotFoundException { + testScan("obb", "opp", "opo"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanOBBToQPP() + throws IOException, InterruptedException, ClassNotFoundException { + testScan("obb", "qpp", "qpo"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanOPPToEmpty() + throws IOException, InterruptedException, ClassNotFoundException { + testScan("opp", null, "zzz"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanYYXToEmpty() + throws IOException, InterruptedException, ClassNotFoundException { + testScan("yyx", null, "zzz"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanYYYToEmpty() + throws IOException, InterruptedException, ClassNotFoundException { + testScan("yyy", null, "zzz"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanYZYToEmpty() + throws IOException, InterruptedException, ClassNotFoundException { + testScan("yzy", null, "zzz"); + } + + @Test + public void testScanFromConfiguration() + throws IOException, InterruptedException, ClassNotFoundException { + testScanFromConfiguration("bba", "bbd", "bbc"); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java similarity index 64% rename from src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan.java rename to src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java index afead7d4a897..76596046ce72 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,7 +18,9 @@ */ package org.apache.hadoop.hbase.mapreduce; -import java.io.File; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + import java.io.IOException; import java.util.Map; import java.util.NavigableMap; @@ -27,10 +28,8 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseTestingUtility; -import org.apache.hadoop.hbase.LargeTests; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.Scan; @@ -40,25 +39,23 @@ import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; -import org.junit.After; import org.junit.AfterClass; -import org.junit.Before; import org.junit.BeforeClass; -import org.junit.Test; -import org.junit.experimental.categories.Category; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; /** + *

* Tests various scan start and stop row scenarios. This is set in a scan and * tested in a MapReduce job to see if that is handed over and done properly * too. + *

+ *

+ * This test is broken into two parts in order to side-step the test timeout + * period of 900, as documented in HBASE-8326. + *

*/ -@Category(LargeTests.class) -public class TestTableInputFormatScan { +public abstract class TestTableInputFormatScanBase { - static final Log LOG = LogFactory.getLog(TestTableInputFormatScan.class); + static final Log LOG = LogFactory.getLog(TestTableInputFormatScanBase.class); static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); static final byte[] TABLE_NAME = Bytes.toBytes("scantest"); @@ -165,155 +162,6 @@ protected void cleanup(Context context) } - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanEmptyToEmpty() - throws IOException, InterruptedException, ClassNotFoundException { - testScan(null, null, null); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanEmptyToAPP() - throws IOException, InterruptedException, ClassNotFoundException { - testScan(null, "app", "apo"); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanEmptyToBBA() - throws IOException, InterruptedException, ClassNotFoundException { - testScan(null, "bba", "baz"); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanEmptyToBBB() - throws IOException, InterruptedException, ClassNotFoundException { - testScan(null, "bbb", "bba"); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanEmptyToOPP() - throws IOException, InterruptedException, ClassNotFoundException { - testScan(null, "opp", "opo"); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanOBBToOPP() - throws IOException, InterruptedException, ClassNotFoundException { - testScan("obb", "opp", "opo"); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanOBBToQPP() - throws IOException, InterruptedException, ClassNotFoundException { - testScan("obb", "qpp", "qpo"); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanOPPToEmpty() - throws IOException, InterruptedException, ClassNotFoundException { - testScan("opp", null, "zzz"); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanYYXToEmpty() - throws IOException, InterruptedException, ClassNotFoundException { - testScan("yyx", null, "zzz"); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanYYYToEmpty() - throws IOException, InterruptedException, ClassNotFoundException { - testScan("yyy", null, "zzz"); - } - - /** - * Tests a MR scan using specific start and stop rows. - * - * @throws IOException - * @throws ClassNotFoundException - * @throws InterruptedException - */ - @Test - public void testScanYZYToEmpty() - throws IOException, InterruptedException, ClassNotFoundException { - testScan("yzy", null, "zzz"); - } - - @Test - public void testScanFromConfiguration() - throws IOException, InterruptedException, ClassNotFoundException { - testScanFromConfiguration("bba", "bbd", "bbc"); - } - /** * Tests an MR Scan initialized from properties set in the Configuration. * @@ -321,7 +169,7 @@ public void testScanFromConfiguration() * @throws ClassNotFoundException * @throws InterruptedException */ - private void testScanFromConfiguration(String start, String stop, String last) + protected void testScanFromConfiguration(String start, String stop, String last) throws IOException, InterruptedException, ClassNotFoundException { String jobName = "ScanFromConfig" + (start != null ? start.toUpperCase() : "Empty") + "To" + (stop != null ? stop.toUpperCase() : "Empty"); @@ -347,8 +195,8 @@ private void testScanFromConfiguration(String start, String stop, String last) job.setInputFormatClass(TableInputFormat.class); job.setNumReduceTasks(1); FileOutputFormat.setOutputPath(job, new Path(job.getJobName())); - job.waitForCompletion(true); - assertTrue(job.isComplete()); + TableMapReduceUtil.addDependencyJars(job); + assertTrue(job.waitForCompletion(true)); } /** @@ -358,7 +206,7 @@ private void testScanFromConfiguration(String start, String stop, String last) * @throws ClassNotFoundException * @throws InterruptedException */ - private void testScan(String start, String stop, String last) + protected void testScan(String start, String stop, String last) throws IOException, InterruptedException, ClassNotFoundException { String jobName = "Scan" + (start != null ? start.toUpperCase() : "Empty") + "To" + (stop != null ? stop.toUpperCase() : "Empty"); @@ -383,13 +231,9 @@ private void testScan(String start, String stop, String last) job.setNumReduceTasks(1); // one to get final "first" and "last" key FileOutputFormat.setOutputPath(job, new Path(job.getJobName())); LOG.info("Started " + job.getJobName()); - job.waitForCompletion(true); - assertTrue(job.isComplete()); + assertTrue(job.waitForCompletion(true)); LOG.info("After map/reduce completion - job " + jobName); } - @org.junit.Rule - public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = - new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); } diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java index 9268d6d46880..8f11e03b8349 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -21,6 +20,7 @@ import java.io.File; import java.io.IOException; +import java.util.Iterator; import java.util.Map; import java.util.NavigableMap; @@ -30,7 +30,6 @@ import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; @@ -59,18 +58,16 @@ public class TestTableMapReduce { private static final Log LOG = LogFactory.getLog(TestTableMapReduce.class); private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); - static final String MULTI_REGION_TABLE_NAME = "mrtest"; + static final byte[] MULTI_REGION_TABLE_NAME = Bytes.toBytes("mrtest"); static final byte[] INPUT_FAMILY = Bytes.toBytes("contents"); static final byte[] OUTPUT_FAMILY = Bytes.toBytes("text"); @BeforeClass public static void beforeClass() throws Exception { - HTableDescriptor desc = new HTableDescriptor(MULTI_REGION_TABLE_NAME); - desc.addFamily(new HColumnDescriptor(INPUT_FAMILY)); - desc.addFamily(new HColumnDescriptor(OUTPUT_FAMILY)); UTIL.startMiniCluster(); - HBaseAdmin admin = new HBaseAdmin(UTIL.getConfiguration()); - admin.createTable(desc, HBaseTestingUtility.KEYS); + HTable table = UTIL.createTable(MULTI_REGION_TABLE_NAME, new byte[][] {INPUT_FAMILY, OUTPUT_FAMILY}); + UTIL.createMultiRegions(table, INPUT_FAMILY); + UTIL.loadTable(table, INPUT_FAMILY); UTIL.startMiniMapReduceCluster(); } @@ -150,7 +147,7 @@ private void runTestOnTable(HTable table) IdentityTableReducer.class, job); FileOutputFormat.setOutputPath(job, new Path("test")); LOG.info("Started " + Bytes.toString(table.getTableName())); - job.waitForCompletion(true); + assertTrue(job.waitForCompletion(true)); LOG.info("After map/reduce completion"); // verify map-reduce results @@ -204,7 +201,10 @@ private void verifyAttempt(final HTable table) throws IOException, NullPointerEx scan.addFamily(OUTPUT_FAMILY); ResultScanner scanner = table.getScanner(scan); try { - for (Result r : scanner) { + Iterator itr = scanner.iterator(); + assertTrue(itr.hasNext()); + while(itr.hasNext()) { + Result r = itr.next(); if (LOG.isDebugEnabled()) { if (r.size() > 2 ) { throw new IOException("Too many results, expected 2 got " + @@ -262,19 +262,14 @@ private void verifyAttempt(final HTable table) throws IOException, NullPointerEx /** * Test that we add tmpjars correctly including the ZK jar. */ + @Test public void testAddDependencyJars() throws Exception { Job job = new Job(); TableMapReduceUtil.addDependencyJars(job); String tmpjars = job.getConfiguration().get("tmpjars"); - System.err.println("tmpjars: " + tmpjars); assertTrue(tmpjars.contains("zookeeper")); - assertFalse(tmpjars.contains("guava")); - - System.err.println("appending guava jar"); - TableMapReduceUtil.addDependencyJars(job.getConfiguration(), - com.google.common.base.Function.class); - tmpjars = job.getConfiguration().get("tmpjars"); + assertTrue(tmpjars.contains("protobuf")); assertTrue(tmpjars.contains("guava")); } diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java new file mode 100644 index 000000000000..2aac1734f4c5 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java @@ -0,0 +1,300 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.mapreduce; + +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.util.Arrays; +import java.util.List; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.io.ImmutableBytesWritable; +import org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat.TableSnapshotRegionSplit; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.mapreduce.InputSplit; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapreduce.RecordReader; +import org.apache.hadoop.mapreduce.Reducer; +import org.apache.hadoop.mapreduce.TaskAttemptContext; +import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat; +import org.junit.After; +import org.junit.Assert; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(LargeTests.class) +public class TestTableSnapshotInputFormat { + + private final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static final int NUM_REGION_SERVERS = 2; + private static final byte[][] FAMILIES = {Bytes.toBytes("f1"), Bytes.toBytes("f2")}; + public static byte[] bbb = Bytes.toBytes("bbb"); + public static byte[] yyy = Bytes.toBytes("yyy"); + + public void setupCluster() throws Exception { + setupConf(UTIL.getConfiguration()); + UTIL.startMiniCluster(NUM_REGION_SERVERS); + } + + public void tearDownCluster() throws Exception { + UTIL.shutdownMiniCluster(); + } + + private static void setupConf(Configuration conf) { + // Enable snapshot + conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + } + + @After + public void tearDown() throws Exception { + } + + public static enum TestTableSnapshotCounters { + VALIDATION_ERROR + } + + public static class TestTableSnapshotMapper + extends TableMapper { + @Override + protected void map(ImmutableBytesWritable key, Result value, + Context context) throws IOException, InterruptedException { + // Validate a single row coming from the snapshot, and emit the row key + verifyRowFromMap(key, value); + context.write(key, NullWritable.get()); + } + } + + public static class TestTableSnapshotReducer + extends Reducer { + HBaseTestingUtility.SeenRowTracker rowTracker = new HBaseTestingUtility.SeenRowTracker(bbb, yyy); + @Override + protected void reduce(ImmutableBytesWritable key, Iterable values, + Context context) throws IOException, InterruptedException { + rowTracker.addRow(key.get()); + } + + @Override + protected void cleanup(Context context) throws IOException, + InterruptedException { + rowTracker.validate(); + } + } + + public static void createTableAndSnapshot(HBaseTestingUtility util, byte[] tableName, + String snapshotName, int numRegions) + throws Exception { + try { + util.deleteTable(tableName); + } catch(Exception ex) { + // ignore + } + + if (numRegions > 1) { + util.createTable(tableName, FAMILIES, 1, bbb, yyy, numRegions); + } else { + util.createTable(tableName, FAMILIES); + } + HBaseAdmin admin = util.getHBaseAdmin(); + + // put some stuff in the table + HTable table = new HTable(util.getConfiguration(), tableName); + util.loadTable(table, FAMILIES); + + Path rootDir = new Path(util.getConfiguration().get(HConstants.HBASE_DIR)); + FileSystem fs = rootDir.getFileSystem(util.getConfiguration()); + + SnapshotTestingUtils.createSnapshotAndValidate(admin, Bytes.toString(tableName), + Arrays.asList(FAMILIES), null, snapshotName, rootDir, fs, true); + + // load different values + byte[] value = Bytes.toBytes("after_snapshot_value"); + util.loadTable(table, FAMILIES, value); + + // cause flush to create new files in the region + admin.flush(tableName); + table.close(); + } + + @Test + public void testWithMockedMapReduceSingleRegion() throws Exception { + testWithMockedMapReduce(UTIL, "testWithMockedMapReduceSingleRegion", 1, 1); + } + + @Test + public void testWithMockedMapReduceMultiRegion() throws Exception { + testWithMockedMapReduce(UTIL, "testWithMockedMapReduceMultiRegion", 10, 8); + } + + public void testWithMockedMapReduce(HBaseTestingUtility util, String snapshotName, int numRegions, int expectedNumSplits) + throws Exception { + setupCluster(); + byte[] tableName = Bytes.toBytes("testWithMockedMapReduce"); + try { + createTableAndSnapshot(util, tableName, snapshotName, numRegions); + + Job job = new Job(util.getConfiguration()); + Path tmpTableDir = util.getDataTestDir(snapshotName); + Scan scan = new Scan(bbb, yyy); // limit the scan + + TableMapReduceUtil.initTableSnapshotMapperJob(snapshotName, + scan, TestTableSnapshotMapper.class, ImmutableBytesWritable.class, + NullWritable.class, job, false, tmpTableDir); + + verifyWithMockedMapReduce(job, numRegions, expectedNumSplits, bbb, yyy); + + } finally { + util.getHBaseAdmin().deleteSnapshot(snapshotName); + util.deleteTable(tableName); + tearDownCluster(); + } + } + + private void verifyWithMockedMapReduce(Job job, int numRegions, int expectedNumSplits, + byte[] startRow, byte[] stopRow) + throws IOException, InterruptedException { + TableSnapshotInputFormat tsif = new TableSnapshotInputFormat(); + List splits = tsif.getSplits(job); + + Assert.assertEquals(expectedNumSplits, splits.size()); + + HBaseTestingUtility.SeenRowTracker rowTracker = new HBaseTestingUtility.SeenRowTracker(startRow, stopRow); + + for (int i = 0; i < splits.size(); i++) { + // validate input split + InputSplit split = splits.get(i); + Assert.assertTrue(split instanceof TableSnapshotRegionSplit); + + // validate record reader + TaskAttemptContext taskAttemptContext = mock(TaskAttemptContext.class); + when(taskAttemptContext.getConfiguration()).thenReturn(job.getConfiguration()); + RecordReader rr = tsif.createRecordReader(split, taskAttemptContext); + rr.initialize(split, taskAttemptContext); + + // validate we can read all the data back + while (rr.nextKeyValue()) { + byte[] row = rr.getCurrentKey().get(); + verifyRowFromMap(rr.getCurrentKey(), rr.getCurrentValue()); + rowTracker.addRow(row); + } + + rr.close(); + } + + // validate all rows are seen + rowTracker.validate(); + } + + public static void verifyRowFromMap(ImmutableBytesWritable key, Result result) throws IOException { + byte[] row = key.get(); + for (KeyValue kv : result.list()) { + //assert that all Cells in the Result have the same key + Assert.assertEquals(0, Bytes.compareTo(row, 0, row.length, + kv.getBuffer(), kv.getRowOffset(), kv.getRowLength())); + } + + for (int j = 0; j < FAMILIES.length; j++) { + byte[] actual = result.getValue(FAMILIES[j], null); + Assert.assertArrayEquals("Row in snapshot does not match, expected:" + Bytes.toString(row) + + " ,actual:" + Bytes.toString(actual), row, actual); + } + } + + @Test + public void testWithMapReduceSingleRegion() throws Exception { + testWithMapReduce(UTIL, "testWithMapReduceSingleRegion", 1, 1, false); + } + + @Test + public void testWithMapReduceMultiRegion() throws Exception { + testWithMapReduce(UTIL, "testWithMapReduceMultiRegion", 10, 8, false); + } + + @Test + // run the MR job while HBase is offline + public void testWithMapReduceAndOfflineHBaseMultiRegion() throws Exception { + testWithMapReduce(UTIL, "testWithMapReduceAndOfflineHBaseMultiRegion", 10, 8, true); + } + + private void testWithMapReduce(HBaseTestingUtility util, String snapshotName, + int numRegions, int expectedNumSplits, boolean shutdownCluster) throws Exception { + setupCluster(); + util.startMiniMapReduceCluster(); + try { + Path tableDir = util.getDataTestDir(snapshotName); + byte[] tableName = Bytes.toBytes("testWithMapReduce"); + doTestWithMapReduce(util, tableName, snapshotName, tableDir, numRegions, + expectedNumSplits, shutdownCluster); + } finally { + util.shutdownMiniMapReduceCluster(); + tearDownCluster(); + } + } + + // this is also called by the IntegrationTestTableSnapshotInputFormat + public static void doTestWithMapReduce(HBaseTestingUtility util, byte[] tableName, + String snapshotName, Path tableDir, int numRegions, int expectedNumSplits, boolean shutdownCluster) + throws Exception { + + //create the table and snapshot + createTableAndSnapshot(util, tableName, snapshotName, numRegions); + + if (shutdownCluster) { + util.shutdownMiniHBaseCluster(); + } + + try { + // create the job + Job job = new Job(util.getConfiguration()); + Scan scan = new Scan(bbb, yyy); // limit the scan + + job.setJarByClass(util.getClass()); + TableMapReduceUtil.addDependencyJars(job.getConfiguration(), TestTableSnapshotInputFormat.class); + + TableMapReduceUtil.initTableSnapshotMapperJob(snapshotName, + scan, TestTableSnapshotMapper.class, ImmutableBytesWritable.class, + NullWritable.class, job, true, tableDir); + + job.setReducerClass(TestTableSnapshotInputFormat.TestTableSnapshotReducer.class); + job.setNumReduceTasks(1); + job.setOutputFormatClass(NullOutputFormat.class); + + Assert.assertTrue(job.waitForCompletion(true)); + } finally { + if (!shutdownCluster) { + util.getHBaseAdmin().deleteSnapshot(snapshotName); + util.deleteTable(tableName); + } + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormatScan.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormatScan.java new file mode 100644 index 000000000000..08c4dcb612aa --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormatScan.java @@ -0,0 +1,208 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.UUID; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.io.ImmutableBytesWritable; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + *

+ * Tests scanning a snapshot. Tests various scan start and stop row scenarios. + * This is set in a scan and tested in a MapReduce job to see if that is handed + * over and done properly too. + *

+ */ +@Category(LargeTests.class) +public class TestTableSnapshotInputFormatScan { + + static final Log LOG = LogFactory.getLog(TestTableSnapshotInputFormatScan.class); + static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + static final byte[] TABLE_NAME = Bytes.toBytes("scantest"); + static final byte[] SNAPSHOT_NAME = Bytes.toBytes("scantest_snaphot"); + static final byte[] INPUT_FAMILY = Bytes.toBytes("contents"); + static final String KEY_STARTROW = "startRow"; + static final String KEY_LASTROW = "stpRow"; + + private static HTable table = null; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + // config snapshot support + TEST_UTIL.getConfiguration().setBoolean( + SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + TEST_UTIL.getConfiguration().setBoolean( + "hbase.master.enabletable.roundrobin", true); + + // switch TIF to log at DEBUG level + TEST_UTIL.enableDebug(TableSnapshotInputFormat.class); + + // start mini hbase cluster + TEST_UTIL.startMiniCluster(3); + + // create and fill table + table = TEST_UTIL.createTable(TABLE_NAME, INPUT_FAMILY); + TEST_UTIL.createMultiRegions(table, INPUT_FAMILY); + TEST_UTIL.loadTable(table, INPUT_FAMILY); + TEST_UTIL.getHBaseAdmin().disableTable(TABLE_NAME); + TEST_UTIL.getHBaseAdmin().snapshot(SNAPSHOT_NAME, TABLE_NAME); + TEST_UTIL.getHBaseAdmin().enableTable(TABLE_NAME); + + // start MR cluster + TEST_UTIL.startMiniMapReduceCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniMapReduceCluster(); + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToEmpty() throws IOException, InterruptedException, + ClassNotFoundException { + testScan(null, null, null); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToAPP() throws IOException, InterruptedException, + ClassNotFoundException { + testScan(null, "app", "apo"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToBBA() throws IOException, InterruptedException, + ClassNotFoundException { + testScan(null, "bba", "baz"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToBBB() throws IOException, InterruptedException, + ClassNotFoundException { + testScan(null, "bbb", "bba"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + @Test + public void testScanEmptyToOPP() throws IOException, InterruptedException, + ClassNotFoundException { + testScan(null, "opp", "opo"); + } + + /** + * Tests a MR scan using specific start and stop rows. + * + * @throws IOException + * @throws ClassNotFoundException + * @throws InterruptedException + */ + protected void testScan(String start, String stop, String last) + throws IOException, InterruptedException, ClassNotFoundException { + String jobName = "Scan" + (start != null ? start.toUpperCase() : "Empty") + + "To" + (stop != null ? stop.toUpperCase() : "Empty"); + LOG.info("Before map/reduce startup - job " + jobName); + Configuration c = new Configuration(TEST_UTIL.getConfiguration()); + Scan scan = new Scan(); + scan.addFamily(INPUT_FAMILY); + if (start != null) { + scan.setStartRow(Bytes.toBytes(start)); + } + c.set(KEY_STARTROW, start != null ? start : ""); + if (stop != null) { + scan.setStopRow(Bytes.toBytes(stop)); + } + c.set(KEY_LASTROW, last != null ? last : ""); + LOG.info("scan before: " + scan); + Job job = new Job(c, jobName); + + FileSystem fs = FileSystem.get(c); + Path tmpDir = new Path("/" + UUID.randomUUID()); + fs.mkdirs(tmpDir); + try { + TableMapReduceUtil.initTableSnapshotMapperJob(Bytes.toString(SNAPSHOT_NAME), + scan, TestTableInputFormatScanBase.ScanMapper.class, + ImmutableBytesWritable.class, ImmutableBytesWritable.class, job, + false, tmpDir); + job.setReducerClass(TestTableInputFormatScanBase.ScanReducer.class); + job.setNumReduceTasks(1); // one to get final "first" and "last" key + FileOutputFormat.setOutputPath(job, new Path(job.getJobName())); + LOG.info("Started " + job.getJobName()); + assertTrue(job.waitForCompletion(true)); + LOG.info("After map/reduce completion - job " + jobName); + } finally { + fs.delete(tmpDir, true); + } + + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java index 74ea4ec35fb4..15d90ec834d8 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java new file mode 100644 index 000000000000..93653afa703d --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java @@ -0,0 +1,103 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertNull; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.client.Delete; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Basic test for the WALPlayer M/R tool + */ +@Category(LargeTests.class) +public class TestWALPlayer { + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static MiniHBaseCluster cluster; + + @BeforeClass + public static void beforeClass() throws Exception { + cluster = TEST_UTIL.startMiniCluster(); + TEST_UTIL.startMiniMapReduceCluster(); + } + + @AfterClass + public static void afterClass() throws Exception { + TEST_UTIL.shutdownMiniMapReduceCluster(); + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * Simple end-to-end test + * @throws Exception + */ + @Test + public void testWALPlayer() throws Exception { + final byte[] TABLENAME1 = Bytes.toBytes("testWALPlayer1"); + final byte[] TABLENAME2 = Bytes.toBytes("testWALPlayer2"); + final byte[] FAMILY = Bytes.toBytes("family"); + final byte[] COLUMN1 = Bytes.toBytes("c1"); + final byte[] COLUMN2 = Bytes.toBytes("c2"); + final byte[] ROW = Bytes.toBytes("row"); + HTable t1 = TEST_UTIL.createTable(TABLENAME1, FAMILY); + HTable t2 = TEST_UTIL.createTable(TABLENAME2, FAMILY); + + // put a row into the first table + Put p = new Put(ROW); + p.add(FAMILY, COLUMN1, COLUMN1); + p.add(FAMILY, COLUMN2, COLUMN2); + t1.put(p); + // delete one column + Delete d = new Delete(ROW); + d.deleteColumns(FAMILY, COLUMN1); + t1.delete(d); + + // replay the WAL, map table 1 to table 2 + HLog log = cluster.getRegionServer(0).getWAL(); + log.rollWriter(); + String walInputDir = new Path(cluster.getMaster().getMasterFileSystem() + .getRootDir(), HConstants.HREGION_LOGDIR_NAME).toString(); + + WALPlayer player = new WALPlayer(TEST_UTIL.getConfiguration()); + assertEquals(0, player.run(new String[] { walInputDir, Bytes.toString(TABLENAME1), + Bytes.toString(TABLENAME2) })); + + // verify the WAL was player into table 2 + Get g = new Get(ROW); + Result r = t2.get(g); + assertEquals(1, r.size()); + assertTrue(Bytes.equals(COLUMN2, r.raw()[0].getQualifier())); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestInputSampler.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestInputSampler.java new file mode 100644 index 000000000000..65b6702efd65 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestInputSampler.java @@ -0,0 +1,165 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce.hadoopbackport; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import static org.junit.Assert.*; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.io.IntWritable; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.mapreduce.InputFormat; +import org.apache.hadoop.mapreduce.InputSplit; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapreduce.JobContext; +import org.apache.hadoop.mapreduce.RecordReader; +import org.apache.hadoop.mapreduce.TaskAttemptContext; + +/** + * The test is ported from Hadoop branch-0.23 with very small changes. + */ +@Category(SmallTests.class) +public class TestInputSampler { + + static class SequentialSplit extends InputSplit { + private int i; + SequentialSplit(int i) { + this.i = i; + } + @Override + public long getLength() { return 0; } + @Override + public String[] getLocations() { return new String[0]; } + public int getInit() { return i; } + } + + static class TestInputSamplerIF + extends InputFormat { + + final int maxDepth; + final ArrayList splits = new ArrayList(); + + TestInputSamplerIF(int maxDepth, int numSplits, int... splitInit) { + this.maxDepth = maxDepth; + assert splitInit.length == numSplits; + for (int i = 0; i < numSplits; ++i) { + splits.add(new SequentialSplit(splitInit[i])); + } + } + + @Override + public List getSplits(JobContext context) + throws IOException, InterruptedException { + return splits; + } + + @Override + public RecordReader createRecordReader( + final InputSplit split, TaskAttemptContext context) + throws IOException, InterruptedException { + return new RecordReader() { + private int maxVal; + private final IntWritable i = new IntWritable(); + @Override + public void initialize(InputSplit split, TaskAttemptContext context) + throws IOException, InterruptedException { + i.set(((SequentialSplit)split).getInit() - 1); + maxVal = i.get() + maxDepth + 1; + } + @Override + public boolean nextKeyValue() { + i.set(i.get() + 1); + return i.get() < maxVal; + } + @Override + public IntWritable getCurrentKey() { return i; } + @Override + public NullWritable getCurrentValue() { return NullWritable.get(); } + @Override + public float getProgress() { return 1.0f; } + @Override + public void close() { } + }; + } + + } + + /** + * Verify SplitSampler contract, that an equal number of records are taken + * from the first splits. + */ + @Test + @SuppressWarnings("unchecked") // IntWritable comparator not typesafe + public void testSplitSampler() throws Exception { + final int TOT_SPLITS = 15; + final int NUM_SPLITS = 5; + final int STEP_SAMPLE = 5; + final int NUM_SAMPLES = NUM_SPLITS * STEP_SAMPLE; + InputSampler.Sampler sampler = + new InputSampler.SplitSampler( + NUM_SAMPLES, NUM_SPLITS); + int inits[] = new int[TOT_SPLITS]; + for (int i = 0; i < TOT_SPLITS; ++i) { + inits[i] = i * STEP_SAMPLE; + } + Job ignored = new Job();//Job.getInstance(); + Object[] samples = sampler.getSample( + new TestInputSamplerIF(100000, TOT_SPLITS, inits), ignored); + assertEquals(NUM_SAMPLES, samples.length); + Arrays.sort(samples, new IntWritable.Comparator()); + for (int i = 0; i < NUM_SAMPLES; ++i) { + assertEquals(i, ((IntWritable)samples[i]).get()); + } + } + + /** + * Verify IntervalSampler contract, that samples are taken at regular + * intervals from the given splits. + */ + @Test + @SuppressWarnings("unchecked") // IntWritable comparator not typesafe + public void testIntervalSampler() throws Exception { + final int TOT_SPLITS = 16; + final int PER_SPLIT_SAMPLE = 4; + final int NUM_SAMPLES = TOT_SPLITS * PER_SPLIT_SAMPLE; + final double FREQ = 1.0 / TOT_SPLITS; + InputSampler.Sampler sampler = + new InputSampler.IntervalSampler( + FREQ, NUM_SAMPLES); + int inits[] = new int[TOT_SPLITS]; + for (int i = 0; i < TOT_SPLITS; ++i) { + inits[i] = i; + } + Job ignored = new Job(); + Object[] samples = sampler.getSample(new TestInputSamplerIF( + NUM_SAMPLES, TOT_SPLITS, inits), ignored); + assertEquals(NUM_SAMPLES, samples.length); + Arrays.sort(samples, new IntWritable.Comparator()); + for (int i = 0; i < NUM_SAMPLES; ++i) { + assertEquals(i, ((IntWritable)samples[i]).get()); + } + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestInputSamplerTool.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestInputSamplerTool.java new file mode 100644 index 000000000000..f90c20447c48 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestInputSamplerTool.java @@ -0,0 +1,207 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.mapreduce.hadoopbackport; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.StringReader; +import java.lang.reflect.Array; +import java.util.ArrayList; +import java.util.Arrays; + +import org.apache.commons.io.FileUtils; +import org.apache.commons.io.IOUtils; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.FileUtil; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.io.LongWritable; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.io.SequenceFile; +import org.apache.hadoop.io.Writable; +import org.apache.hadoop.util.ReflectionUtils; +import org.apache.hadoop.util.Tool; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import static org.junit.Assert.*; + +/** + * Tests {@link InputSampler} as a {@link Tool}. + */ +@Category(SmallTests.class) +public class TestInputSamplerTool { + + private static final int NUM_REDUCES = 4; + + private static final String input1Str = + "2\n" + +"...5\n" + +"......8\n"; + private static final String input2Str = + "2\n" + +".3\n" + +"..4\n" + +"...5\n" + +"....6\n" + +".....7\n" + +"......8\n" + +".......9\n"; + + private static File tempDir; + private static String input1, input2, output; + + @BeforeClass + public static void beforeClass() throws IOException { + tempDir = FileUtil.createLocalTempFile( + new File(FileUtils.getTempDirectory(), TestInputSamplerTool.class.getName() + "-tmp-"), + "", false); + tempDir.delete(); + tempDir.mkdirs(); + assertTrue(tempDir.exists()); + assertTrue(tempDir.isDirectory()); + // define files: + input1 = tempDir.getAbsolutePath() + "/input1"; + input2 = tempDir.getAbsolutePath() + "/input2"; + output = tempDir.getAbsolutePath() + "/output"; + // create 2 input files: + IOUtils.copy(new StringReader(input1Str), new FileOutputStream(input1)); + IOUtils.copy(new StringReader(input2Str), new FileOutputStream(input2)); + } + + @AfterClass + public static void afterClass() throws IOException { + final File td = tempDir; + if (td != null && td.exists()) { + FileUtil.fullyDelete(tempDir); + } + } + + @Test + public void testIncorrectParameters() throws Exception { + Tool tool = new InputSampler(new Configuration()); + + int result = tool.run(new String[] { "-r" }); + assertTrue(result != 0); + + result = tool.run(new String[] { "-r", "not-a-number" }); + assertTrue(result != 0); + + // more than one reducer is required: + result = tool.run(new String[] { "-r", "1" }); + assertTrue(result != 0); + + try { + result = tool.run(new String[] { "-inFormat", "java.lang.Object" }); + fail("ClassCastException expected"); + } catch (ClassCastException cce) { + // expected + } + + try { + result = tool.run(new String[] { "-keyClass", "java.lang.Object" }); + fail("ClassCastException expected"); + } catch (ClassCastException cce) { + // expected + } + + result = tool.run(new String[] { "-splitSample", "1", }); + assertTrue(result != 0); + + result = tool.run(new String[] { "-splitRandom", "1.0", "2", "xxx" }); + assertTrue(result != 0); + + result = tool.run(new String[] { "-splitInterval", "yyy", "5" }); + assertTrue(result != 0); + + // not enough subsequent arguments: + result = tool.run(new String[] { "-r", "2", "-splitInterval", "11.0f", "0", "input" }); + assertTrue(result != 0); + } + + @Test + public void testSplitSample() throws Exception { + Tool tool = new InputSampler(new Configuration()); + int result = tool.run(new String[] { "-r", Integer.toString(NUM_REDUCES), + "-splitSample", "10", "100", + input1, input2, output }); + assertEquals(0, result); + + Object[] partitions = readPartitions(output); + assertArrayEquals( + new LongWritable[] { new LongWritable(2L), new LongWritable(7L), new LongWritable(20L),}, + partitions); + } + + @Test + @SuppressWarnings("unchecked") + public void testSplitRamdom() throws Exception { + Tool tool = new InputSampler(new Configuration()); + int result = tool.run(new String[] { "-r", Integer.toString(NUM_REDUCES), + // Use 0.999 probability to reduce the flakiness of the test because + // the test will fail if the number of samples is less than (number of reduces + 1). + "-splitRandom", "0.999f", "20", "100", + input1, input2, output }); + assertEquals(0, result); + Object[] partitions = readPartitions(output); + // must be 3 split points since NUM_REDUCES = 4: + assertEquals(3, partitions.length); + // check that the partition array is sorted: + Object[] sortedPartitions = Arrays.copyOf(partitions, partitions.length); + Arrays.sort(sortedPartitions, new LongWritable.Comparator()); + assertArrayEquals(sortedPartitions, partitions); + } + + @Test + public void testSplitInterval() throws Exception { + Tool tool = new InputSampler(new Configuration()); + int result = tool.run(new String[] { "-r", Integer.toString(NUM_REDUCES), + "-splitInterval", "0.5f", "0", + input1, input2, output }); + assertEquals(0, result); + Object[] partitions = readPartitions(output); + assertArrayEquals(new LongWritable[] { new LongWritable(7L), new LongWritable(9L), + new LongWritable(35L),}, partitions); + } + + private Object[] readPartitions(String filePath) throws Exception { + Configuration conf = new Configuration(); + TotalOrderPartitioner.setPartitionFile(conf, new Path(filePath)); + Object[] partitions = readPartitions(FileSystem.getLocal(conf), new Path(filePath), + LongWritable.class, conf); + return partitions; + } + + private Object[] readPartitions(FileSystem fs, Path p, Class keyClass, + Configuration conf) throws IOException { + SequenceFile.Reader reader = new SequenceFile.Reader(fs, p, conf); + ArrayList parts = new ArrayList(); + Writable key = (Writable)ReflectionUtils.newInstance(keyClass, conf); + NullWritable value = NullWritable.get(); + while (reader.next(key, value)) { + parts.add(key); + key = (Writable)ReflectionUtils.newInstance(keyClass, conf); + } + reader.close(); + return parts.toArray((Object[])Array.newInstance(keyClass, parts.size())); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestJarFinder.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestJarFinder.java new file mode 100644 index 000000000000..fb56993c27a1 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestJarFinder.java @@ -0,0 +1,132 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.mapreduce.hadoopbackport; + +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Assert; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.FileOutputStream; +import java.io.FileWriter; +import java.io.IOException; +import java.io.OutputStream; +import java.io.Writer; +import java.text.MessageFormat; +import java.util.Properties; +import java.util.jar.JarInputStream; +import java.util.jar.JarOutputStream; +import java.util.jar.Manifest; + +/** + * This file was forked from hadoop/common/branches/branch-2@1350012. + */ +@Category(SmallTests.class) +public class TestJarFinder { + + @Test + public void testJar() throws Exception { + + //picking a class that is for sure in a JAR in the classpath + String jar = JarFinder.getJar(LogFactory.class); + Assert.assertTrue(new File(jar).exists()); + } + + private static void delete(File file) throws IOException { + if (file.getAbsolutePath().length() < 5) { + throw new IllegalArgumentException( + MessageFormat.format("Path [{0}] is too short, not deleting", + file.getAbsolutePath())); + } + if (file.exists()) { + if (file.isDirectory()) { + File[] children = file.listFiles(); + if (children != null) { + for (File child : children) { + delete(child); + } + } + } + if (!file.delete()) { + throw new RuntimeException( + MessageFormat.format("Could not delete path [{0}]", + file.getAbsolutePath())); + } + } + } + + @Test + public void testExpandedClasspath() throws Exception { + //picking a class that is for sure in a directory in the classpath + //in this case the JAR is created on the fly + String jar = JarFinder.getJar(TestJarFinder.class); + Assert.assertTrue(new File(jar).exists()); + } + + @Test + public void testExistingManifest() throws Exception { + File dir = new File(System.getProperty("test.build.dir", "target/test-dir"), + TestJarFinder.class.getName() + "-testExistingManifest"); + delete(dir); + dir.mkdirs(); + + File metaInfDir = new File(dir, "META-INF"); + metaInfDir.mkdirs(); + File manifestFile = new File(metaInfDir, "MANIFEST.MF"); + Manifest manifest = new Manifest(); + OutputStream os = new FileOutputStream(manifestFile); + manifest.write(os); + os.close(); + + File propsFile = new File(dir, "props.properties"); + Writer writer = new FileWriter(propsFile); + new Properties().store(writer, ""); + writer.close(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + JarOutputStream zos = new JarOutputStream(baos); + JarFinder.jarDir(dir, "", zos); + JarInputStream jis = + new JarInputStream(new ByteArrayInputStream(baos.toByteArray())); + Assert.assertNotNull(jis.getManifest()); + jis.close(); + } + + @Test + public void testNoManifest() throws Exception { + File dir = new File(System.getProperty("test.build.dir", "target/test-dir"), + TestJarFinder.class.getName() + "-testNoManifest"); + delete(dir); + dir.mkdirs(); + File propsFile = new File(dir, "props.properties"); + Writer writer = new FileWriter(propsFile); + new Properties().store(writer, ""); + writer.close(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + JarOutputStream zos = new JarOutputStream(baos); + JarFinder.jarDir(dir, "", zos); + JarInputStream jis = + new JarInputStream(new ByteArrayInputStream(baos.toByteArray())); + Assert.assertNotNull(jis.getManifest()); + jis.close(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestTotalOrderPartitioner.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestTotalOrderPartitioner.java new file mode 100644 index 000000000000..7cb1f99d65c5 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestTotalOrderPartitioner.java @@ -0,0 +1,205 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.mapreduce.hadoopbackport; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; + +import junit.framework.TestCase; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.io.RawComparator; +import org.apache.hadoop.io.SequenceFile; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.io.WritableComparable; +import org.apache.hadoop.io.WritableComparator; +import org.apache.hadoop.io.WritableUtils; +import org.apache.hadoop.mapred.JobConf; +import org.junit.experimental.categories.Category; + +/** + * The test is ported from Hadoop branch-0.23 with very small changes. + */ +@Category(SmallTests.class) +public class TestTotalOrderPartitioner extends TestCase { + + private static final Text[] splitStrings = new Text[] { + // -inf // 0 + new Text("aabbb"), // 1 + new Text("babbb"), // 2 + new Text("daddd"), // 3 + new Text("dddee"), // 4 + new Text("ddhee"), // 5 + new Text("dingo"), // 6 + new Text("hijjj"), // 7 + new Text("n"), // 8 + new Text("yak"), // 9 + }; + + static class Check { + T data; + int part; + Check(T data, int part) { + this.data = data; + this.part = part; + } + } + + private static final ArrayList> testStrings = + new ArrayList>(); + static { + testStrings.add(new Check(new Text("aaaaa"), 0)); + testStrings.add(new Check(new Text("aaabb"), 0)); + testStrings.add(new Check(new Text("aabbb"), 1)); + testStrings.add(new Check(new Text("aaaaa"), 0)); + testStrings.add(new Check(new Text("babbb"), 2)); + testStrings.add(new Check(new Text("baabb"), 1)); + testStrings.add(new Check(new Text("yai"), 8)); + testStrings.add(new Check(new Text("yak"), 9)); + testStrings.add(new Check(new Text("z"), 9)); + testStrings.add(new Check(new Text("ddngo"), 5)); + testStrings.add(new Check(new Text("hi"), 6)); + }; + + private static > Path writePartitionFile( + String testname, Configuration conf, T[] splits) throws IOException { + final FileSystem fs = FileSystem.getLocal(conf); + final Path testdir = new Path(System.getProperty("test.build.data", "/tmp") + ).makeQualified(fs); + Path p = new Path(testdir, testname + "/_partition.lst"); + TotalOrderPartitioner.setPartitionFile(conf, p); + conf.setInt("mapreduce.job.reduces", splits.length + 1); + SequenceFile.Writer w = null; + try { + w = SequenceFile.createWriter(fs, conf, p, + splits[0].getClass(), NullWritable.class, + SequenceFile.CompressionType.NONE); + for (int i = 0; i < splits.length; ++i) { + w.append(splits[i], NullWritable.get()); + } + } finally { + if (null != w) + w.close(); + } + return p; + } + + public void testTotalOrderMemCmp() throws Exception { + TotalOrderPartitioner partitioner = + new TotalOrderPartitioner(); + + // Need to use old JobConf-based variant here: + JobConf conf = new JobConf(); + conf.setMapOutputKeyClass(Text.class); + conf.setNumReduceTasks(splitStrings.length + 1); + + Path p = TestTotalOrderPartitioner.writePartitionFile( + "totalordermemcmp", conf, splitStrings); + try { + partitioner.setConf(conf); + NullWritable nw = NullWritable.get(); + for (Check chk : testStrings) { + assertEquals(chk.data.toString(), chk.part, + partitioner.getPartition(chk.data, nw, splitStrings.length + 1)); + } + } finally { + p.getFileSystem(conf).delete(p, true); + } + } + + public void testTotalOrderBinarySearch() throws Exception { + TotalOrderPartitioner partitioner = + new TotalOrderPartitioner(); + JobConf conf = new JobConf(); + conf.setMapOutputKeyClass(Text.class); + conf.setNumReduceTasks(splitStrings.length + 1); + + Path p = TestTotalOrderPartitioner.writePartitionFile( + "totalorderbinarysearch", conf, splitStrings); + conf.setBoolean(TotalOrderPartitioner.NATURAL_ORDER, false); + try { + partitioner.setConf(conf); + NullWritable nw = NullWritable.get(); + for (Check chk : testStrings) { + assertEquals(chk.data.toString(), chk.part, + partitioner.getPartition(chk.data, nw, splitStrings.length + 1)); + } + } finally { + p.getFileSystem(conf).delete(p, true); + } + } + + public static class ReverseStringComparator implements RawComparator { + @Override + public int compare(Text a, Text b) { + return -a.compareTo(b); + } + @Override + public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) { + int n1 = WritableUtils.decodeVIntSize(b1[s1]); + int n2 = WritableUtils.decodeVIntSize(b2[s2]); + return -1 * WritableComparator.compareBytes(b1, s1+n1, l1-n1, + b2, s2+n2, l2-n2); + } + } + + public void testTotalOrderCustomComparator() throws Exception { + TotalOrderPartitioner partitioner = + new TotalOrderPartitioner(); + + final JobConf conf = new JobConf(); + conf.setMapOutputKeyClass(Text.class); + conf.setNumReduceTasks(splitStrings.length + 1); + + Text[] revSplitStrings = Arrays.copyOf(splitStrings, splitStrings.length); + Arrays.sort(revSplitStrings, new ReverseStringComparator()); + Path p = TestTotalOrderPartitioner.writePartitionFile( + "totalordercustomcomparator", conf, revSplitStrings); + conf.setBoolean(TotalOrderPartitioner.NATURAL_ORDER, false); + conf.setOutputKeyComparatorClass(ReverseStringComparator.class); + + ArrayList> revCheck = new ArrayList>(); + revCheck.add(new Check(new Text("aaaaa"), 9)); + revCheck.add(new Check(new Text("aaabb"), 9)); + revCheck.add(new Check(new Text("aabbb"), 9)); + revCheck.add(new Check(new Text("aaaaa"), 9)); + revCheck.add(new Check(new Text("babbb"), 8)); + revCheck.add(new Check(new Text("baabb"), 8)); + revCheck.add(new Check(new Text("yai"), 1)); + revCheck.add(new Check(new Text("yak"), 1)); + revCheck.add(new Check(new Text("z"), 0)); + revCheck.add(new Check(new Text("ddngo"), 4)); + revCheck.add(new Check(new Text("hi"), 3)); + try { + partitioner.setConf(conf); + NullWritable nw = NullWritable.get(); + for (Check chk : revCheck) { + assertEquals(chk.data.toString(), chk.part, + partitioner.getPartition(chk.data, nw, splitStrings.length + 1)); + } + } finally { + p.getFileSystem(conf).delete(p, true); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/Mocking.java b/src/test/java/org/apache/hadoop/hbase/master/Mocking.java new file mode 100644 index 000000000000..6dd379f38320 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/Mocking.java @@ -0,0 +1,59 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master; + +import org.apache.hadoop.hbase.master.AssignmentManager.RegionState; + +/** + * Package scoped mocking utility. + */ +public class Mocking { + + static void waitForRegionPendingOpenInRIT(AssignmentManager am, String encodedName) + throws InterruptedException { + // We used to do a check like this: + //!Mocking.verifyRegionState(this.watcher, REGIONINFO, EventType.M_ZK_REGION_OFFLINE)) { + // There is a race condition with this: because we may do the transition to + // RS_ZK_REGION_OPENING before the RIT is internally updated. We need to wait for the + // RIT to be as we need it to be instead. This cannot happen in a real cluster as we + // update the RIT before sending the openRegion request. + + boolean wait = true; + while (wait) { + RegionState state = am.getRegionsInTransition().get(encodedName); + if (state != null && state.isPendingOpen()){ + wait = false; + } else { + Thread.sleep(1); + } + } + } + + static void waitForRegionOfflineInRIT(AssignmentManager am, String encodedName) + throws InterruptedException { + boolean wait = true; + while (wait) { + RegionState state = am.getRegionsInTransition().get(encodedName); + if (state != null && state.isOffline()) { + wait = false; + } else { + Thread.sleep(1); + } + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java b/src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java index c00b08c16ec2..52e555e7cd01 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -36,6 +35,7 @@ import org.apache.hadoop.hbase.zookeeper.ZKUtil; import org.apache.hadoop.hbase.zookeeper.ZooKeeperListener; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.apache.hadoop.hbase.zookeeper.ClusterStatusTracker; import org.apache.zookeeper.KeeperException; import org.junit.AfterClass; import org.junit.BeforeClass; @@ -66,28 +66,31 @@ public static void tearDownAfterClass() throws Exception { "testActiveMasterManagerFromZK", null, true); try { ZKUtil.deleteNode(zk, zk.masterAddressZNode); + ZKUtil.deleteNode(zk, zk.clusterStateZNode); } catch(KeeperException.NoNodeException nne) {} // Create the master node with a dummy address ServerName master = new ServerName("localhost", 1, System.currentTimeMillis()); // Should not have a master yet - DummyMaster dummyMaster = new DummyMaster(); - ActiveMasterManager activeMasterManager = new ActiveMasterManager(zk, - master, dummyMaster); - zk.registerListener(activeMasterManager); + DummyMaster dummyMaster = new DummyMaster(zk,master); + ClusterStatusTracker clusterStatusTracker = + dummyMaster.getClusterStatusTracker(); + ActiveMasterManager activeMasterManager = + dummyMaster.getActiveMasterManager(); assertFalse(activeMasterManager.clusterHasActiveMaster.get()); // First test becoming the active master uninterrupted MonitoredTask status = Mockito.mock(MonitoredTask.class); + clusterStatusTracker.setClusterUp(); + activeMasterManager.blockUntilBecomingActiveMaster(status); assertTrue(activeMasterManager.clusterHasActiveMaster.get()); assertMaster(zk, master); // Now pretend master restart - DummyMaster secondDummyMaster = new DummyMaster(); - ActiveMasterManager secondActiveMasterManager = new ActiveMasterManager(zk, - master, secondDummyMaster); - zk.registerListener(secondActiveMasterManager); + DummyMaster secondDummyMaster = new DummyMaster(zk,master); + ActiveMasterManager secondActiveMasterManager = + secondDummyMaster.getActiveMasterManager(); assertFalse(secondActiveMasterManager.clusterHasActiveMaster.get()); activeMasterManager.blockUntilBecomingActiveMaster(status); assertTrue(activeMasterManager.clusterHasActiveMaster.get()); @@ -105,6 +108,7 @@ public void testActiveMasterManagerFromZK() throws Exception { "testActiveMasterManagerFromZK", null, true); try { ZKUtil.deleteNode(zk, zk.masterAddressZNode); + ZKUtil.deleteNode(zk, zk.clusterStateZNode); } catch(KeeperException.NoNodeException nne) {} // Create the master node with a dummy address @@ -114,13 +118,15 @@ public void testActiveMasterManagerFromZK() throws Exception { new ServerName("localhost", 2, System.currentTimeMillis()); // Should not have a master yet - DummyMaster ms1 = new DummyMaster(); - ActiveMasterManager activeMasterManager = new ActiveMasterManager(zk, - firstMasterAddress, ms1); - zk.registerListener(activeMasterManager); + DummyMaster ms1 = new DummyMaster(zk,firstMasterAddress); + ActiveMasterManager activeMasterManager = + ms1.getActiveMasterManager(); assertFalse(activeMasterManager.clusterHasActiveMaster.get()); // First test becoming the active master uninterrupted + ClusterStatusTracker clusterStatusTracker = + ms1.getClusterStatusTracker(); + clusterStatusTracker.setClusterUp(); activeMasterManager.blockUntilBecomingActiveMaster( Mockito.mock(MonitoredTask.class)); assertTrue(activeMasterManager.clusterHasActiveMaster.get()); @@ -128,7 +134,6 @@ public void testActiveMasterManagerFromZK() throws Exception { // New manager will now try to become the active master in another thread WaitToBeMasterThread t = new WaitToBeMasterThread(zk, secondMasterAddress); - zk.registerListener(t.manager); t.start(); // Wait for this guy to figure out there is another active master // Wait for 1 second at most @@ -193,11 +198,12 @@ private void assertMaster(ZooKeeperWatcher zk, public static class WaitToBeMasterThread extends Thread { ActiveMasterManager manager; + DummyMaster dummyMaster; boolean isActiveMaster; public WaitToBeMasterThread(ZooKeeperWatcher zk, ServerName address) { - this.manager = new ActiveMasterManager(zk, address, - new DummyMaster()); + this.dummyMaster = new DummyMaster(zk,address); + this.manager = this.dummyMaster.getActiveMasterManager(); isActiveMaster = false; } @@ -240,6 +246,18 @@ public void waitForDeletion() throws InterruptedException { */ public static class DummyMaster implements Server { private volatile boolean stopped; + private ClusterStatusTracker clusterStatusTracker; + private ActiveMasterManager activeMasterManager; + + public DummyMaster(ZooKeeperWatcher zk, ServerName master) { + this.clusterStatusTracker = + new ClusterStatusTracker(zk, this); + clusterStatusTracker.start(); + + this.activeMasterManager = + new ActiveMasterManager(zk, master, this); + zk.registerListener(activeMasterManager); + } @Override public void abort(final String msg, final Throwable t) {} @@ -278,6 +296,14 @@ public void stop(String why) { public CatalogTracker getCatalogTracker() { return null; } + + public ClusterStatusTracker getClusterStatusTracker() { + return clusterStatusTracker; + } + + public ActiveMasterManager getActiveMasterManager() { + return activeMasterManager; + } } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java b/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java index d68ce332ac6a..e56068464691 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java @@ -17,22 +17,29 @@ */ package org.apache.hadoop.hbase.master; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotSame; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; import java.io.IOException; -import java.util.ArrayList; -import java.util.List; +import java.util.*; +import java.util.concurrent.atomic.AtomicBoolean; +import com.google.common.collect.Lists; +import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HConstants; import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HServerLoad; import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.Server; import org.apache.hadoop.hbase.ServerName; -import org.apache.hadoop.hbase.SmallTests; import org.apache.hadoop.hbase.ZooKeeperConnectionException; import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HConnection; import org.apache.hadoop.hbase.client.HConnectionTestingUtility; import org.apache.hadoop.hbase.client.Result; @@ -42,47 +49,62 @@ import org.apache.hadoop.hbase.executor.ExecutorService.ExecutorType; import org.apache.hadoop.hbase.executor.RegionTransitionData; import org.apache.hadoop.hbase.ipc.HRegionInterface; +import org.apache.hadoop.hbase.master.AssignmentManager.RegionState; +import org.apache.hadoop.hbase.master.AssignmentManager.RegionState.State; import org.apache.hadoop.hbase.master.handler.ServerShutdownHandler; import org.apache.hadoop.hbase.regionserver.RegionOpeningState; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.hbase.util.Writables; +import org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper; import org.apache.hadoop.hbase.zookeeper.ZKAssign; +import org.apache.hadoop.hbase.zookeeper.ZKTable.TableState; import org.apache.hadoop.hbase.zookeeper.ZKUtil; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.KeeperException.NodeExistsException; +import org.apache.zookeeper.Watcher; import org.junit.After; import org.junit.AfterClass; import org.junit.Before; import org.junit.BeforeClass; +import org.junit.FixMethodOrder; import org.junit.Test; import org.junit.experimental.categories.Category; +import org.junit.runners.MethodSorters; import org.mockito.Mockito; -import org.apache.hadoop.hbase.util.Pair; -import org.apache.hadoop.hbase.client.Get; -import java.util.Map; -import java.util.concurrent.atomic.AtomicBoolean; +import org.mockito.internal.util.reflection.Whitebox; + +import com.google.protobuf.ServiceException; /** * Test {@link AssignmentManager} */ -@Category(SmallTests.class) +@FixMethodOrder(MethodSorters.NAME_ASCENDING) +@Category(MediumTests.class) public class TestAssignmentManager { private static final HBaseTestingUtility HTU = new HBaseTestingUtility(); private static final ServerName SERVERNAME_A = new ServerName("example.org", 1234, 5678); private static final ServerName SERVERNAME_B = new ServerName("example.org", 0, 5678); + private static final ServerName SERVERNAME_C = + new ServerName("example.org", 123, 5678); private static final HRegionInfo REGIONINFO = new HRegionInfo(Bytes.toBytes("t"), HConstants.EMPTY_START_ROW, HConstants.EMPTY_START_ROW); + private static final HRegionInfo REGIONINFO_2 = new HRegionInfo(Bytes.toBytes("t"), + Bytes.toBytes("a"),Bytes.toBytes( "b")); + private static int assignmentCount; + private static boolean enabling = false; // Mocked objects or; get redone for each test. private Server server; private ServerManager serverManager; private ZooKeeperWatcher watcher; + private LoadBalancer balancer; @BeforeClass public static void beforeClass() throws Exception { @@ -112,22 +134,23 @@ public void before() throws ZooKeeperConnectionException, IOException { // Mock a ServerManager. Say server SERVERNAME_{A,B} are online. Also // make it so if close or open, we return 'success'. - this.serverManager = Mockito.mock(ServerManager.class); - Mockito.when(this.serverManager.isServerOnline(SERVERNAME_A)).thenReturn(true); - Mockito.when(this.serverManager.isServerOnline(SERVERNAME_B)).thenReturn(true); - final List onlineServers = new ArrayList(); - onlineServers.add(SERVERNAME_B); - onlineServers.add(SERVERNAME_A); - Mockito.when(this.serverManager.getOnlineServersList()).thenReturn(onlineServers); - Mockito.when(this.serverManager.sendRegionClose(SERVERNAME_A, REGIONINFO, -1)). - thenReturn(true); - Mockito.when(this.serverManager.sendRegionClose(SERVERNAME_B, REGIONINFO, -1)). - thenReturn(true); - // Ditto on open. - Mockito.when(this.serverManager.sendRegionOpen(SERVERNAME_A, REGIONINFO, -1)). - thenReturn(RegionOpeningState.OPENED); - Mockito.when(this.serverManager.sendRegionOpen(SERVERNAME_B, REGIONINFO, -1)). - thenReturn(RegionOpeningState.OPENED); + this.serverManager = mockManager(SERVERNAME_A, SERVERNAME_B); + } + + private ServerManager mockManager(ServerName... servers) throws IOException { + ServerManager serverManager = Mockito.mock(ServerManager.class); + final Map onlineServers = new HashMap(); + for (ServerName server : servers) { + Mockito.when(serverManager.isServerOnline(server)).thenReturn(true); + onlineServers.put(server, new HServerLoad()); + Mockito.when(serverManager.sendRegionClose(server, REGIONINFO, -1)).thenReturn(true); + Mockito.when(serverManager.sendRegionOpen(server, REGIONINFO, -1)). + thenReturn(RegionOpeningState.OPENED); + } + Mockito.when(serverManager.getOnlineServersList()).thenReturn( + new ArrayList(onlineServers.keySet())); + Mockito.when(serverManager.getOnlineServers()).thenReturn(onlineServers); + return serverManager; } @After @@ -141,12 +164,12 @@ public void after() throws KeeperException { /** * Test a balance going on at same time as a master failover - * + * * @throws IOException * @throws KeeperException * @throws InterruptedException */ - @Test(timeout = 5000) + @Test(timeout = 60000) public void testBalanceOnMasterFailoverScenarioWithOpenedNode() throws IOException, KeeperException, InterruptedException { AssignmentManagerWithExtrasForTesting am = @@ -163,13 +186,17 @@ public void testBalanceOnMasterFailoverScenarioWithOpenedNode() int versionid = ZKAssign.transitionNodeClosed(this.watcher, REGIONINFO, SERVERNAME_A, -1); assertNotSame(versionid, -1); - while (!ZKAssign.verifyRegionState(this.watcher, REGIONINFO, - EventType.M_ZK_REGION_OFFLINE)) { - Threads.sleep(1); - } - // Get current versionid else will fail on transition from OFFLINE to + Mocking.waitForRegionOfflineInRIT(am, REGIONINFO.getEncodedName()); + + // Get the OFFLINE version id. May have to wait some for it to happen. // OPENING below - versionid = ZKAssign.getVersion(this.watcher, REGIONINFO); + while (true) { + int vid = ZKAssign.getVersion(this.watcher, REGIONINFO); + if (vid != versionid) { + versionid = vid; + break; + } + } assertNotSame(-1, versionid); // This uglyness below is what the openregionhandler on RS side does. versionid = ZKAssign.transitionNode(server.getZooKeeper(), REGIONINFO, @@ -189,7 +216,7 @@ public void testBalanceOnMasterFailoverScenarioWithOpenedNode() } } - @Test(timeout = 5000) + @Test(timeout = 60000) public void testBalanceOnMasterFailoverScenarioWithClosedNode() throws IOException, KeeperException, InterruptedException { AssignmentManagerWithExtrasForTesting am = @@ -207,13 +234,17 @@ public void testBalanceOnMasterFailoverScenarioWithClosedNode() ZKAssign.transitionNodeClosed(this.watcher, REGIONINFO, SERVERNAME_A, -1); assertNotSame(versionid, -1); am.gate.set(false); - while (!ZKAssign.verifyRegionState(this.watcher, REGIONINFO, - EventType.M_ZK_REGION_OFFLINE)) { - Threads.sleep(1); - } + Mocking.waitForRegionOfflineInRIT(am, REGIONINFO.getEncodedName()); + // Get current versionid else will fail on transition from OFFLINE to // OPENING below - versionid = ZKAssign.getVersion(this.watcher, REGIONINFO); + while (true) { + int vid = ZKAssign.getVersion(this.watcher, REGIONINFO); + if (vid != versionid) { + versionid = vid; + break; + } + } assertNotSame(-1, versionid); // This uglyness below is what the openregionhandler on RS side does. versionid = ZKAssign.transitionNode(server.getZooKeeper(), REGIONINFO, @@ -233,7 +264,7 @@ public void testBalanceOnMasterFailoverScenarioWithClosedNode() } } - @Test(timeout = 5000) + @Test(timeout = 60000) public void testBalanceOnMasterFailoverScenarioWithOfflineNode() throws IOException, KeeperException, InterruptedException { AssignmentManagerWithExtrasForTesting am = @@ -250,14 +281,18 @@ public void testBalanceOnMasterFailoverScenarioWithOfflineNode() int versionid = ZKAssign.transitionNodeClosed(this.watcher, REGIONINFO, SERVERNAME_A, -1); assertNotSame(versionid, -1); - while (!ZKAssign.verifyRegionState(this.watcher, REGIONINFO, - EventType.M_ZK_REGION_OFFLINE)) { - Threads.sleep(1); - } + Mocking.waitForRegionOfflineInRIT(am, REGIONINFO.getEncodedName()); + am.gate.set(false); // Get current versionid else will fail on transition from OFFLINE to // OPENING below - versionid = ZKAssign.getVersion(this.watcher, REGIONINFO); + while (true) { + int vid = ZKAssign.getVersion(this.watcher, REGIONINFO); + if (vid != versionid) { + versionid = vid; + break; + } + } assertNotSame(-1, versionid); // This uglyness below is what the openregionhandler on RS side does. versionid = ZKAssign.transitionNode(server.getZooKeeper(), REGIONINFO, @@ -292,19 +327,22 @@ private void createRegionPlanAndBalance(final AssignmentManager am, * from one server to another mocking regionserver responding over zk. * @throws IOException * @throws KeeperException + * @throws InterruptedException */ - @Test + @Test(timeout = 60000) public void testBalance() - throws IOException, KeeperException { + throws IOException, KeeperException, InterruptedException { // Create and startup an executor. This is used by AssignmentManager // handling zk callbacks. ExecutorService executor = startupMasterExecutor("testBalanceExecutor"); // We need a mocked catalog tracker. CatalogTracker ct = Mockito.mock(CatalogTracker.class); + LoadBalancer balancer = LoadBalancerFactory.getLoadBalancer(server + .getConfiguration()); // Create an AM. - AssignmentManager am = - new AssignmentManager(this.server, this.serverManager, ct, executor); + AssignmentManager am = new AssignmentManager(this.server, + this.serverManager, ct, balancer, executor); try { // Make sure our new AM gets callbacks; once registered, can't unregister. // Thats ok because we make a new zk watcher for each test. @@ -327,13 +365,17 @@ public void testBalance() // AM is going to notice above CLOSED and queue up a new assign. The // assign will go to open the region in the new location set by the // balancer. The zk node will be OFFLINE waiting for regionserver to - // transition it through OPENING, OPENED. Wait till we see the OFFLINE - // zk node before we proceed. - while (!ZKAssign.verifyRegionState(this.watcher, REGIONINFO, EventType.M_ZK_REGION_OFFLINE)) { - Threads.sleep(1); - } + // transition it through OPENING, OPENED. Wait till we see the RIT + // before we proceed. + Mocking.waitForRegionOfflineInRIT(am, REGIONINFO.getEncodedName()); // Get current versionid else will fail on transition from OFFLINE to OPENING below - versionid = ZKAssign.getVersion(this.watcher, REGIONINFO); + while (true) { + int vid = ZKAssign.getVersion(this.watcher, REGIONINFO); + if (vid != versionid) { + versionid = vid; + break; + } + } assertNotSame(-1, versionid); // This uglyness below is what the openregionhandler on RS side does. versionid = ZKAssign.transitionNode(server.getZooKeeper(), REGIONINFO, @@ -354,6 +396,35 @@ public void testBalance() } } + @Test + public void testGettingAssignmentsExcludesDrainingServers() throws Exception { + List availableServers = + Lists.newArrayList(SERVERNAME_A, SERVERNAME_B, SERVERNAME_C); + ServerManager serverManager = mockManager(availableServers.toArray(new ServerName[0])); + + + ExecutorService executor = startupMasterExecutor("testAssignmentsWithRSInDraining"); + CatalogTracker ct = Mockito.mock(CatalogTracker.class); + + LoadBalancer balancer = LoadBalancerFactory.getLoadBalancer(server.getConfiguration()); + + Mockito.when(serverManager.getDrainingServersList()).thenReturn( + Lists.newArrayList(SERVERNAME_C)); + AssignmentManager am = new AssignmentManager(this.server, serverManager, ct, balancer, executor); + + for (ServerName availableServer : availableServers) { + HRegionInfo info = Mockito.mock(HRegionInfo.class); + Mockito.when(info.getEncodedName()).thenReturn(UUID.randomUUID().toString()); + am.regionOnline(info, availableServer); + } + + Map>> result = am.getAssignmentsByTable(); + for (Map> map : result.values()) { + System.out.println(map.keySet()); + assertFalse(map.containsKey(SERVERNAME_C)); + } + } + /** * Run a simple server shutdown handler. * @throws KeeperException @@ -367,47 +438,100 @@ public void testShutdownHandler() throws KeeperException, IOException { // We need a mocked catalog tracker. CatalogTracker ct = Mockito.mock(CatalogTracker.class); + LoadBalancer balancer = LoadBalancerFactory.getLoadBalancer(server + .getConfiguration()); // Create an AM. AssignmentManager am = - new AssignmentManager(this.server, this.serverManager, ct, executor); + new AssignmentManager(this.server, this.serverManager, ct, balancer, executor); try { - // Make sure our new AM gets callbacks; once registered, can't unregister. - // Thats ok because we make a new zk watcher for each test. - this.watcher.registerListenerFirst(am); + processServerShutdownHandler(ct, am, false, null); + } finally { + executor.shutdown(); + am.shutdown(); + // Clean up all znodes + ZKAssign.deleteAllNodes(this.watcher); + } + } + + /** + * To test closed region handler to remove rit and delete corresponding znode if region in pending + * close or closing while processing shutdown of a region server.(HBASE-5927). + * @throws KeeperException + * @throws IOException + */ + @Test + public void testSSHWhenDisableTableInProgress() + throws KeeperException, IOException { + testCaseWithPartiallyDisabledState(TableState.DISABLING, false); + testCaseWithPartiallyDisabledState(TableState.DISABLED, false); + } + + @Test + public void testSSHWhenDisablingTableRegionsInOpeningState() + throws KeeperException, IOException { + testCaseWithPartiallyDisabledState(TableState.DISABLING, true); + testCaseWithPartiallyDisabledState(TableState.DISABLED, true); + } + + + /** + * To test if the split region is removed from RIT if the region was in SPLITTING state + * but the RS has actually completed the splitting in META but went down. See HBASE-6070 + * and also HBASE-5806 + * @throws KeeperException + * @throws IOException + */ + @Test + public void testSSHWhenSplitRegionInProgress() + throws KeeperException, IOException, Exception { + // true indicates the region is split but still in RIT + testCaseWithSplitRegionPartial(true); + // false indicate the region is not split + testCaseWithSplitRegionPartial(false); + + } + + private void testCaseWithSplitRegionPartial(boolean regionSplitDone) throws KeeperException, IOException, + NodeExistsException, InterruptedException { + // Create and startup an executor. This is used by AssignmentManager + // handling zk callbacks. + ExecutorService executor = startupMasterExecutor("testSSHWhenSplitRegionInProgress"); + + // We need a mocked catalog tracker. + CatalogTracker ct = Mockito.mock(CatalogTracker.class); + // Create an AM. + AssignmentManagerWithExtrasForTesting am = setUpMockedAssignmentManager(this.server, this.serverManager); + // adding region to regions and servers maps. + am.regionOnline(REGIONINFO, SERVERNAME_A); + // adding region in pending close. + am.regionsInTransition.put(REGIONINFO.getEncodedName(), new RegionState(REGIONINFO, + State.SPLITTING, System.currentTimeMillis(), SERVERNAME_A)); + am.getZKTable().setEnabledTable(REGIONINFO.getTableNameAsString()); + + RegionTransitionData data = new RegionTransitionData(EventType.RS_ZK_REGION_SPLITTING, + REGIONINFO.getRegionName(), SERVERNAME_A); + String node = ZKAssign.getNodeName(this.watcher, REGIONINFO.getEncodedName()); + // create znode in M_ZK_REGION_CLOSING state. + ZKUtil.createAndWatch(this.watcher, node, data.getBytes()); + + try { + processServerShutdownHandler(ct, am, regionSplitDone, null); + // check znode deleted or not. + // In both cases the znode should be deleted. - // Need to set up a fake scan of meta for the servershutdown handler - // Make an RS Interface implementation. Make it so a scanner can go against it. - HRegionInterface implementation = Mockito.mock(HRegionInterface.class); - // Get a meta row result that has region up on SERVERNAME_A - Result r = getMetaTableRowResult(REGIONINFO, SERVERNAME_A); - Mockito.when(implementation.openScanner((byte [])Mockito.any(), (Scan)Mockito.any())). - thenReturn(System.currentTimeMillis()); - // Return a good result first and then return null to indicate end of scan - Mockito.when(implementation.next(Mockito.anyLong(), Mockito.anyInt())). - thenReturn(new Result [] {r}, (Result [])null); - - // Get a connection w/ mocked up common methods. - HConnection connection = - HConnectionTestingUtility.getMockedConnectionAndDecorate(HTU.getConfiguration(), - implementation, SERVERNAME_B, REGIONINFO); - - // Make it so we can get a catalogtracker from servermanager.. .needed - // down in guts of server shutdown handler. - Mockito.when(ct.getConnection()).thenReturn(connection); - Mockito.when(this.server.getCatalogTracker()).thenReturn(ct); - - // Now make a server shutdown handler instance and invoke process. - // Have it that SERVERNAME_A died. - DeadServer deadServers = new DeadServer(); - deadServers.add(SERVERNAME_A); - // I need a services instance that will return the AM - MasterServices services = Mockito.mock(MasterServices.class); - Mockito.when(services.getAssignmentManager()).thenReturn(am); - ServerShutdownHandler handler = new ServerShutdownHandler(this.server, - services, deadServers, SERVERNAME_A, false); - handler.process(); - // The region in r will have been assigned. It'll be up in zk as unassigned. + if(regionSplitDone){ + assertTrue("Region state of region in SPLITTING should be removed from rit.", + am.regionsInTransition.isEmpty()); + } + else{ + while (!am.assignInvoked) { + Thread.sleep(1); + } + assertTrue("Assign should be invoked.", am.assignInvoked); + } } finally { + REGIONINFO.setOffline(false); + REGIONINFO.setSplit(false); executor.shutdown(); am.shutdown(); // Clean up all znodes @@ -415,12 +539,129 @@ public void testShutdownHandler() throws KeeperException, IOException { } } + private void testCaseWithPartiallyDisabledState(TableState state, boolean opening) + throws KeeperException, IOException, NodeExistsException { + // Create and startup an executor. This is used by AssignmentManager + // handling zk callbacks. + ExecutorService executor = startupMasterExecutor("testSSHWhenDisableTableInProgress"); + + // We need a mocked catalog tracker. + CatalogTracker ct = Mockito.mock(CatalogTracker.class); + LoadBalancer balancer = LoadBalancerFactory.getLoadBalancer(server.getConfiguration()); + // Create an AM. + AssignmentManager am = new AssignmentManager(this.server, this.serverManager, ct, balancer, + executor); + if (opening) { + am.regionsInTransition.put(REGIONINFO.getEncodedName(), new RegionState(REGIONINFO, + State.OPENING, System.currentTimeMillis(), SERVERNAME_A)); + } else { + // adding region to regions and servers maps. + am.regionOnline(REGIONINFO, SERVERNAME_A); + // adding region in pending close. + am.regionsInTransition.put(REGIONINFO.getEncodedName(), new RegionState(REGIONINFO, + State.PENDING_CLOSE, System.currentTimeMillis(), SERVERNAME_A)); + } + if (state == TableState.DISABLING) { + am.getZKTable().setDisablingTable(REGIONINFO.getTableNameAsString()); + } else { + am.getZKTable().setDisabledTable(REGIONINFO.getTableNameAsString()); + } + RegionTransitionData data = null; + if (opening) { + data = + new RegionTransitionData(EventType.RS_ZK_REGION_OPENING, REGIONINFO.getRegionName(), + SERVERNAME_A); + + } else { + data = + new RegionTransitionData(EventType.M_ZK_REGION_CLOSING, REGIONINFO.getRegionName(), + SERVERNAME_A); + } + String node = ZKAssign.getNodeName(this.watcher, REGIONINFO.getEncodedName()); + // create znode in M_ZK_REGION_CLOSING state. + ZKUtil.createAndWatch(this.watcher, node, data.getBytes()); + + try { + processServerShutdownHandler(ct, am, false, null); + // check znode deleted or not. + // In both cases the znode should be deleted. + assertTrue("The znode should be deleted.",ZKUtil.checkExists(this.watcher, node) == -1); + assertTrue("Region state of region in pending close should be removed from rit.", + am.regionsInTransition.isEmpty()); + } finally { + executor.shutdown(); + am.shutdown(); + // Clean up all znodes + ZKAssign.deleteAllNodes(this.watcher); + } + } + + private void processServerShutdownHandler(CatalogTracker ct, AssignmentManager am, + boolean splitRegion, ServerName sn) + throws IOException { + // Make sure our new AM gets callbacks; once registered, can't unregister. + // Thats ok because we make a new zk watcher for each test. + this.watcher.registerListenerFirst(am); + // Need to set up a fake scan of meta for the servershutdown handler + // Make an RS Interface implementation. Make it so a scanner can go against it. + HRegionInterface implementation = Mockito.mock(HRegionInterface.class); + // Get a meta row result that has region up on SERVERNAME_A + + Result r = null; + if (sn == null) { + if (splitRegion) { + r = getMetaTableRowResultAsSplitRegion(REGIONINFO, SERVERNAME_A); + } else { + r = getMetaTableRowResult(REGIONINFO, SERVERNAME_A); + } + } else { + if (sn.equals(SERVERNAME_A)) { + r = getMetaTableRowResult(REGIONINFO, SERVERNAME_A); + } else if (sn.equals(SERVERNAME_B)) { + r = new Result(new KeyValue[0]); + } + } + + Mockito.when(implementation.openScanner((byte [])Mockito.any(), (Scan)Mockito.any())). + thenReturn(System.currentTimeMillis()); + // Return a good result first and then return null to indicate end of scan + Mockito.when(implementation.next(Mockito.anyLong(), Mockito.anyInt(), Mockito.anyInt())). + thenReturn(new Result [] {r}, (Result [])null); + + // Get a connection w/ mocked up common methods. + HConnection connection = + HConnectionTestingUtility.getMockedConnectionAndDecorate(HTU.getConfiguration(), + implementation, SERVERNAME_B, REGIONINFO); + + // Make it so we can get a catalogtracker from servermanager.. .needed + // down in guts of server shutdown handler. + Mockito.when(ct.getConnection()).thenReturn(connection); + Mockito.when(this.server.getCatalogTracker()).thenReturn(ct); + + // Now make a server shutdown handler instance and invoke process. + // Have it that SERVERNAME_A died. + DeadServer deadServers = new DeadServer(); + deadServers.add(SERVERNAME_A); + // I need a services instance that will return the AM + MasterServices services = Mockito.mock(MasterServices.class); + Mockito.when(services.getAssignmentManager()).thenReturn(am); + Mockito.when(services.getZooKeeper()).thenReturn(this.watcher); + ServerShutdownHandler handler = null; + if (sn != null) { + handler = new ServerShutdownHandler(this.server, services, deadServers, sn, false); + } else { + handler = new ServerShutdownHandler(this.server, services, deadServers, SERVERNAME_A, false); + } + handler.process(); + // The region in r will have been assigned. It'll be up in zk as unassigned. + } + /** * @param sn ServerName to use making startcode and server in meta * @param hri Region to serialize into HRegionInfo * @return A mocked up Result that fakes a Get on a row in the * .META. table. - * @throws IOException + * @throws IOException */ private Result getMetaTableRowResult(final HRegionInfo hri, final ServerName sn) @@ -440,6 +681,20 @@ private Result getMetaTableRowResult(final HRegionInfo hri, return new Result(kvs); } + /** + * @param sn ServerName to use making startcode and server in meta + * @param hri Region to serialize into HRegionInfo + * @return A mocked up Result that fakes a Get on a row in the + * .META. table. + * @throws IOException + */ + private Result getMetaTableRowResultAsSplitRegion(final HRegionInfo hri, final ServerName sn) + throws IOException { + hri.setOffline(true); + hri.setSplit(true); + return getMetaTableRowResult(hri, sn); + } + /** * Create and startup executor pools. Start same set as master does (just * run a few less). @@ -466,9 +721,11 @@ public void testUnassignWithSplitAtSameTime() throws KeeperException, IOExceptio Mockito.when(this.serverManager.sendRegionClose(SERVERNAME_A, hri, -1)).thenReturn(true); // Need a mocked catalog tracker. CatalogTracker ct = Mockito.mock(CatalogTracker.class); + LoadBalancer balancer = LoadBalancerFactory.getLoadBalancer(server + .getConfiguration()); // Create an AM. AssignmentManager am = - new AssignmentManager(this.server, this.serverManager, ct, null); + new AssignmentManager(this.server, this.serverManager, ct, balancer, null); try { // First make sure my mock up basically works. Unassign a region. unassign(am, SERVERNAME_A, hri); @@ -492,6 +749,39 @@ public void testUnassignWithSplitAtSameTime() throws KeeperException, IOExceptio } } + /** + * Tests the processDeadServersAndRegionsInTransition should not fail with NPE + * when it failed to get the children. Let's abort the system in this + * situation + * @throws ServiceException + */ + @Test(timeout = 60000) + public void testProcessDeadServersAndRegionsInTransitionShouldNotFailWithNPE() + throws IOException, KeeperException, InterruptedException, ServiceException { + final RecoverableZooKeeper recoverableZk = Mockito + .mock(RecoverableZooKeeper.class); + AssignmentManagerWithExtrasForTesting am = setUpMockedAssignmentManager( + this.server, this.serverManager); + Watcher zkw = new ZooKeeperWatcher(HBaseConfiguration.create(), "unittest", + null) { + public RecoverableZooKeeper getRecoverableZooKeeper() { + return recoverableZk; + } + }; + ((ZooKeeperWatcher) zkw).registerListener(am); + Mockito.doThrow(new InterruptedException()).when(recoverableZk) + .getChildren("/hbase/unassigned", zkw); + am.setWatcher((ZooKeeperWatcher) zkw); + try { + am.processDeadServersAndRegionsInTransition(); + fail("Expected to abort"); + } catch (NullPointerException e) { + fail("Should not throw NPE"); + } catch (RuntimeException e) { + assertEquals("Aborted", e.getLocalizedMessage()); + } + } + /** * Creates a new ephemeral node in the SPLITTING state for the specified region. * Create it ephemeral in case regionserver dies mid-split. @@ -503,8 +793,8 @@ public void testUnassignWithSplitAtSameTime() throws KeeperException, IOExceptio * @param region region to be created as offline * @param serverName server event originates from * @return Version of znode created. - * @throws KeeperException - * @throws IOException + * @throws KeeperException + * @throws IOException */ // Copied from SplitTransaction rather than open the method over there in // the regionserver package. @@ -563,15 +853,27 @@ private AssignmentManagerWithExtrasForTesting setUpMockedAssignmentManager(final // with an encoded name by doing a Get on .META. HRegionInterface ri = Mockito.mock(HRegionInterface.class); // Get a meta row result that has region up on SERVERNAME_A for REGIONINFO + Result[] result = null; + if (enabling) { + result = new Result[2]; + result[0] = getMetaTableRowResult(REGIONINFO, SERVERNAME_A); + result[1] = getMetaTableRowResult(REGIONINFO_2, SERVERNAME_A); + } Result r = getMetaTableRowResult(REGIONINFO, SERVERNAME_A); Mockito.when(ri .openScanner((byte[]) Mockito.any(), (Scan) Mockito.any())). thenReturn(System.currentTimeMillis()); - // Return good result 'r' first and then return null to indicate end of scan - Mockito.when(ri.next(Mockito.anyLong(), Mockito.anyInt())). - thenReturn(new Result[] { r }, (Result[]) null); - // If a get, return the above result too for REGIONINFO - Mockito.when(ri.get((byte[]) Mockito.any(), (Get) Mockito.any())). - thenReturn(r); + if (enabling) { + Mockito.when(ri.next(Mockito.anyLong(), Mockito.anyInt(), Mockito.anyInt())).thenReturn(result, result, result, + (Result[]) null); + // If a get, return the above result too for REGIONINFO_2 + Mockito.when(ri.get((byte[]) Mockito.any(), (Get) Mockito.any())).thenReturn( + getMetaTableRowResult(REGIONINFO_2, SERVERNAME_A)); + } else { + // Return good result 'r' first and then return null to indicate end of scan + Mockito.when(ri.next(Mockito.anyLong(), Mockito.anyInt(), Mockito.anyInt())).thenReturn(new Result[] { r }); + // If a get, return the above result too for REGIONINFO + Mockito.when(ri.get((byte[]) Mockito.any(), (Get) Mockito.any())).thenReturn(r); + } // Get a connection w/ mocked up common methods. HConnection connection = HConnectionTestingUtility. getMockedConnectionAndDecorate(HTU.getConfiguration(), ri, SERVERNAME_B, @@ -580,11 +882,271 @@ private AssignmentManagerWithExtrasForTesting setUpMockedAssignmentManager(final Mockito.when(ct.getConnection()).thenReturn(connection); // Create and startup an executor. Used by AM handling zk callbacks. ExecutorService executor = startupMasterExecutor("mockedAMExecutor"); - AssignmentManagerWithExtrasForTesting am = - new AssignmentManagerWithExtrasForTesting(server, manager, ct, executor); + this.balancer = LoadBalancerFactory.getLoadBalancer(server.getConfiguration()); + AssignmentManagerWithExtrasForTesting am = new AssignmentManagerWithExtrasForTesting( + server, manager, ct, balancer, executor); return am; } + /** + * TestCase verifies that the regionPlan is updated whenever a region fails to open + * and the master tries to process RS_ZK_FAILED_OPEN state.(HBASE-5546). + */ + @Test + public void testRegionPlanIsUpdatedWhenRegionFailsToOpen() throws IOException, KeeperException, + ServiceException, InterruptedException { + this.server.getConfiguration().setClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, + MockedLoadBalancer.class, LoadBalancer.class); + AssignmentManagerWithExtrasForTesting am = setUpMockedAssignmentManager(this.server, + this.serverManager); + try { + // Boolean variable used for waiting until randomAssignment is called and new + // plan is generated. + AtomicBoolean gate = new AtomicBoolean(false); + if (balancer instanceof MockedLoadBalancer) { + ((MockedLoadBalancer) balancer).setGateVariable(gate); + } + ZKAssign.createNodeOffline(this.watcher, REGIONINFO, SERVERNAME_A); + int v = ZKAssign.getVersion(this.watcher, REGIONINFO); + ZKAssign.transitionNode(this.watcher, REGIONINFO, SERVERNAME_A, EventType.M_ZK_REGION_OFFLINE, + EventType.RS_ZK_REGION_FAILED_OPEN, v); + String path = ZKAssign.getNodeName(this.watcher, REGIONINFO.getEncodedName()); + RegionState state = new RegionState(REGIONINFO, State.OPENING, System.currentTimeMillis(), + SERVERNAME_A); + am.regionsInTransition.put(REGIONINFO.getEncodedName(), state); + // a dummy plan inserted into the regionPlans. This plan is cleared and new one is formed + am.regionPlans.put(REGIONINFO.getEncodedName(), new RegionPlan(REGIONINFO, null, SERVERNAME_A)); + RegionPlan regionPlan = am.regionPlans.get(REGIONINFO.getEncodedName()); + List serverList = new ArrayList(2); + serverList.add(SERVERNAME_B); + Mockito.when(this.serverManager.getOnlineServersList()).thenReturn(serverList); + am.nodeDataChanged(path); + // here we are waiting until the random assignment in the load balancer is called. + while (!gate.get()) { + Thread.sleep(10); + } + // new region plan may take some time to get updated after random assignment is called and + // gate is set to true. + RegionPlan newRegionPlan = am.regionPlans.get(REGIONINFO.getEncodedName()); + while (newRegionPlan == null) { + Thread.sleep(10); + newRegionPlan = am.regionPlans.get(REGIONINFO.getEncodedName()); + } + // the new region plan created may contain the same RS as destination but it should + // be new plan. + assertNotSame("Same region plan should not come", regionPlan, newRegionPlan); + assertTrue("Destnation servers should be different.", !(regionPlan.getDestination().equals( + newRegionPlan.getDestination()))); + Mocking.waitForRegionOfflineInRIT(am, REGIONINFO.getEncodedName()); + } finally { + this.server.getConfiguration().setClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, + DefaultLoadBalancer.class, LoadBalancer.class); + am.shutdown(); + } + } + + /** + * Test verifies whether assignment is skipped for regions of tables in DISABLING state during + * clean cluster startup. See HBASE-6281. + * + * @throws KeeperException + * @throws IOException + * @throws Exception + */ + @Test + public void testDisablingTableRegionsAssignmentDuringCleanClusterStartup() + throws KeeperException, IOException, Exception { + this.server.getConfiguration().setClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, + MockedLoadBalancer.class, LoadBalancer.class); + Mockito.when(this.serverManager.getOnlineServers()).thenReturn( + new HashMap(0)); + List destServers = new ArrayList(1); + destServers.add(SERVERNAME_A); + Mockito.when(this.serverManager.getDrainingServersList()).thenReturn(destServers); + // To avoid cast exception in DisableTableHandler process. + //Server server = new HMaster(HTU.getConfiguration()); + AssignmentManagerWithExtrasForTesting am = setUpMockedAssignmentManager(server, + this.serverManager); + AtomicBoolean gate = new AtomicBoolean(false); + if (balancer instanceof MockedLoadBalancer) { + ((MockedLoadBalancer) balancer).setGateVariable(gate); + } + try{ + // set table in disabling state. + am.getZKTable().setDisablingTable(REGIONINFO.getTableNameAsString()); + am.joinCluster(); + // should not call retainAssignment if we get empty regions in assignAllUserRegions. + assertFalse( + "Assign should not be invoked for disabling table regions during clean cluster startup.", + gate.get()); + // need to change table state from disabling to disabled. + assertTrue("Table should be disabled.", + am.getZKTable().isDisabledTable(REGIONINFO.getTableNameAsString())); + } finally { + this.server.getConfiguration().setClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, + DefaultLoadBalancer.class, LoadBalancer.class); + am.getZKTable().setEnabledTable(REGIONINFO.getTableNameAsString()); + am.shutdown(); + } + } + + /** + * Test verifies whether stale znodes of unknown tables as for the hbase:meta will be removed or + * not. + * @throws KeeperException + * @throws IOException + * @throws Exception + */ + @Test + public void testMasterRestartShouldRemoveStaleZnodesOfUnknownTableAsForMeta() + throws KeeperException, IOException, Exception { + List destServers = new ArrayList(1); + destServers.add(SERVERNAME_A); + Mockito.when(this.serverManager.getOnlineServersList()).thenReturn(destServers); + Mockito.when(this.serverManager.isServerOnline(SERVERNAME_A)).thenReturn(true); + HTU.getConfiguration().setInt(HConstants.MASTER_PORT, 0); + Server server = new HMaster(HTU.getConfiguration()); + Whitebox.setInternalState(server, "serverManager", this.serverManager); + AssignmentManagerWithExtrasForTesting am = setUpMockedAssignmentManager(server, + this.serverManager); + try { + String tableName = "dummyTable"; + am.enablingTables.put(tableName, null); + // set table in enabling state. + am.getZKTable().setEnablingTable(tableName); + am.joinCluster(); + assertFalse("Table should not be present in zookeeper.", + am.getZKTable().isTablePresent(tableName)); + } finally { + } + } + + /** + * Test verifies whether all the enabling table regions assigned only once during master startup. + * + * @throws KeeperException + * @throws IOException + * @throws Exception + */ + @Test + public void testMasterRestartWhenTableInEnabling() throws KeeperException, IOException, Exception { + enabling = true; + this.server.getConfiguration().setClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, + DefaultLoadBalancer.class, LoadBalancer.class); + Map serverAndLoad = new HashMap(); + serverAndLoad.put(SERVERNAME_A, null); + Mockito.when(this.serverManager.getOnlineServers()).thenReturn(serverAndLoad); + Mockito.when(this.serverManager.isServerOnline(SERVERNAME_B)).thenReturn(false); + Mockito.when(this.serverManager.isServerOnline(SERVERNAME_A)).thenReturn(true); + HTU.getConfiguration().setInt(HConstants.MASTER_PORT, 0); + Server server = new HMaster(HTU.getConfiguration()); + Whitebox.setInternalState(server, "serverManager", this.serverManager); + assignmentCount = 0; + AssignmentManagerWithExtrasForTesting am = setUpMockedAssignmentManager(server, + this.serverManager); + am.regionOnline(new HRegionInfo("t1".getBytes(), HConstants.EMPTY_START_ROW, + HConstants.EMPTY_END_ROW), SERVERNAME_A); + am.gate.set(false); + try { + // set table in enabling state. + am.getZKTable().setEnablingTable(REGIONINFO.getTableNameAsString()); + ZKAssign.createNodeOffline(this.watcher, REGIONINFO_2, SERVERNAME_B); + + am.joinCluster(); + while (!am.getZKTable().isEnabledTable(REGIONINFO.getTableNameAsString())) { + Thread.sleep(10); + } + assertEquals("Number of assignments should be equal.", 2, assignmentCount); + assertTrue("Table should be enabled.", + am.getZKTable().isEnabledTable(REGIONINFO.getTableNameAsString())); + } finally { + enabling = false; + am.getZKTable().setEnabledTable(REGIONINFO.getTableNameAsString()); + am.shutdown(); + ZKAssign.deleteAllNodes(this.watcher); + assignmentCount = 0; + } + } + + + + /** + * When region in transition if region server opening the region gone down then region assignment + * taking long time(Waiting for timeout monitor to trigger assign). HBASE-5396(HBASE-6060) fixes this + * scenario. This test case verifies whether SSH calling assign for the region in transition or not. + * + * @throws KeeperException + * @throws IOException + * @throws ServiceException + */ + @Test + public void testSSHWhenSourceRSandDestRSInRegionPlanGoneDown() throws KeeperException, IOException, + ServiceException { + testSSHWhenSourceRSandDestRSInRegionPlanGoneDown(true); + testSSHWhenSourceRSandDestRSInRegionPlanGoneDown(false); + } + + private void testSSHWhenSourceRSandDestRSInRegionPlanGoneDown(boolean regionInOffline) + throws IOException, KeeperException, ServiceException { + // We need a mocked catalog tracker. + CatalogTracker ct = Mockito.mock(CatalogTracker.class); + // Create an AM. + AssignmentManagerWithExtrasForTesting am = + setUpMockedAssignmentManager(this.server, this.serverManager); + // adding region in pending open. + if (regionInOffline) { + ServerName MASTER_SERVERNAME = new ServerName("example.org", 1111, 1111); + am.regionsInTransition.put(REGIONINFO.getEncodedName(), new RegionState(REGIONINFO, + State.OFFLINE, System.currentTimeMillis(), MASTER_SERVERNAME)); + } else { + am.regionsInTransition.put(REGIONINFO.getEncodedName(), new RegionState(REGIONINFO, + State.OPENING, System.currentTimeMillis(), SERVERNAME_B)); + } + // adding region plan + am.regionPlans.put(REGIONINFO.getEncodedName(), new RegionPlan(REGIONINFO, SERVERNAME_A, SERVERNAME_B)); + am.getZKTable().setEnabledTable(REGIONINFO.getTableNameAsString()); + + try { + processServerShutdownHandler(ct, am, false, SERVERNAME_A); + processServerShutdownHandler(ct, am, false, SERVERNAME_B); + if(regionInOffline){ + assertFalse("Assign should not be invoked.", am.assignInvoked); + } else { + assertTrue("Assign should be invoked.", am.assignInvoked); + } + } finally { + am.regionsInTransition.remove(REGIONINFO.getEncodedName()); + am.regionPlans.remove(REGIONINFO.getEncodedName()); + } + } + + /** + * Mocked load balancer class used in the testcase to make sure that the testcase waits until + * random assignment is called and the gate variable is set to true. + */ + public static class MockedLoadBalancer extends DefaultLoadBalancer { + private AtomicBoolean gate; + + public void setGateVariable(AtomicBoolean gate) { + this.gate = gate; + } + + @Override + public ServerName randomAssignment(List servers) { + ServerName randomServerName = super.randomAssignment(servers); + this.gate.set(true); + return randomServerName; + } + + @Override + public Map> retainAssignment( + Map regions, List servers) { + this.gate.set(true); + return super.retainAssignment(regions, servers); + } + + } + /** * An {@link AssignmentManager} with some extra facility used testing */ @@ -594,13 +1156,14 @@ class AssignmentManagerWithExtrasForTesting extends AssignmentManager { // Ditto for ct private final CatalogTracker ct; boolean processRITInvoked = false; + boolean assignInvoked = false; AtomicBoolean gate = new AtomicBoolean(true); public AssignmentManagerWithExtrasForTesting(final Server master, - final ServerManager serverManager, - final CatalogTracker catalogTracker, final ExecutorService service) + final ServerManager serverManager, final CatalogTracker catalogTracker, + final LoadBalancer balancer, final ExecutorService service) throws KeeperException, IOException { - super(master, serverManager, catalogTracker, service); + super(master, serverManager, catalogTracker, balancer, service); this.es = service; this.ct = catalogTracker; } @@ -622,6 +1185,28 @@ void processRegionsInTransition(final RegionTransitionData data, while (this.gate.get()) Threads.sleep(1); super.processRegionsInTransition(data, regionInfo, deadServers, expectedVersion); } + + @Override + public void assign(HRegionInfo region, boolean setOfflineInZK, boolean forceNewPlan, + boolean hijack) { + if (enabling) { + assignmentCount++; + this.regionOnline(region, SERVERNAME_A); + } else { + assignInvoked = true; + super.assign(region, setOfflineInZK, forceNewPlan, hijack); + } + } + + @Override + public ServerName getRegionServerOfRegion(HRegionInfo hri) { + return SERVERNAME_A; + } + + /** reset the watcher */ + void setWatcher(ZooKeeperWatcher watcher) { + this.watcher = watcher; + } /** * @return ExecutorService used by this instance. diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java index a5628b95e2fb..073345fdffc0 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,12 +18,15 @@ */ package org.apache.hadoop.hbase.master; +import static org.apache.hadoop.hbase.util.HFileArchiveTestingUtil.assertArchiveEqualToOriginal; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.spy; -import java.io.FileNotFoundException; import java.io.IOException; import java.util.ArrayList; import java.util.List; @@ -33,23 +35,37 @@ import java.util.TreeMap; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException; +import org.apache.hadoop.hbase.Server; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.TableDescriptors; import org.apache.hadoop.hbase.catalog.CatalogTracker; import org.apache.hadoop.hbase.client.HConnection; import org.apache.hadoop.hbase.client.HConnectionManager; import org.apache.hadoop.hbase.client.HConnectionTestingUtility; import org.apache.hadoop.hbase.client.Result; -import org.apache.hadoop.hbase.executor.EventHandler; import org.apache.hadoop.hbase.executor.ExecutorService; import org.apache.hadoop.hbase.io.Reference; +import org.apache.hadoop.hbase.ipc.CoprocessorProtocol; import org.apache.hadoop.hbase.ipc.HRegionInterface; +import org.apache.hadoop.hbase.master.CatalogJanitor.SplitParentFirstComparator; import org.apache.hadoop.hbase.regionserver.Store; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.HFileArchiveUtil; +import org.apache.hadoop.hbase.util.Pair; import org.apache.hadoop.hbase.util.Writables; -import org.apache.hadoop.hbase.zookeeper.MasterSchemaChangeTracker; -import org.apache.hadoop.hbase.zookeeper.RegionServerTracker; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -84,7 +100,7 @@ class MockServer implements Server { this.ct = Mockito.mock(CatalogTracker.class); HRegionInterface hri = Mockito.mock(HRegionInterface.class); Mockito.when(this.ct.getConnection()).thenReturn(this.connection); - Mockito.when(ct.waitForMetaServerConnectionDefault()).thenReturn(hri); + Mockito.when(ct.waitForMetaServerConnection(Mockito.anyLong())).thenReturn(hri); } @Override @@ -111,7 +127,7 @@ public ZooKeeperWatcher getZooKeeper() { public void abort(String why, Throwable e) { //no-op } - + @Override public boolean isAborted() { return false; @@ -128,7 +144,7 @@ public void stop(String why) { this.ct.stop(); } if (this.connection != null) { - HConnectionManager.deleteConnection(this.connection.getConfiguration(), true); + HConnectionManager.deleteConnection(this.connection.getConfiguration()); } } } @@ -141,10 +157,15 @@ class MockMasterServices implements MasterServices { private final AssignmentManager asm; MockMasterServices(final Server server) throws IOException { - this.mfs = new MasterFileSystem(server, this, null); + this.mfs = new MasterFileSystem(server, this, null, false); this.asm = Mockito.mock(AssignmentManager.class); } + @Override + public void checkTableModifiable(byte[] tableName) throws IOException { + //no-op + } + @Override public void createTable(HTableDescriptor desc, byte[][] splitKeys) throws IOException { @@ -161,11 +182,6 @@ public ExecutorService getExecutorService() { return null; } - public void checkTableModifiable(byte[] tableName, - EventHandler.EventType eventType) - throws IOException { - } - @Override public MasterFileSystem getMasterFileSystem() { return this.mfs; @@ -188,7 +204,7 @@ public CatalogTracker getCatalogTracker() { @Override public Configuration getConfiguration() { - return null; + return mfs.conf; } @Override @@ -200,7 +216,7 @@ public ServerName getServerName() { public void abort(String why, Throwable e) { //no-op } - + @Override public boolean isAborted() { return false; @@ -223,43 +239,79 @@ public TableDescriptors getTableDescriptors() { return new TableDescriptors() { @Override public HTableDescriptor remove(String tablename) throws IOException { - // TODO Auto-generated method stub return null; } - + @Override public Map getAll() throws IOException { - // TODO Auto-generated method stub return null; } - + @Override public HTableDescriptor get(byte[] tablename) - throws FileNotFoundException, IOException { + throws IOException { return get(Bytes.toString(tablename)); } - + @Override public HTableDescriptor get(String tablename) - throws FileNotFoundException, IOException { + throws IOException { return createHTableDescriptor(); } - + @Override public void add(HTableDescriptor htd) throws IOException { - // TODO Auto-generated method stub - } }; } - public MasterSchemaChangeTracker getSchemaChangeTracker() { - return null; + @Override + public boolean isServerShutdownHandlerEnabled() { + return true; } - public RegionServerTracker getRegionServerTracker() { + @Override + public MasterCoprocessorHost getCoprocessorHost() { return null; } + + @Override + public boolean registerProtocol(Class protocol, T handler) { + return false; + } + + @Override + public void deleteTable(byte[] tableName) throws IOException { + } + + @Override + public void modifyTable(byte[] tableName, HTableDescriptor descriptor) throws IOException { + } + + @Override + public void enableTable(byte[] tableName) throws IOException { + } + + @Override + public void disableTable(byte[] tableName) throws IOException { + } + + @Override + public void addColumn(byte[] tableName, HColumnDescriptor column) throws IOException { + } + + @Override + public void modifyColumn(byte[] tableName, HColumnDescriptor descriptor) throws IOException { + } + + @Override + public void deleteColumn(byte[] tableName, byte[] columnName) throws IOException { + } + + @Override + public boolean shouldSplitMetaSeparately() { + return false; + } } @Test @@ -469,6 +521,333 @@ private void parentWithSpecifiedEndKeyCleanedEvenIfDaughterGoneFirst( janitor.join(); } + /** + * CatalogJanitor.scan() should not clean parent regions if their own + * parents are still referencing them. This ensures that grandfather regions + * do not point to deleted parent regions. + */ + @Test + public void testScanDoesNotCleanRegionsWithExistingParents() throws Exception { + HBaseTestingUtility htu = new HBaseTestingUtility(); + setRootDirAndCleanIt(htu, "testScanDoesNotCleanRegionsWithExistingParents"); + Server server = new MockServer(htu); + MasterServices services = new MockMasterServices(server); + + final HTableDescriptor htd = createHTableDescriptor(); + + // Create regions: aaa->{lastEndKey}, aaa->ccc, aaa->bbb, bbb->ccc, etc. + + // Parent + HRegionInfo parent = new HRegionInfo(htd.getName(), Bytes.toBytes("aaa"), + new byte[0], true); + // Sleep a second else the encoded name on these regions comes out + // same for all with same start key and made in same second. + Thread.sleep(1001); + + // Daughter a + HRegionInfo splita = new HRegionInfo(htd.getName(), Bytes.toBytes("aaa"), + Bytes.toBytes("ccc"), true); + Thread.sleep(1001); + // Make daughters of daughter a; splitaa and splitab. + HRegionInfo splitaa = new HRegionInfo(htd.getName(), Bytes.toBytes("aaa"), + Bytes.toBytes("bbb"), false); + HRegionInfo splitab = new HRegionInfo(htd.getName(), Bytes.toBytes("bbb"), + Bytes.toBytes("ccc"), false); + + // Daughter b + HRegionInfo splitb = new HRegionInfo(htd.getName(), Bytes.toBytes("ccc"), + new byte[0]); + Thread.sleep(1001); + + final Map splitParents = + new TreeMap(new SplitParentFirstComparator()); + splitParents.put(parent, makeResultFromHRegionInfo(parent, splita, splitb)); + splita.setOffline(true);//simulate that splita goes offline when it is split + splitParents.put(splita, makeResultFromHRegionInfo(splita, splitaa, splitab)); + + CatalogJanitor janitor = spy(new CatalogJanitor(server, services)); + doReturn(new Pair>( + 10, splitParents)).when(janitor).getSplitParents(); + + //create ref from splita to parent + Path splitaRef = + createReferences(services, htd, parent, splita, Bytes.toBytes("ccc"), false); + + //parent and A should not be removed + assertEquals(0, janitor.scan()); + + //now delete the ref + FileSystem fs = FileSystem.get(htu.getConfiguration()); + assertTrue(fs.delete(splitaRef, true)); + + //now, both parent, and splita can be deleted + assertEquals(2, janitor.scan()); + + services.stop("test finished"); + janitor.join(); + } + + @Test + public void testSplitParentFirstComparator() { + SplitParentFirstComparator comp = new SplitParentFirstComparator(); + final HTableDescriptor htd = createHTableDescriptor(); + + /* Region splits: + * + * rootRegion --- firstRegion --- firstRegiona + * | |- firstRegionb + * | + * |- lastRegion --- lastRegiona --- lastRegionaa + * | |- lastRegionab + * |- lastRegionb + * + * rootRegion : [] - [] + * firstRegion : [] - bbb + * lastRegion : bbb - [] + * firstRegiona : [] - aaa + * firstRegionb : aaa - bbb + * lastRegiona : bbb - ddd + * lastRegionb : ddd - [] + */ + + // root region + HRegionInfo rootRegion = new HRegionInfo(htd.getName(), HConstants.EMPTY_START_ROW, + HConstants.EMPTY_END_ROW, true); + HRegionInfo firstRegion = new HRegionInfo(htd.getName(), HConstants.EMPTY_START_ROW, + Bytes.toBytes("bbb"), true); + HRegionInfo lastRegion = new HRegionInfo(htd.getName(), Bytes.toBytes("bbb"), + HConstants.EMPTY_END_ROW, true); + + assertTrue(comp.compare(rootRegion, rootRegion) == 0); + assertTrue(comp.compare(firstRegion, firstRegion) == 0); + assertTrue(comp.compare(lastRegion, lastRegion) == 0); + assertTrue(comp.compare(rootRegion, firstRegion) < 0); + assertTrue(comp.compare(rootRegion, lastRegion) < 0); + assertTrue(comp.compare(firstRegion, lastRegion) < 0); + + //first region split into a, b + HRegionInfo firstRegiona = new HRegionInfo(htd.getName(), HConstants.EMPTY_START_ROW, + Bytes.toBytes("aaa"), true); + HRegionInfo firstRegionb = new HRegionInfo(htd.getName(), Bytes.toBytes("aaa"), + Bytes.toBytes("bbb"), true); + //last region split into a, b + HRegionInfo lastRegiona = new HRegionInfo(htd.getName(), Bytes.toBytes("bbb"), + Bytes.toBytes("ddd"), true); + HRegionInfo lastRegionb = new HRegionInfo(htd.getName(), Bytes.toBytes("ddd"), + HConstants.EMPTY_END_ROW, true); + + assertTrue(comp.compare(firstRegiona, firstRegiona) == 0); + assertTrue(comp.compare(firstRegionb, firstRegionb) == 0); + assertTrue(comp.compare(rootRegion, firstRegiona) < 0); + assertTrue(comp.compare(rootRegion, firstRegionb) < 0); + assertTrue(comp.compare(firstRegion, firstRegiona) < 0); + assertTrue(comp.compare(firstRegion, firstRegionb) < 0); + assertTrue(comp.compare(firstRegiona, firstRegionb) < 0); + + assertTrue(comp.compare(lastRegiona, lastRegiona) == 0); + assertTrue(comp.compare(lastRegionb, lastRegionb) == 0); + assertTrue(comp.compare(rootRegion, lastRegiona) < 0); + assertTrue(comp.compare(rootRegion, lastRegionb) < 0); + assertTrue(comp.compare(lastRegion, lastRegiona) < 0); + assertTrue(comp.compare(lastRegion, lastRegionb) < 0); + assertTrue(comp.compare(lastRegiona, lastRegionb) < 0); + + assertTrue(comp.compare(firstRegiona, lastRegiona) < 0); + assertTrue(comp.compare(firstRegiona, lastRegionb) < 0); + assertTrue(comp.compare(firstRegionb, lastRegiona) < 0); + assertTrue(comp.compare(firstRegionb, lastRegionb) < 0); + + HRegionInfo lastRegionaa = new HRegionInfo(htd.getName(), Bytes.toBytes("bbb"), + Bytes.toBytes("ccc"), false); + HRegionInfo lastRegionab = new HRegionInfo(htd.getName(), Bytes.toBytes("ccc"), + Bytes.toBytes("ddd"), false); + + assertTrue(comp.compare(lastRegiona, lastRegionaa) < 0); + assertTrue(comp.compare(lastRegiona, lastRegionab) < 0); + assertTrue(comp.compare(lastRegionaa, lastRegionab) < 0); + + } + + @Test + public void testArchiveOldRegion() throws Exception { + String table = "table"; + HBaseTestingUtility htu = new HBaseTestingUtility(); + setRootDirAndCleanIt(htu, "testCleanParent"); + Server server = new MockServer(htu); + MasterServices services = new MockMasterServices(server); + + // create the janitor + CatalogJanitor janitor = new CatalogJanitor(server, services); + + // Create regions. + HTableDescriptor htd = new HTableDescriptor(table); + htd.addFamily(new HColumnDescriptor("f")); + HRegionInfo parent = new HRegionInfo(htd.getName(), Bytes.toBytes("aaa"), Bytes.toBytes("eee")); + HRegionInfo splita = new HRegionInfo(htd.getName(), Bytes.toBytes("aaa"), Bytes.toBytes("ccc")); + HRegionInfo splitb = new HRegionInfo(htd.getName(), Bytes.toBytes("ccc"), Bytes.toBytes("eee")); + // Test that when both daughter regions are in place, that we do not + // remove the parent. + List kvs = new ArrayList(); + kvs.add(new KeyValue(parent.getRegionName(), HConstants.CATALOG_FAMILY, + HConstants.SPLITA_QUALIFIER, Writables.getBytes(splita))); + kvs.add(new KeyValue(parent.getRegionName(), HConstants.CATALOG_FAMILY, + HConstants.SPLITB_QUALIFIER, Writables.getBytes(splitb))); + Result r = new Result(kvs); + + FileSystem fs = FileSystem.get(htu.getConfiguration()); + Path rootdir = services.getMasterFileSystem().getRootDir(); + // have to set the root directory since we use it in HFileDisposer to figure out to get to the + // archive directory. Otherwise, it just seems to pick the first root directory it can find (so + // the single test passes, but when the full suite is run, things get borked). + FSUtils.setRootDir(fs.getConf(), rootdir); + Path tabledir = HTableDescriptor.getTableDir(rootdir, htd.getName()); + Path storedir = Store.getStoreHomedir(tabledir, parent.getEncodedName(), + htd.getColumnFamilies()[0].getName()); + + // delete the file and ensure that the files have been archived + Path storeArchive = HFileArchiveUtil.getStoreArchivePath(services.getConfiguration(), parent, + tabledir, htd.getColumnFamilies()[0].getName()); + + // enable archiving, make sure that files get archived + addMockStoreFiles(2, services, storedir); + // get the current store files for comparison + FileStatus[] storeFiles = fs.listStatus(storedir); + for (FileStatus file : storeFiles) { + System.out.println("Have store file:" + file.getPath()); + } + + // do the cleaning of the parent + assertTrue(janitor.cleanParent(parent, r)); + + // and now check to make sure that the files have actually been archived + FileStatus[] archivedStoreFiles = fs.listStatus(storeArchive); + assertArchiveEqualToOriginal(storeFiles, archivedStoreFiles, fs); + + // cleanup + services.stop("Test finished"); + server.stop("shutdown"); + janitor.join(); + } + + /** + * Test that if a store file with the same name is present as those already backed up cause the + * already archived files to be timestamped backup + */ + @Test + public void testDuplicateHFileResolution() throws Exception { + String table = "table"; + HBaseTestingUtility htu = new HBaseTestingUtility(); + setRootDirAndCleanIt(htu, "testCleanParent"); + Server server = new MockServer(htu); + MasterServices services = new MockMasterServices(server); + + // create the janitor + CatalogJanitor janitor = new CatalogJanitor(server, services); + + // Create regions. + HTableDescriptor htd = new HTableDescriptor(table); + htd.addFamily(new HColumnDescriptor("f")); + HRegionInfo parent = new HRegionInfo(htd.getName(), Bytes.toBytes("aaa"), Bytes.toBytes("eee")); + HRegionInfo splita = new HRegionInfo(htd.getName(), Bytes.toBytes("aaa"), Bytes.toBytes("ccc")); + HRegionInfo splitb = new HRegionInfo(htd.getName(), Bytes.toBytes("ccc"), Bytes.toBytes("eee")); + // Test that when both daughter regions are in place, that we do not + // remove the parent. + List kvs = new ArrayList(); + kvs.add(new KeyValue(parent.getRegionName(), HConstants.CATALOG_FAMILY, + HConstants.SPLITA_QUALIFIER, Writables.getBytes(splita))); + kvs.add(new KeyValue(parent.getRegionName(), HConstants.CATALOG_FAMILY, + HConstants.SPLITB_QUALIFIER, Writables.getBytes(splitb))); + Result r = new Result(kvs); + + FileSystem fs = FileSystem.get(htu.getConfiguration()); + + Path rootdir = services.getMasterFileSystem().getRootDir(); + // have to set the root directory since we use it in HFileDisposer to figure out to get to the + // archive directory. Otherwise, it just seems to pick the first root directory it can find (so + // the single test passes, but when the full suite is run, things get borked). + FSUtils.setRootDir(fs.getConf(), rootdir); + Path tabledir = HTableDescriptor.getTableDir(rootdir, parent.getTableName()); + Path storedir = Store.getStoreHomedir(tabledir, parent.getEncodedName(), + htd.getColumnFamilies()[0].getName()); + System.out.println("Old root:" + rootdir); + System.out.println("Old table:" + tabledir); + System.out.println("Old store:" + storedir); + + Path storeArchive = HFileArchiveUtil.getStoreArchivePath(services.getConfiguration(), parent, + tabledir, htd.getColumnFamilies()[0].getName()); + System.out.println("Old archive:" + storeArchive); + + // enable archiving, make sure that files get archived + addMockStoreFiles(2, services, storedir); + // get the current store files for comparison + FileStatus[] storeFiles = fs.listStatus(storedir); + + // do the cleaning of the parent + assertTrue(janitor.cleanParent(parent, r)); + + // and now check to make sure that the files have actually been archived + FileStatus[] archivedStoreFiles = fs.listStatus(storeArchive); + assertArchiveEqualToOriginal(storeFiles, archivedStoreFiles, fs); + + // now add store files with the same names as before to check backup + // enable archiving, make sure that files get archived + addMockStoreFiles(2, services, storedir); + + // do the cleaning of the parent + assertTrue(janitor.cleanParent(parent, r)); + + // and now check to make sure that the files have actually been archived + archivedStoreFiles = fs.listStatus(storeArchive); + assertArchiveEqualToOriginal(storeFiles, archivedStoreFiles, fs, true); + + // cleanup + services.stop("Test finished"); + server.stop("shutdown"); + janitor.join(); + } + + private void addMockStoreFiles(int count, MasterServices services, Path storedir) + throws IOException { + // get the existing store files + FileSystem fs = services.getMasterFileSystem().getFileSystem(); + fs.mkdirs(storedir); + // create the store files in the parent + for (int i = 0; i < count; i++) { + Path storeFile = new Path(storedir, "_store" + i); + FSDataOutputStream dos = fs.create(storeFile, true); + dos.writeBytes("Some data: " + i); + dos.close(); + } + // make sure the mock store files are there + FileStatus[] storeFiles = fs.listStatus(storedir); + assertEquals(count, storeFiles.length); + } + + private Result makeResultFromHRegionInfo(HRegionInfo region, HRegionInfo splita, + HRegionInfo splitb) throws IOException { + List kvs = new ArrayList(); + kvs.add(new KeyValue( + region.getRegionName(), + HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER, + Writables.getBytes(region))); + + if (splita != null) { + kvs.add(new KeyValue( + region.getRegionName(), + HConstants.CATALOG_FAMILY, HConstants.SPLITA_QUALIFIER, + Writables.getBytes(splita))); + } + + if (splitb != null) { + kvs.add(new KeyValue( + region.getRegionName(), + HConstants.CATALOG_FAMILY, HConstants.SPLITB_QUALIFIER, + Writables.getBytes(splitb))); + } + + return new Result(kvs); + } + private String setRootDirAndCleanIt(final HBaseTestingUtility htu, final String subdir) throws IOException { diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestClockSkewDetection.java b/src/test/java/org/apache/hadoop/hbase/master/TestClockSkewDetection.java index 977a5c7a7564..7c6c5694bb03 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestClockSkewDetection.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestClockSkewDetection.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,6 +18,8 @@ */ package org.apache.hadoop.hbase.master; +import static org.junit.Assert.fail; + import java.net.InetAddress; import junit.framework.Assert; @@ -82,18 +83,44 @@ public void stop(String why) { InetAddress ia1 = InetAddress.getLocalHost(); sm.regionServerStartup(ia1, 1234, -1, System.currentTimeMillis()); - long maxSkew = 30000; + final Configuration c = HBaseConfiguration.create(); + long maxSkew = c.getLong("hbase.master.maxclockskew", 30000); + long warningSkew = c.getLong("hbase.master.warningclockskew", 1000); try { + // Master Time > Region Server Time + LOG.debug("Test: Master Time > Region Server Time"); LOG.debug("regionServerStartup 2"); InetAddress ia2 = InetAddress.getLocalHost(); sm.regionServerStartup(ia2, 1235, -1, System.currentTimeMillis() - maxSkew * 2); - Assert.assertTrue("HMaster should have thrown an ClockOutOfSyncException " - + "but didn't.", false); - } catch(ClockOutOfSyncException e) { - //we want an exception - LOG.info("Recieved expected exception: "+e); + fail("HMaster should have thrown a ClockOutOfSyncException but didn't."); + } catch (ClockOutOfSyncException e) { + // we want an exception + LOG.info("Recieved expected exception: " + e); + } + + try { + // Master Time < Region Server Time + LOG.debug("Test: Master Time < Region Server Time"); + LOG.debug("regionServerStartup 3"); + InetAddress ia3 = InetAddress.getLocalHost(); + sm.regionServerStartup(ia3, 1236, -1, System.currentTimeMillis() + maxSkew * 2); + fail("HMaster should have thrown a ClockOutOfSyncException but didn't."); + } catch (ClockOutOfSyncException e) { + // we want an exception + LOG.info("Recieved expected exception: " + e); } + + // make sure values above warning threshold but below max threshold don't kill + LOG.debug("regionServerStartup 4"); + InetAddress ia4 = InetAddress.getLocalHost(); + sm.regionServerStartup(ia4, 1237, -1, System.currentTimeMillis() - warningSkew * 2); + + // make sure values above warning threshold but below max threshold don't kill + LOG.debug("regionServerStartup 5"); + InetAddress ia5 = InetAddress.getLocalHost(); + sm.regionServerStartup(ia5, 1238, -1, System.currentTimeMillis() + warningSkew * 2); + } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java b/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java index 40721cc0198d..dbb1e286f7e2 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java b/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java index 3a0c6b07e89b..73a2d3b28028 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,22 +18,29 @@ */ package org.apache.hadoop.hbase.master; -import static org.apache.hadoop.hbase.zookeeper.ZKSplitLog.Counters.*; +import static org.apache.hadoop.hbase.zookeeper.ZKSplitLog.Counters.tot_mgr_wait_for_zk_delete; +import static org.apache.hadoop.hbase.zookeeper.ZKSplitLog.Counters.tot_wkr_final_transistion_failed; +import static org.apache.hadoop.hbase.zookeeper.ZKSplitLog.Counters.tot_wkr_preempt_task; +import static org.apache.hadoop.hbase.zookeeper.ZKSplitLog.Counters.tot_wkr_task_acquired; +import static org.apache.hadoop.hbase.zookeeper.ZKSplitLog.Counters.tot_wkr_task_done; +import static org.apache.hadoop.hbase.zookeeper.ZKSplitLog.Counters.tot_wkr_task_err; +import static org.apache.hadoop.hbase.zookeeper.ZKSplitLog.Counters.tot_wkr_task_resigned; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; import java.io.IOException; +import java.util.Arrays; import java.util.Iterator; import java.util.List; import java.util.NavigableSet; import java.util.TreeSet; -import java.util.concurrent.atomic.AtomicLong; -import java.util.concurrent.Executors; import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; import java.util.concurrent.Future; -import java.util.concurrent.TimeoutException; import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicLong; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -43,7 +49,13 @@ import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.master.SplitLogManager.TaskBatch; @@ -54,15 +66,22 @@ import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; import org.apache.hadoop.hbase.util.FSUtils; -import org.apache.hadoop.hbase.util.Threads; +import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread; import org.apache.hadoop.hbase.util.JVMClusterUtil.RegionServerThread; +import org.apache.hadoop.hbase.util.Threads; +import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster; import org.apache.hadoop.hbase.zookeeper.ZKAssign; import org.apache.hadoop.hbase.zookeeper.ZKSplitLog; +import org.apache.hadoop.hbase.zookeeper.ZKUtil; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.log4j.Level; import org.apache.log4j.Logger; import org.apache.zookeeper.KeeperException; import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -80,15 +99,42 @@ public class TestDistributedLogSplitting { MiniHBaseCluster cluster; HMaster master; Configuration conf; - HBaseTestingUtility TEST_UTIL; + static HBaseTestingUtility TEST_UTIL; + static Configuration originalConf; + static MiniDFSCluster dfsCluster; + static MiniZooKeeperCluster zkCluster; + + @BeforeClass + public static void setup() throws Exception { + TEST_UTIL = new HBaseTestingUtility(HBaseConfiguration.create()); + dfsCluster = TEST_UTIL.startMiniDFSCluster(1); + zkCluster = TEST_UTIL.startMiniZKCluster(); + originalConf = TEST_UTIL.getConfiguration(); + } + + @AfterClass + public static void tearDown() throws Exception { + TEST_UTIL.shutdownMiniZKCluster(); + TEST_UTIL.shutdownMiniDFSCluster(); + TEST_UTIL.shutdownMiniHBaseCluster(); + } private void startCluster(int num_rs) throws Exception{ + conf = HBaseConfiguration.create(); + startCluster(NUM_MASTERS, num_rs, conf); + } + + private void startCluster(int num_master, int num_rs, Configuration inConf) throws Exception { ZKSplitLog.Counters.resetCounters(); LOG.info("Starting cluster"); - conf = HBaseConfiguration.create(); + this.conf = inConf; conf.getLong("hbase.splitlog.max.resubmit", 0); + // Make the failure test faster + conf.setInt("zookeeper.recovery.retry", 0); TEST_UTIL = new HBaseTestingUtility(conf); - TEST_UTIL.startMiniCluster(NUM_MASTERS, num_rs); + TEST_UTIL.setDFSCluster(dfsCluster); + TEST_UTIL.setZkCluster(zkCluster); + TEST_UTIL.startMiniHBaseCluster(num_master, num_rs); cluster = TEST_UTIL.getHBaseCluster(); LOG.info("Waiting for active/ready master"); cluster.waitForActiveAndReadyMaster(); @@ -98,9 +144,22 @@ private void startCluster(int num_rs) throws Exception{ } } + @Before + public void before() throws Exception { + // refresh configuration + conf = HBaseConfiguration.create(originalConf); + } + @After public void after() throws Exception { - TEST_UTIL.shutdownMiniCluster(); + if (TEST_UTIL.getHBaseCluster() != null) { + for (MasterThread mt : TEST_UTIL.getHBaseCluster().getLiveMasterThreads()) { + mt.getMaster().abort("closing...", new Exception("Trace info")); + } + } + TEST_UTIL.shutdownMiniHBaseCluster(); + TEST_UTIL.getTestFileSystem().delete(FSUtils.getRootDir(TEST_UTIL.getConfiguration()), true); + ZKUtil.deleteNodeRecursively(HBaseTestingUtility.getZooKeeperWatcher(TEST_UTIL), "/hbase"); } @Test (timeout=300000) @@ -166,9 +225,16 @@ public void testRecoveredEdits() throws Exception { List regions = null; HRegionServer hrs = null; for (int i = 0; i < NUM_RS; i++) { + boolean foundRS = false; hrs = rsts.get(i).getRegionServer(); regions = hrs.getOnlineRegions(); - if (regions.size() != 0) break; + for (HRegionInfo region : regions) { + if (region.getTableNameAsString().equalsIgnoreCase("table")) { + foundRS = true; + break; + } + } + if (foundRS) break; } final Path logDir = new Path(rootdir, HLog.getHLogDirectoryName(hrs .getServerName().toString())); @@ -203,6 +269,89 @@ public void testRecoveredEdits() throws Exception { assertEquals(NUM_LOG_LINES, count); } + @Test(timeout = 300000) + public void testMasterStartsUpWithLogSplittingWork() throws Exception { + LOG.info("testMasterStartsUpWithLogSplittingWork"); + Configuration curConf = HBaseConfiguration.create(); + curConf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, NUM_RS - 1); + startCluster(2, NUM_RS, curConf); + + final int NUM_REGIONS_TO_CREATE = 40; + final int NUM_LOG_LINES = 1000; + // turn off load balancing to prevent regions from moving around otherwise + // they will consume recovered.edits + master.balanceSwitch(false); + + List rsts = cluster.getLiveRegionServerThreads(); + final ZooKeeperWatcher zkw = new ZooKeeperWatcher(conf, "table-creation", null); + HTable ht = installTable(zkw, "table", "f", NUM_REGIONS_TO_CREATE); + + List regions = null; + HRegionServer hrs = null; + for (int i = 0; i < NUM_RS; i++) { + boolean isCarryingMeta = false; + hrs = rsts.get(i).getRegionServer(); + regions = hrs.getOnlineRegions(); + for (HRegionInfo region : regions) { + if (region.isRootRegion() || region.isMetaRegion()) { + isCarryingMeta = true; + break; + } + } + if (isCarryingMeta) { + continue; + } + break; + } + + LOG.info("#regions = " + regions.size()); + Iterator it = regions.iterator(); + while (it.hasNext()) { + HRegionInfo region = it.next(); + if (region.isMetaTable()) { + it.remove(); + } + } + makeHLog(hrs.getWAL(), regions, "table", NUM_LOG_LINES, 100); + + // abort master + abortMaster(cluster); + + // abort RS + int numRS = cluster.getLiveRegionServerThreads().size(); + LOG.info("Aborting region server: " + hrs.getServerName()); + hrs.abort("testing"); + + // wait for the RS dies + long start = EnvironmentEdgeManager.currentTimeMillis(); + while (cluster.getLiveRegionServerThreads().size() > (numRS - 1)) { + if (EnvironmentEdgeManager.currentTimeMillis() - start > 60000) { + assertTrue(false); + } + Thread.sleep(200); + } + + Thread.sleep(2000); + LOG.info("Current Open Regions:" + getAllOnlineRegions(cluster).size()); + + startMasterTillNoDeadServers(cluster); + + start = EnvironmentEdgeManager.currentTimeMillis(); + while (getAllOnlineRegions(cluster).size() < (NUM_REGIONS_TO_CREATE + 2)) { + if (EnvironmentEdgeManager.currentTimeMillis() - start > 60000) { + assertTrue("Timedout", false); + } + Thread.sleep(200); + } + + LOG.info("Current Open Regions After Master Node Starts Up:" + + getAllOnlineRegions(cluster).size()); + + assertEquals(NUM_LOG_LINES, TEST_UTIL.countRows(ht)); + + ht.close(); + } + /** * The original intention of this test was to force an abort of a region * server and to make sure that the failure path in the region servers is @@ -245,7 +394,7 @@ public void run() { slm.enqueueSplitTask(logfiles[0].getPath().toString(), batch); //waitForCounter but for one of the 2 counters long curt = System.currentTimeMillis(); - long waitTime = 30000; + long waitTime = 80000; long endt = curt + waitTime; while (curt < endt) { if ((tot_wkr_task_resigned.get() + tot_wkr_task_err.get() + @@ -267,7 +416,7 @@ public void run() { "tot_wkr_preempt_task"); } - @Test(timeout=25000) + @Test(timeout=30000) public void testDelayedDeleteOnFailure() throws Exception { LOG.info("testDelayedDeleteOnFailure"); startCluster(1); @@ -391,33 +540,40 @@ public void makeHLog(HLog log, List hris, String tname, int num_edits, int edit_size) throws IOException { + // remove root and meta region + hris.remove(HRegionInfo.ROOT_REGIONINFO); + hris.remove(HRegionInfo.FIRST_META_REGIONINFO); byte[] table = Bytes.toBytes(tname); HTableDescriptor htd = new HTableDescriptor(tname); byte[] value = new byte[edit_size]; for (int i = 0; i < edit_size; i++) { - value[i] = (byte)('a' + (i % 26)); + value[i] = (byte) ('a' + (i % 26)); } int n = hris.size(); int[] counts = new int[n]; - int j = 0; if (n > 0) { for (int i = 0; i < num_edits; i += 1) { WALEdit e = new WALEdit(); - byte [] row = Bytes.toBytes("r" + Integer.toString(i)); - byte [] family = Bytes.toBytes("f"); - byte [] qualifier = Bytes.toBytes("c" + Integer.toString(i)); - e.add(new KeyValue(row, family, qualifier, - System.currentTimeMillis(), value)); - j++; - log.append(hris.get(j % n), table, e, System.currentTimeMillis(), htd); - counts[j % n] += 1; + HRegionInfo curRegionInfo = hris.get(i % n); + byte[] startRow = curRegionInfo.getStartKey(); + if (startRow == null || startRow.length == 0) { + startRow = new byte[] { 0, 0, 0, 0, 1 }; + } + byte[] row = Bytes.incrementBytes(startRow, counts[i % n]); + row = Arrays.copyOfRange(row, 3, 8); // use last 5 bytes because + // HBaseTestingUtility.createMultiRegions use 5 bytes + // key + byte[] family = Bytes.toBytes("f"); + byte[] qualifier = Bytes.toBytes("c" + Integer.toString(i)); + e.add(new KeyValue(row, family, qualifier, System.currentTimeMillis(), value)); + log.append(curRegionInfo, table, e, System.currentTimeMillis(), htd); + counts[i % n] += 1; } } log.sync(); log.close(); for (int i = 0; i < n; i++) { - LOG.info("region " + hris.get(i).getRegionNameAsString() + - " has " + counts[i] + " edits"); + LOG.info("region " + hris.get(i).getRegionNameAsString() + " has " + counts[i] + " edits"); } return; } @@ -477,6 +633,30 @@ private void waitForCounter(AtomicLong ctr, long oldval, long newval, assertTrue(false); } + private void abortMaster(MiniHBaseCluster cluster) throws InterruptedException { + for (MasterThread mt : cluster.getLiveMasterThreads()) { + if (mt.getMaster().isActiveMaster()) { + mt.getMaster().abort("Aborting for tests", new Exception("Trace info")); + mt.join(); + break; + } + } + LOG.debug("Master is aborted"); + } + + private void startMasterTillNoDeadServers(MiniHBaseCluster cluster) + throws IOException, InterruptedException { + cluster.startMaster(); + HMaster master = cluster.getMaster(); + while (!master.isInitialized()) { + Thread.sleep(100); + } + ServerManager serverManager = master.getServerManager(); + while (serverManager.areDeadServersInProgress()) { + Thread.sleep(100); + } + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java b/src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java index 9ff83c50e0f4..33442910011c 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,6 +19,7 @@ package org.apache.hadoop.hbase.master; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -29,6 +29,7 @@ import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.ipc.HBaseRPC; import org.apache.hadoop.hbase.ipc.HMasterInterface; +import org.apache.hadoop.hbase.ipc.RpcEngine; import org.apache.hadoop.ipc.RemoteException; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -47,9 +48,11 @@ public void testRPCException() throws Exception { ServerName sm = hm.getServerName(); InetSocketAddress isa = new InetSocketAddress(sm.getHostname(), sm.getPort()); + RpcEngine rpcEngine = null; try { - HMasterInterface inf = (HMasterInterface) HBaseRPC.getProxy( - HMasterInterface.class, HMasterInterface.VERSION, isa, conf, 100); + rpcEngine = HBaseRPC.getProtocolEngine(conf); + HMasterInterface inf = rpcEngine.getProxy( + HMasterInterface.class, HMasterInterface.VERSION, isa, conf, 100 * 10); inf.isMasterRunning(); fail(); } catch (RemoteException ex) { @@ -57,6 +60,10 @@ public void testRPCException() throws Exception { "org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet")); } catch (Throwable t) { fail("Unexpected throwable: " + t); + } finally { + if (rpcEngine != null) { + rpcEngine.close(); + } } } diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java b/src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java index 379f70c0ca89..ec63d81b1d38 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java @@ -25,11 +25,14 @@ import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HServerLoad; +import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.regionserver.HRegionServer; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; +import org.junit.experimental.categories.Category; +@Category(MediumTests.class) public class TestMXBean { private static final HBaseTestingUtility TEST_UTIL = diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java b/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java index b7a8270cf3b5..4bcc544e9ca0 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -23,7 +22,6 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.catalog.MetaReader; -import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.executor.EventHandler; import org.apache.hadoop.hbase.executor.EventHandler.EventHandlerListener; @@ -55,7 +53,7 @@ public class TestMaster { @BeforeClass public static void beforeAllTests() throws Exception { // Start a cluster of two regionservers. - TEST_UTIL.startMiniCluster(1); + TEST_UTIL.startMiniCluster(2); } @AfterClass @@ -69,6 +67,8 @@ public void testMasterOpsWhileSplitting() throws Exception { HMaster m = cluster.getMaster(); HTable ht = TEST_UTIL.createTable(TABLENAME, FAMILYNAME); + assertTrue(m.assignmentManager.getZKTable().isEnabledTable + (Bytes.toString(TABLENAME))); TEST_UTIL.loadTable(ht, FAMILYNAME); ht.close(); @@ -114,6 +114,22 @@ public void testMasterOpsWhileSplitting() throws Exception { } } + @Test + public void testMoveRegionWhenNotInitialized() { + MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + HMaster m = cluster.getMaster(); + try { + m.initialized = false; // fake it, set back later + HRegionInfo meta = HRegionInfo.FIRST_META_REGIONINFO; + m.move(meta.getEncodedNameAsBytes(), null); + fail("Region should not be moved since master is not initialized"); + } catch (IOException ioe) { + assertTrue(ioe.getCause() instanceof PleaseHoldException); + } finally { + m.initialized = true; + } + } + static class RegionSplitListener implements EventHandlerListener { CountDownLatch split, proceed; @@ -145,4 +161,3 @@ public void beforeProcess(EventHandler event) { public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); } - diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java b/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java index 63882e199fe4..962b075cf2cc 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -24,7 +23,10 @@ import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; +import java.io.IOException; import java.util.ArrayList; +import java.util.HashSet; +import java.util.Iterator; import java.util.List; import java.util.Set; import java.util.TreeSet; @@ -34,7 +36,19 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.Abortable; +import org.apache.hadoop.hbase.ClusterStatus; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.MasterNotRunningException; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.executor.EventHandler.EventType; import org.apache.hadoop.hbase.executor.RegionTransitionData; import org.apache.hadoop.hbase.master.AssignmentManager.RegionState; @@ -43,9 +57,9 @@ import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.FSTableDescriptors; import org.apache.hadoop.hbase.util.JVMClusterUtil; -import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread; import org.apache.hadoop.hbase.util.JVMClusterUtil.RegionServerThread; +import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.hbase.zookeeper.ZKAssign; import org.apache.hadoop.hbase.zookeeper.ZKTable; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; @@ -56,6 +70,55 @@ public class TestMasterFailover { private static final Log LOG = LogFactory.getLog(TestMasterFailover.class); + @Test (timeout=180000) + public void testShouldCheckMasterFailOverWhenMETAIsInOpenedState() + throws Exception { + LOG.info("Starting testShouldCheckMasterFailOverWhenMETAIsInOpenedState"); + final int NUM_MASTERS = 1; + final int NUM_RS = 2; + + Configuration conf = HBaseConfiguration.create(); + conf.setInt("hbase.master.assignment.timeoutmonitor.period", 2000); + conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 8000); + // Start the cluster + HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(conf); + + TEST_UTIL.startMiniCluster(NUM_MASTERS, NUM_RS); + MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + + // Find regionserver carrying meta. + List regionServerThreads = + cluster.getRegionServerThreads(); + int count = -1; + HRegion metaRegion = null; + for (RegionServerThread regionServerThread : regionServerThreads) { + HRegionServer regionServer = regionServerThread.getRegionServer(); + metaRegion = regionServer.getOnlineRegion(HRegionInfo.FIRST_META_REGIONINFO.getRegionName()); + count++; + regionServer.abort(""); + if (null != metaRegion) break; + } + HRegionServer regionServer = cluster.getRegionServer(count); + + TEST_UTIL.shutdownMiniHBaseCluster(); + + // Create a ZKW to use in the test + ZooKeeperWatcher zkw = + HBaseTestingUtility.createAndForceNodeToOpenedState(TEST_UTIL, + metaRegion, regionServer.getServerName()); + + LOG.info("Staring cluster for second time"); + TEST_UTIL.startMiniHBaseCluster(NUM_MASTERS, NUM_RS); + + // Failover should be completed, now wait for no RIT + log("Waiting for no more RIT"); + ZKAssign.blockUntilNoRIT(zkw); + + zkw.close(); + // Stop the cluster + TEST_UTIL.shutdownMiniCluster(); + } + /** * Simple test of master failover. *

@@ -101,6 +164,7 @@ public void testSimpleMasterFailover() throws Exception { } assertEquals(1, numActive); assertEquals(NUM_MASTERS, masterThreads.size()); + LOG.info("Active master " + activeName); // Check that ClusterStatus reports the correct active and backup masters assertNotNull(active); @@ -110,16 +174,16 @@ public void testSimpleMasterFailover() throws Exception { assertEquals(2, status.getBackupMasters().size()); // attempt to stop one of the inactive masters - LOG.debug("\n\nStopping a backup master\n"); int backupIndex = (activeIndex == 0 ? 1 : activeIndex - 1); + HMaster master = cluster.getMaster(backupIndex); + LOG.debug("\n\nStopping a backup master: " + master.getServerName() + "\n"); cluster.stopMaster(backupIndex, false); cluster.waitOnMaster(backupIndex); - // verify still one active master and it's the same + // Verify still one active master and it's the same for (int i = 0; i < masterThreads.size(); i++) { if (masterThreads.get(i).getMaster().isActiveMaster()) { - assertTrue(activeName.equals( - masterThreads.get(i).getMaster().getServerName())); + assertTrue(activeName.equals(masterThreads.get(i).getMaster().getServerName())); activeIndex = i; active = masterThreads.get(activeIndex).getMaster(); } @@ -127,7 +191,7 @@ public void testSimpleMasterFailover() throws Exception { assertEquals(1, numActive); assertEquals(2, masterThreads.size()); int rsCount = masterThreads.get(activeIndex).getMaster().getClusterStatus().getServersSize(); - LOG.info("Active master managing " + rsCount + " regions servers"); + LOG.info("Active master " + active.getServerName() + " managing " + rsCount + " regions servers"); assertEquals(3, rsCount); // Check that ClusterStatus reports the correct active and backup masters @@ -138,7 +202,7 @@ public void testSimpleMasterFailover() throws Exception { assertEquals(1, status.getBackupMasters().size()); // kill the active master - LOG.debug("\n\nStopping the active master\n"); + LOG.debug("\n\nStopping the active master " + active.getServerName() + "\n"); cluster.stopMaster(activeIndex, false); cluster.waitOnMaster(activeIndex); @@ -159,7 +223,7 @@ public void testSimpleMasterFailover() throws Exception { assertEquals(0, status.getBackupMastersSize()); assertEquals(0, status.getBackupMasters().size()); int rss = status.getServersSize(); - LOG.info("Active master " + mastername.getHostname() + " managing " + + LOG.info("Active master " + mastername.getServerName() + " managing " + rss + " region servers"); assertEquals(3, rss); @@ -167,83 +231,6 @@ public void testSimpleMasterFailover() throws Exception { TEST_UTIL.shutdownMiniCluster(); } - @Test - public void testShouldCheckMasterFailOverWhenMETAIsInOpenedState() - throws Exception { - final int NUM_MASTERS = 1; - final int NUM_RS = 2; - - Configuration conf = HBaseConfiguration.create(); - conf.setInt("hbase.master.assignment.timeoutmonitor.period", 2000); - conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 8000); - // Start the cluster - HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(conf); - TEST_UTIL.startMiniCluster(NUM_MASTERS, NUM_RS); - MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); - - // get all the master threads - List masterThreads = cluster.getMasterThreads(); - - // wait for each to come online - for (MasterThread mt : masterThreads) { - assertTrue(mt.isAlive()); - } - - // verify only one is the active master and we have right number - int numActive = 0; - ServerName activeName = null; - for (int i = 0; i < masterThreads.size(); i++) { - if (masterThreads.get(i).getMaster().isActiveMaster()) { - numActive++; - activeName = masterThreads.get(i).getMaster().getServerName(); - } - } - assertEquals(1, numActive); - assertEquals(NUM_MASTERS, masterThreads.size()); - - // verify still one active master and it's the same - for (int i = 0; i < masterThreads.size(); i++) { - if (masterThreads.get(i).getMaster().isActiveMaster()) { - assertTrue(activeName.equals(masterThreads.get(i).getMaster() - .getServerName())); - } - } - assertEquals(1, numActive); - assertEquals(1, masterThreads.size()); - - List regionServerThreads = cluster - .getRegionServerThreads(); - int count = -1; - HRegion metaRegion = null; - for (RegionServerThread regionServerThread : regionServerThreads) { - HRegionServer regionServer = regionServerThread.getRegionServer(); - metaRegion = regionServer - .getOnlineRegion(HRegionInfo.FIRST_META_REGIONINFO.getRegionName()); - count++; - regionServer.abort(""); - if (null != metaRegion) { - break; - } - } - HRegionServer regionServer = cluster.getRegionServer(count); - - cluster.shutdown(); - // Create a ZKW to use in the test - ZooKeeperWatcher zkw = - HBaseTestingUtility.createAndForceNodeToOpenedState(TEST_UTIL, - metaRegion, regionServer.getServerName()); - - TEST_UTIL.startMiniHBaseCluster(1, 1); - - // Failover should be completed, now wait for no RIT - log("Waiting for no more RIT"); - ZKAssign.blockUntilNoRIT(zkw); - - // Stop the cluster - TEST_UTIL.shutdownMiniCluster(); - } - - /** * Complex test of master failover that tests as many permutations of the * different possible states that regions in transition could be in within ZK. @@ -334,8 +321,8 @@ public void testMasterFailoverWithMockedRIT() throws Exception { // Need to drop the timeout much lower conf.setInt("hbase.master.assignment.timeoutmonitor.period", 2000); conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 4000); - conf.setInt("hbase.master.wait.on.regionservers.mintostart", 3); - conf.setInt("hbase.master.wait.on.regionservers.maxtostart", 3); + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, 3); + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MAXTOSTART, 3); // Start the cluster HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(conf); @@ -379,7 +366,7 @@ public void testMasterFailoverWithMockedRIT() throws Exception { FSTableDescriptors.createTableDescriptor(filesystem, rootdir, htdEnabled); HRegionInfo hriEnabled = new HRegionInfo(htdEnabled.getName(), null, null); - HRegion.createHRegion(hriEnabled, rootdir, conf, htdEnabled); + createRegion(hriEnabled, rootdir, conf, htdEnabled); List enabledRegions = TEST_UTIL.createMultiRegionsInMeta( TEST_UTIL.getConfiguration(), htdEnabled, SPLIT_KEYS); @@ -390,7 +377,7 @@ public void testMasterFailoverWithMockedRIT() throws Exception { // Write the .tableinfo FSTableDescriptors.createTableDescriptor(filesystem, rootdir, htdDisabled); HRegionInfo hriDisabled = new HRegionInfo(htdDisabled.getName(), null, null); - HRegion.createHRegion(hriDisabled, rootdir, conf, htdDisabled); + createRegion(hriDisabled, rootdir, conf, htdDisabled); List disabledRegions = TEST_UTIL.createMultiRegionsInMeta( TEST_UTIL.getConfiguration(), htdDisabled, SPLIT_KEYS); @@ -408,7 +395,7 @@ public void testMasterFailoverWithMockedRIT() throws Exception { enabledAndAssignedRegions.add(enabledRegions.remove(0)); enabledAndAssignedRegions.add(enabledRegions.remove(0)); enabledAndAssignedRegions.add(closingRegion); - + List disabledAndAssignedRegions = new ArrayList(); disabledAndAssignedRegions.add(disabledRegions.remove(0)); disabledAndAssignedRegions.add(disabledRegions.remove(0)); @@ -623,7 +610,7 @@ public void testMasterFailoverWithMockedRIT() throws Exception { * * @throws Exception */ - @Test (timeout=180000) + @Test(timeout = 180000) public void testMasterFailoverWithMockedRITOnDeadRS() throws Exception { final int NUM_MASTERS = 1; @@ -632,10 +619,10 @@ public void testMasterFailoverWithMockedRITOnDeadRS() throws Exception { // Create config to use for this cluster Configuration conf = HBaseConfiguration.create(); // Need to drop the timeout much lower - conf.setInt("hbase.master.assignment.timeoutmonitor.period", 2000); - conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 4000); - conf.setInt("hbase.master.wait.on.regionservers.mintostart", 1); - conf.setInt("hbase.master.wait.on.regionservers.maxtostart", 2); + conf.setInt("hbase.master.assignment.timeoutmonitor.period", 4000); + conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 8000); + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, 1); + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MAXTOSTART, 2); // Create and start the cluster HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(conf); @@ -646,18 +633,18 @@ public void testMasterFailoverWithMockedRITOnDeadRS() throws Exception { // Create a ZKW to use in the test ZooKeeperWatcher zkw = new ZooKeeperWatcher(TEST_UTIL.getConfiguration(), "unittest", new Abortable() { - + @Override public void abort(String why, Throwable e) { LOG.error("Fatal ZK Error: " + why, e); org.junit.Assert.assertFalse("Fatal ZK error", true); } - + @Override public boolean isAborted() { return false; } - + }); // get all the master threads @@ -673,14 +660,10 @@ public boolean isAborted() { // disable load balancing on this master master.balanceSwitch(false); - // create two tables in META, each with 10 regions + // create two tables in META, each with 30 regions byte [] FAMILY = Bytes.toBytes("family"); - byte [][] SPLIT_KEYS = new byte [][] { - new byte[0], Bytes.toBytes("aaa"), Bytes.toBytes("bbb"), - Bytes.toBytes("ccc"), Bytes.toBytes("ddd"), Bytes.toBytes("eee"), - Bytes.toBytes("fff"), Bytes.toBytes("ggg"), Bytes.toBytes("hhh"), - Bytes.toBytes("iii"), Bytes.toBytes("jjj") - }; + byte[][] SPLIT_KEYS = + TEST_UTIL.getRegionSplitStartKeys(Bytes.toBytes("aaa"), Bytes.toBytes("zzz"), 30); byte [] enabledTable = Bytes.toBytes("enabledTable"); HTableDescriptor htdEnabled = new HTableDescriptor(enabledTable); @@ -692,7 +675,7 @@ public boolean isAborted() { FSTableDescriptors.createTableDescriptor(filesystem, rootdir, htdEnabled); HRegionInfo hriEnabled = new HRegionInfo(htdEnabled.getName(), null, null); - HRegion.createHRegion(hriEnabled, rootdir, conf, htdEnabled); + createRegion(hriEnabled, rootdir, conf, htdEnabled); List enabledRegions = TEST_UTIL.createMultiRegionsInMeta( TEST_UTIL.getConfiguration(), htdEnabled, SPLIT_KEYS); @@ -703,7 +686,7 @@ public boolean isAborted() { // Write the .tableinfo FSTableDescriptors.createTableDescriptor(filesystem, rootdir, htdDisabled); HRegionInfo hriDisabled = new HRegionInfo(htdDisabled.getName(), null, null); - HRegion.createHRegion(hriDisabled, rootdir, conf, htdDisabled); + createRegion(hriDisabled, rootdir, conf, htdDisabled); List disabledRegions = TEST_UTIL.createMultiRegionsInMeta( TEST_UTIL.getConfiguration(), htdDisabled, SPLIT_KEYS); @@ -725,11 +708,11 @@ public boolean isAborted() { // we'll need some regions to already be assigned out properly on live RS List enabledAndAssignedRegions = new ArrayList(); - enabledAndAssignedRegions.add(enabledRegions.remove(0)); - enabledAndAssignedRegions.add(enabledRegions.remove(0)); + enabledAndAssignedRegions.addAll(enabledRegions.subList(0, 6)); + enabledRegions.removeAll(enabledAndAssignedRegions); List disabledAndAssignedRegions = new ArrayList(); - disabledAndAssignedRegions.add(disabledRegions.remove(0)); - disabledAndAssignedRegions.add(disabledRegions.remove(0)); + disabledAndAssignedRegions.addAll(disabledRegions.subList(0, 6)); + disabledRegions.removeAll(disabledAndAssignedRegions); // now actually assign them for (HRegionInfo hri : enabledAndAssignedRegions) { @@ -743,13 +726,20 @@ public boolean isAborted() { master.assignRegion(hri); } + log("Waiting for assignment to finish"); + ZKAssign.blockUntilNoRIT(zkw); + master.assignmentManager.waitUntilNoRegionsInTransition(60000); + log("Assignment completed"); + + assertTrue(" Table must be enabled.", master.getAssignmentManager() + .getZKTable().isEnabledTable("enabledTable")); // we also need regions assigned out on the dead server List enabledAndOnDeadRegions = new ArrayList(); - enabledAndOnDeadRegions.add(enabledRegions.remove(0)); - enabledAndOnDeadRegions.add(enabledRegions.remove(0)); + enabledAndOnDeadRegions.addAll(enabledRegions.subList(0, 6)); + enabledRegions.removeAll(enabledAndOnDeadRegions); List disabledAndOnDeadRegions = new ArrayList(); - disabledAndOnDeadRegions.add(disabledRegions.remove(0)); - disabledAndOnDeadRegions.add(disabledRegions.remove(0)); + disabledAndOnDeadRegions.addAll(disabledRegions.subList(0, 6)); + disabledRegions.removeAll(disabledAndOnDeadRegions); // set region plan to server to be killed and trigger assign for (HRegionInfo hri : enabledAndOnDeadRegions) { @@ -766,8 +756,25 @@ public boolean isAborted() { // wait for no more RIT log("Waiting for assignment to finish"); ZKAssign.blockUntilNoRIT(zkw); + master.assignmentManager.waitUntilNoRegionsInTransition(60000); log("Assignment completed"); + // Due to master.assignRegion(hri) could fail to assign a region to a specified RS + // therefore, we need make sure that regions are in the expected RS + verifyRegionLocation(hrs, enabledAndAssignedRegions); + verifyRegionLocation(hrs, disabledAndAssignedRegions); + verifyRegionLocation(hrsDead, enabledAndOnDeadRegions); + verifyRegionLocation(hrsDead, disabledAndOnDeadRegions); + + assertTrue(" Didn't get enough regions of enabledTalbe on live rs.", + enabledAndAssignedRegions.size() >= 2); + assertTrue(" Didn't get enough regions of disalbedTable on live rs.", + disabledAndAssignedRegions.size() >= 2); + assertTrue(" Didn't get enough regions of enabledTalbe on dead rs.", + enabledAndOnDeadRegions.size() >= 2); + assertTrue(" Didn't get enough regions of disalbedTable on dead rs.", + disabledAndOnDeadRegions.size() >= 2); + // Stop the master log("Aborting master"); cluster.abortMaster(0); @@ -788,6 +795,9 @@ public boolean isAborted() { ZKTable zktable = new ZKTable(zkw); zktable.setDisabledTable(Bytes.toString(disabledTable)); + assertTrue(" The enabled table should be identified on master fail over.", + zktable.isEnabledTable("enabledTable")); + /* * ZK = CLOSING */ @@ -1019,6 +1029,169 @@ public boolean isAborted() { TEST_UTIL.shutdownMiniCluster(); } + @Test(timeout = 180000) + public void testRSKilledWithMockedOpeningRITGoingToDeadRS() throws Exception { + final int NUM_MASTERS = 1; + final int NUM_RS = 2; + + // Create config to use for this cluster + Configuration conf = HBaseConfiguration.create(); + // Need to drop the timeout much lower + conf.setInt("hbase.master.assignment.timeoutmonitor.period", 10000); + conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 30000); + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, 1); + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MAXTOSTART, 2); + + // Create and start the cluster + HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(conf); + TEST_UTIL.startMiniCluster(NUM_MASTERS, NUM_RS); + MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + log("Cluster started"); + + // Create a ZKW to use in the test + ZooKeeperWatcher zkw = + new ZooKeeperWatcher(TEST_UTIL.getConfiguration(), "unittest", new Abortable() { + + @Override + public void abort(String why, Throwable e) { + LOG.error("Fatal ZK Error: " + why, e); + org.junit.Assert.assertFalse("Fatal ZK error", true); + } + + @Override + public boolean isAborted() { + return false; + } + + }); + + // get all the master threads + List masterThreads = cluster.getMasterThreads(); + assertEquals(1, masterThreads.size()); + + // only one master thread, let's wait for it to be initialized + assertTrue(cluster.waitForActiveAndReadyMaster()); + HMaster master = masterThreads.get(0).getMaster(); + assertTrue(master.isActiveMaster()); + assertTrue(master.isInitialized()); + + // disable load balancing on this master + master.balanceSwitch(false); + + // create two tables in META, each with 30 regions + byte[] FAMILY = Bytes.toBytes("family"); + byte[][] SPLIT_KEYS = + TEST_UTIL.getRegionSplitStartKeys(Bytes.toBytes("aaa"), Bytes.toBytes("zzz"), 15); + + FileSystem filesystem = FileSystem.get(conf); + Path rootdir = filesystem.makeQualified(new Path(conf.get(HConstants.HBASE_DIR))); + + byte[] disabledTable = Bytes.toBytes("disabledTable"); + HTableDescriptor htdDisabled = new HTableDescriptor(disabledTable); + htdDisabled.addFamily(new HColumnDescriptor(FAMILY)); + // Write the .tableinfo + FSTableDescriptors.createTableDescriptor(filesystem, rootdir, htdDisabled); + HRegionInfo hriDisabled = new HRegionInfo(htdDisabled.getName(), null, null); + createRegion(hriDisabled, rootdir, conf, htdDisabled); + + List tableRegions = + TEST_UTIL.createMultiRegionsInMeta(TEST_UTIL.getConfiguration(), htdDisabled, SPLIT_KEYS); + + log("Regions in META have been created"); + + // at this point we only expect 2 regions to be assigned out (catalogs) + assertEquals(2, cluster.countServedRegions()); + + // The first RS will stay online + List regionservers = cluster.getRegionServerThreads(); + HRegionServer hrs = regionservers.get(0).getRegionServer(); + + // The second RS is going to be hard-killed + RegionServerThread hrsDeadThread = regionservers.get(1); + HRegionServer hrsDead = hrsDeadThread.getRegionServer(); + ServerName deadServerName = hrsDead.getServerName(); + + // we'll need some regions to already be assigned out properly on live RS + List assignedRegionsOnLiveRS = new ArrayList(); + assignedRegionsOnLiveRS.addAll(tableRegions.subList(0, 3)); + tableRegions.removeAll(assignedRegionsOnLiveRS); + + // now actually assign them + for (HRegionInfo hri : assignedRegionsOnLiveRS) { + master.assignmentManager.regionPlans.put(hri.getEncodedName(), + new RegionPlan(hri, null, hrs.getServerName())); + master.assignRegion(hri); + } + + log("Waiting for assignment to finish"); + ZKAssign.blockUntilNoRIT(zkw); + master.assignmentManager.waitUntilNoRegionsInTransition(60000); + log("Assignment completed"); + + // Due to master.assignRegion(hri) could fail to assign a region to a specified RS + // therefore, we need make sure that regions are in the expected RS + verifyRegionLocation(hrs, assignedRegionsOnLiveRS); + + assertTrue(" Table must be enabled.", master.getAssignmentManager().getZKTable() + .isEnabledTable("disabledTable")); + + assertTrue(" Didn't get enough regions of enabledTalbe on live rs.", + assignedRegionsOnLiveRS.size() >= 1); + + // Disable the disabledTable in ZK + ZKTable zktable = master.assignmentManager.getZKTable(); + zktable.setDisablingTable("disabledTable"); + + // RS was opening a region of disabled table then died + HRegionInfo region = assignedRegionsOnLiveRS.remove(0); + master.assignmentManager.regionOffline(region); + master.assignmentManager.regionsInTransition.put(region.getEncodedName(), new RegionState( + region, RegionState.State.OPENING, System.currentTimeMillis(), deadServerName)); + ZKAssign.createNodeOffline(zkw, region, deadServerName); + ZKAssign.transitionNodeOpening(zkw, region, deadServerName); + + // Kill the RS that had a hard death + log("Killing RS " + deadServerName); + hrsDead.abort("Killing for unit test"); + while (hrsDeadThread.isAlive()) { + Threads.sleep(10); + } + log("RS " + deadServerName + " killed"); + + log("Waiting for no more RIT"); + ZKAssign.blockUntilNoRIT(zkw); + log("No more RIT in ZK"); + assertTrue(master.assignmentManager.waitUntilNoRegionsInTransition(120000)); + } + + /** + * Verify regions are on the expected region server + */ + private void verifyRegionLocation(HRegionServer hrs, List regions) + throws IOException { + List tmpOnlineRegions = hrs.getOnlineRegions(); + Iterator itr = regions.iterator(); + while (itr.hasNext()) { + HRegionInfo tmp = itr.next(); + if (!tmpOnlineRegions.contains(tmp)) { + itr.remove(); + } + } + } + + HRegion createRegion(final HRegionInfo hri, final Path rootdir, final Configuration c, + final HTableDescriptor htd) + throws IOException { + HRegion r = HRegion.createHRegion(hri, rootdir, c, htd); + // The above call to create a region will create an hlog file. Each + // log file create will also create a running thread to do syncing. We need + // to close out this log else we will have a running thread trying to sync + // the file system continuously which is ugly when dfs is taken away at the + // end of the test. + HRegion.closeHRegion(r); + return r; + } + // TODO: Next test to add is with testing permutations of the RIT or the RS // killed are hosting ROOT and META regions. diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java b/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java new file mode 100644 index 000000000000..0781d1190dab --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystem.java @@ -0,0 +1,66 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master; + +import static org.junit.Assert.assertEquals; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test the master filesystem in a local cluster + */ +@Category(MediumTests.class) +public class TestMasterFileSystem { + + private static final Log LOG = LogFactory.getLog(TestMasterFileSystem.class); + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @BeforeClass + public static void setupTest() throws Exception { + UTIL.startMiniCluster(); + } + + @AfterClass + public static void teardownTest() throws Exception { + UTIL.shutdownMiniCluster(); + } + + @Test + public void testFsUriSetProperly() throws Exception { + HMaster master = UTIL.getMiniHBaseCluster().getMaster(); + MasterFileSystem fs = master.getMasterFileSystem(); + Path masterRoot = FSUtils.getRootDir(fs.conf); + Path rootDir = FSUtils.getRootDir(fs.getFileSystem().getConf()); + // make sure the fs and the found root dir have the same scheme + LOG.debug("from fs uri:" + FileSystem.getDefaultUri(fs.getFileSystem().getConf())); + LOG.debug("from configuration uri:" + FileSystem.getDefaultUri(fs.conf)); + // make sure the set uri matches by forcing it. + assertEquals(masterRoot, rootDir); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java b/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java index 777c67d9bd40..6c65b4e12a15 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,6 +19,7 @@ package org.apache.hadoop.hbase.master; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; import java.io.IOException; import java.util.List; @@ -96,6 +96,9 @@ public void testForCheckingIfEnableAndDisableWorksFineAfterSwitch() cluster.hbaseCluster.waitOnMaster(activeMaster); cluster.waitForActiveAndReadyMaster(); + assertTrue("The table should not be in enabled state", cluster.getMaster() + .getAssignmentManager().getZKTable().isDisablingOrDisabledTable( + "tableRestart")); log("Enabling table\n"); // Need a new Admin, the previous one is on the old master HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); @@ -108,6 +111,8 @@ public void testForCheckingIfEnableAndDisableWorksFineAfterSwitch() assertEquals( "The assigned regions were not onlined after master switch except for the catalog tables.", 6, regions.size()); + assertTrue("The table should be in enabled state", cluster.getMaster() + .getAssignmentManager().getZKTable().isEnabledTable("tableRestart")); ht.close(); TEST_UTIL.shutdownMiniCluster(); } diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java b/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java new file mode 100644 index 000000000000..01e6bd8e118f --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java @@ -0,0 +1,140 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.util.List; + +import org.apache.hadoop.hbase.*; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(LargeTests.class) +public class TestMasterShutdown { + private static final Log LOG = LogFactory.getLog(TestMasterShutdown.class); + + /** + * Simple test of shutdown. + *

+ * Starts with three masters. Tells the active master to shutdown the cluster. + * Verifies that all masters are properly shutdown. + * @throws Exception + */ + @Test (timeout=240000) + public void testMasterShutdown() throws Exception { + + final int NUM_MASTERS = 3; + final int NUM_RS = 3; + + // Create config to use for this cluster + Configuration conf = HBaseConfiguration.create(); + + // Start the cluster + HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(conf); + TEST_UTIL.startMiniCluster(NUM_MASTERS, NUM_RS); + MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + + // get all the master threads + List masterThreads = cluster.getMasterThreads(); + + // wait for each to come online + for (MasterThread mt : masterThreads) { + assertTrue(mt.isAlive()); + } + + // find the active master + HMaster active = null; + for (int i = 0; i < masterThreads.size(); i++) { + if (masterThreads.get(i).getMaster().isActiveMaster()) { + active = masterThreads.get(i).getMaster(); + break; + } + } + assertNotNull(active); + // make sure the other two are backup masters + ClusterStatus status = active.getClusterStatus(); + assertEquals(2, status.getBackupMastersSize()); + assertEquals(2, status.getBackupMasters().size()); + + // tell the active master to shutdown the cluster + active.shutdown(); + + for (int i = NUM_MASTERS - 1; i >= 0 ;--i) { + cluster.waitOnMaster(i); + } + // make sure all the masters properly shutdown + assertEquals(0,masterThreads.size()); + + TEST_UTIL.shutdownMiniCluster(); + } + + @Test//(timeout = 180000) + public void testMasterShutdownBeforeStartingAnyRegionServer() throws Exception { + + final int NUM_MASTERS = 1; + final int NUM_RS = 0; + + // Create config to use for this cluster + Configuration conf = HBaseConfiguration.create(); + + // Start the cluster + final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(conf); + TEST_UTIL.startMiniDFSCluster(1); + TEST_UTIL.startMiniZKCluster(); + TEST_UTIL.createRootDir(); + final LocalHBaseCluster cluster = + new LocalHBaseCluster(conf, NUM_MASTERS, NUM_RS, HMaster.class, + MiniHBaseCluster.MiniHBaseClusterRegionServer.class); + final MasterThread master = cluster.getMasters().get(0); + master.start(); + Thread shutdownThread = new Thread() { + public void run() { + try { + TEST_UTIL.getHBaseAdmin().shutdown(); + cluster.waitOnMaster(0); + } catch (Exception e) { + } + }; + }; + shutdownThread.start(); + master.join(); + shutdownThread.join(); + + List masterThreads = cluster.getMasters(); + // make sure all the masters properly shutdown + assertEquals(0, masterThreads.size()); + + TEST_UTIL.shutdownMiniZKCluster(); + TEST_UTIL.cleanupTestDir(); + TEST_UTIL.shutdownMiniDFSCluster(); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} + diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java b/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java index be5dea526e94..56f4fe8d4397 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,7 +18,8 @@ */ package org.apache.hadoop.hbase.master; -import static org.junit.Assert.*; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; import java.io.IOException; import java.io.StringWriter; @@ -31,15 +31,17 @@ import java.util.regex.Pattern; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.ServerName; import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.master.AssignmentManager.RegionState; -import org.apache.hadoop.hbase.master.HMaster; -import org.apache.hadoop.hbase.master.ServerManager; -import org.apache.hadoop.hbase.util.Bytes; -import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.apache.hadoop.hbase.tmpl.master.AssignmentManagerStatusTmpl; import org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.junit.Before; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -73,6 +75,11 @@ public void setupBasicMocks() { Mockito.doReturn(FAKE_HOST).when(master).getServerName(); Mockito.doReturn(conf).when(master).getConfiguration(); + // Fake ActiveMasterManager + ActiveMasterManager amm = Mockito.mock(ActiveMasterManager.class); + Mockito.doReturn(amm).when(master).getActiveMasterManager(); + Mockito.doReturn(FAKE_HOST).when(amm).getActiveMaster(); + // Fake serverManager ServerManager serverManager = Mockito.mock(ServerManager.class); Mockito.doReturn(1.0).when(serverManager).getAverageLoad(); @@ -94,6 +101,7 @@ public void setupBasicMocks() { // Mock admin admin = Mockito.mock(HBaseAdmin.class); + Mockito.when(admin.getConfiguration()).thenReturn(conf); } diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java b/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java index e0a486134838..3b4cc0d2ea44 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -59,10 +58,11 @@ public class TestMasterTransitions { TEST_UTIL.getConfiguration().setBoolean("dfs.support.append", true); TEST_UTIL.startMiniCluster(2); // Create a table of three families. This will assign a region. - TEST_UTIL.createTable(Bytes.toBytes(TABLENAME), FAMILIES); + byte[] tableName = Bytes.toBytes(TABLENAME); + TEST_UTIL.createTable(tableName, FAMILIES); HTable t = new HTable(TEST_UTIL.getConfiguration(), TABLENAME); int countOfRegions = TEST_UTIL.createMultiRegions(t, getTestFamily()); - TEST_UTIL.waitUntilAllRegionsAssigned(countOfRegions); + TEST_UTIL.waitUntilAllRegionsAssigned(tableName); addToEachStartKey(countOfRegions); t.close(); } diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java b/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java index 1b9b24ec560b..d643e90f087e 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java @@ -19,13 +19,31 @@ */ package org.apache.hadoop.hbase.master; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.List; +import java.util.Map; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.zookeeper.ZKAssign; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.apache.zookeeper.KeeperException; import org.junit.After; import org.junit.Before; @@ -46,6 +64,8 @@ public class TestMasterZKSessionRecovery { static { Configuration conf = TEST_UTIL.getConfiguration(); conf.setLong("hbase.master.zksession.recover.timeout", 50000); + conf.setClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, + MockLoadBalancer.class, LoadBalancer.class); } @Before @@ -60,37 +80,95 @@ public void tearDown() throws Exception { } /** - * Negative test of master recovery from zk session expiry. - *

- * Starts with one master. Fakes the master zk session expired. - * Ensures the master cannot recover the expired zk session since - * the master zk node is still there. - * @throws Exception + * Tests that the master does not call retainAssignment after recovery from + * expired zookeeper session. Without the HBASE-6046 fix master always tries + * to assign all the user regions by calling retainAssignment. */ - @Test(timeout=10000) - public void testMasterZKSessionRecoveryFailure() throws Exception { + @Test + public void testRegionAssignmentAfterMasterRecoveryDueToZKExpiry() throws Exception { MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + cluster.startRegionServer(); HMaster m = cluster.getMaster(); - m.abort("Test recovery from zk session expired", - new KeeperException.SessionExpiredException()); - assertTrue(m.isStopped()); + ZooKeeperWatcher zkw = m.getZooKeeperWatcher(); + int expectedNumOfListeners = zkw.getNumberOfListeners(); + // now the cluster is up. So assign some regions. + HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + byte[][] SPLIT_KEYS = new byte[][] { Bytes.toBytes("a"), Bytes.toBytes("b"), + Bytes.toBytes("c"), Bytes.toBytes("d"), Bytes.toBytes("e"), Bytes.toBytes("f"), + Bytes.toBytes("g"), Bytes.toBytes("h"), Bytes.toBytes("i"), Bytes.toBytes("j") }; + + String tableName = "testRegionAssignmentAfterMasterRecoveryDueToZKExpiry"; + admin.createTable(new HTableDescriptor(tableName), SPLIT_KEYS); + ZooKeeperWatcher zooKeeperWatcher = HBaseTestingUtility.getZooKeeperWatcher(TEST_UTIL); + ZKAssign.blockUntilNoRIT(zooKeeperWatcher); + m.getZooKeeperWatcher().close(); + MockLoadBalancer.retainAssignCalled = false; + m.abort("Test recovery from zk session expired", new KeeperException.SessionExpiredException()); + assertFalse(m.isStopped()); + // The recovered master should not call retainAssignment, as it is not a + // clean startup. + assertFalse("Retain assignment should not be called", MockLoadBalancer.retainAssignCalled); + // number of listeners should be same as the value before master aborted + assertEquals(expectedNumOfListeners, zkw.getNumberOfListeners()); + } + + static class MockLoadBalancer extends DefaultLoadBalancer { + static boolean retainAssignCalled = false; + + @Override + public Map> retainAssignment( + Map regions, List servers) { + retainAssignCalled = true; + return super.retainAssignment(regions, servers); + } } /** - * Positive test of master recovery from zk session expiry. - *

- * Starts with one master. Closes the master zk session. - * Ensures the master can recover the expired zk session. - * @throws Exception + * Tests whether the logs are split when master recovers from a expired + * zookeeper session and an RS goes down. */ - @Test(timeout=60000) - public void testMasterZKSessionRecoverySuccess() throws Exception { + @Test(timeout = 60000) + public void testLogSplittingAfterMasterRecoveryDueToZKExpiry() throws IOException, + KeeperException, InterruptedException { MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + cluster.startRegionServer(); HMaster m = cluster.getMaster(); + // now the cluster is up. So assign some regions. + HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + byte[][] SPLIT_KEYS = new byte[][] { Bytes.toBytes("1"), Bytes.toBytes("2"), + Bytes.toBytes("3"), Bytes.toBytes("4"), Bytes.toBytes("5") }; + + String tableName = "testLogSplittingAfterMasterRecoveryDueToZKExpiry"; + HTableDescriptor htd = new HTableDescriptor(tableName); + HColumnDescriptor hcd = new HColumnDescriptor("col"); + htd.addFamily(hcd); + admin.createTable(htd, SPLIT_KEYS); + ZooKeeperWatcher zooKeeperWatcher = HBaseTestingUtility.getZooKeeperWatcher(TEST_UTIL); + ZKAssign.blockUntilNoRIT(zooKeeperWatcher); + HTable table = new HTable(TEST_UTIL.getConfiguration(), tableName); + + Put p = null; + int numberOfPuts = 0; + for (numberOfPuts = 0; numberOfPuts < 6; numberOfPuts++) { + p = new Put(Bytes.toBytes(numberOfPuts)); + p.add(Bytes.toBytes("col"), Bytes.toBytes("ql"), Bytes.toBytes("value" + numberOfPuts)); + table.put(p); + } m.getZooKeeperWatcher().close(); - m.abort("Test recovery from zk session expired", - new KeeperException.SessionExpiredException()); + m.abort("Test recovery from zk session expired", new KeeperException.SessionExpiredException()); assertFalse(m.isStopped()); + cluster.getRegionServer(0).abort("Aborting"); + // Without patch for HBASE-6046 this test case will always timeout + // with patch the test case should pass. + Scan scan = new Scan(); + int numberOfRows = 0; + ResultScanner scanner = table.getScanner(scan); + Result[] result = scanner.next(1); + while (result != null && result.length > 0) { + numberOfRows++; + result = scanner.next(1); + } + assertEquals("Number of rows should be equal to number of puts.", numberOfPuts, numberOfRows); } } diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java b/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java index 6633a20b9fee..a38b49d9d798 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java @@ -73,7 +73,6 @@ public void setUp() throws Exception { public void tearDown() throws Exception { // Stop the cluster TEST_UTIL.shutdownMiniCluster(); - TEST_UTIL = new HBaseTestingUtility(resetConf); } @Test @@ -112,6 +111,7 @@ public void testOpenedRegionHandlerOnMasterRestart() throws Exception { @Test public void testShouldNotCompeleteOpenedRegionSuccessfullyIfVersionMismatches() throws Exception { + HRegion region = null; try { int testIndex = 0; TEST_UTIL.startMiniZKCluster(); @@ -120,8 +120,7 @@ public void testShouldNotCompeleteOpenedRegionSuccessfullyIfVersionMismatches() "testShouldNotCompeleteOpenedRegionSuccessfullyIfVersionMismatches"); HRegionInfo hri = new HRegionInfo(htd.getName(), Bytes.toBytes(testIndex), Bytes.toBytes(testIndex + 1)); - HRegion region = HRegion.createHRegion(hri, TEST_UTIL - .getDataTestDir(), TEST_UTIL.getConfiguration(), htd); + region = HRegion.createHRegion(hri, TEST_UTIL.getDataTestDir(), TEST_UTIL.getConfiguration(), htd); assertNotNull(region); AssignmentManager am = Mockito.mock(AssignmentManager.class); when(am.isRegionInTransition(hri)).thenReturn( @@ -160,6 +159,8 @@ public void testShouldNotCompeleteOpenedRegionSuccessfullyIfVersionMismatches() assertEquals("The region should not be opened successfully.", regionName, region.getRegionInfo().getEncodedName()); } finally { + region.close(); + region.getLog().closeAndDelete(); TEST_UTIL.shutdownMiniZKCluster(); } } diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java b/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java index ed3208c5ea75..70b2793dd094 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -22,7 +21,6 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; -import java.io.IOException; import java.util.List; import org.apache.commons.logging.Log; @@ -32,11 +30,11 @@ import org.apache.hadoop.hbase.client.MetaScanner; import org.apache.hadoop.hbase.executor.EventHandler.EventType; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.hbase.zookeeper.ZKAssign; import org.apache.hadoop.hbase.zookeeper.ZKUtil; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.junit.After; -import org.junit.Before; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -91,6 +89,9 @@ public class TestRestartCluster { @Test (timeout=300000) public void testClusterRestart() throws Exception { UTIL.startMiniCluster(3); + while (!UTIL.getMiniHBaseCluster().getMaster().isInitialized()) { + Threads.sleep(1); + } LOG.info("\n\nCreating tables"); for(byte [] TABLE : TABLES) { UTIL.createTable(TABLE, FAMILY); @@ -100,7 +101,7 @@ public void testClusterRestart() throws Exception { } List allRegions = - MetaScanner.listAllRegions(UTIL.getConfiguration()); + MetaScanner.listAllRegions(UTIL.getConfiguration(), true); assertEquals(3, allRegions.size()); LOG.info("\n\nShutting down cluster"); @@ -116,7 +117,7 @@ public void testClusterRestart() throws Exception { // Otherwise we're reusing an HConnection that has gone stale because // the shutdown of the cluster also called shut of the connection. allRegions = MetaScanner. - listAllRegions(new Configuration(UTIL.getConfiguration())); + listAllRegions(new Configuration(UTIL.getConfiguration()), true); assertEquals(3, allRegions.size()); LOG.info("\n\nWaiting for tables to be available"); diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java b/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java index 9462d114a552..e2d334ff8d9b 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java b/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java index 6602134aacee..e09e6c398c11 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -23,10 +22,10 @@ import java.io.IOException; import java.util.Arrays; -import java.util.List; import java.util.UUID; import java.util.concurrent.atomic.AtomicLong; +import junit.framework.Assert; import static org.junit.Assert.*; import org.apache.commons.logging.Log; @@ -48,12 +47,14 @@ import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.ZooDefs.Ids; import org.junit.After; -import org.junit.AfterClass; import org.junit.Before; -import org.junit.BeforeClass; +import org.junit.FixMethodOrder; import org.junit.Test; import org.junit.experimental.categories.Category; +import org.junit.runners.MethodSorters; +import org.mockito.Mockito; +@FixMethodOrder(MethodSorters.NAME_ASCENDING) @Category(MediumTests.class) public class TestSplitLogManager { private static final Log LOG = LogFactory.getLog(TestSplitLogManager.class); @@ -65,6 +66,9 @@ public class TestSplitLogManager { private static boolean stopped = false; private SplitLogManager slm; private Configuration conf; + private int to; + private final ServerManager sm = Mockito.mock(ServerManager.class); + private final MasterServices master = Mockito.mock(MasterServices.class); private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); @@ -82,14 +86,6 @@ public boolean isStopped() { }; - @BeforeClass - public static void setUpBeforeClass() throws Exception { - } - - @AfterClass - public static void tearDownAfterClass() throws Exception { - } - @Before public void setup() throws Exception { TEST_UTIL.startMiniZKCluster(); @@ -105,6 +101,16 @@ public void setup() throws Exception { stopped = false; resetCounters(); + to = 6000; + conf.setInt("hbase.splitlog.manager.timeout", to); + conf.setInt("hbase.splitlog.manager.unassigned.timeout", 2 * to); + conf.setInt("hbase.splitlog.manager.timeoutmonitor.period", 100); + to = to + 4 * 100; + + // By default, we let the test manage the error as before, so the server + // does not appear as dead from the master point of view, only from the split log pov. + Mockito.when(sm.isServerOnline(Mockito.any(ServerName.class))).thenReturn(true); + Mockito.when(master.getServerManager()).thenReturn(sm); } @After @@ -174,7 +180,7 @@ private String submitTaskAndWait(TaskBatch batch, String name) public void testTaskCreation() throws Exception { LOG.info("TestTaskCreation - test the creation of a task in zk"); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); TaskBatch batch = new TaskBatch(); @@ -194,26 +200,20 @@ public void testOrphanTaskAcquisition() throws Exception { TaskState.TASK_OWNED.get("dummy-worker"), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); - int to = 1000; - conf.setInt("hbase.splitlog.manager.timeout", to); - conf.setInt("hbase.splitlog.manager.timeoutmonitor.period", 100); - to = to + 2 * 100; - - - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); - waitForCounter(tot_mgr_orphan_task_acquired, 0, 1, 100); + waitForCounter(tot_mgr_orphan_task_acquired, 0, 1, to/2); Task task = slm.findOrCreateOrphanTask(tasknode); assertTrue(task.isOrphan()); - waitForCounter(tot_mgr_heartbeat, 0, 1, 100); + waitForCounter(tot_mgr_heartbeat, 0, 1, to/2); assertFalse(task.isUnassigned()); long curt = System.currentTimeMillis(); assertTrue((task.last_update <= curt) && (task.last_update > (curt - 1000))); LOG.info("waiting for manager to resubmit the orphan task"); - waitForCounter(tot_mgr_resubmit, 0, 1, to + 100); + waitForCounter(tot_mgr_resubmit, 0, 1, to + to/2); assertTrue(task.isUnassigned()); - waitForCounter(tot_mgr_rescan, 0, 1, to + 100); + waitForCounter(tot_mgr_rescan, 0, 1, to + to/2); } @Test @@ -227,14 +227,14 @@ public void testUnassignedOrphan() throws Exception { CreateMode.PERSISTENT); int version = ZKUtil.checkExists(zkw, tasknode); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); - waitForCounter(tot_mgr_orphan_task_acquired, 0, 1, 100); + waitForCounter(tot_mgr_orphan_task_acquired, 0, 1, to/2); Task task = slm.findOrCreateOrphanTask(tasknode); assertTrue(task.isOrphan()); assertTrue(task.isUnassigned()); // wait for RESCAN node to be created - waitForCounter(tot_mgr_rescan, 0, 1, 500); + waitForCounter(tot_mgr_rescan, 0, 1, to/2); Task task2 = slm.findOrCreateOrphanTask(tasknode); assertTrue(task == task2); LOG.debug("task = " + task); @@ -250,13 +250,8 @@ public void testUnassignedOrphan() throws Exception { public void testMultipleResubmits() throws Exception { LOG.info("TestMultipleResbmits - no indefinite resubmissions"); - int to = 1000; - conf.setInt("hbase.splitlog.manager.timeout", to); - conf.setInt("hbase.splitlog.manager.timeoutmonitor.period", 100); - to = to + 2 * 100; - conf.setInt("hbase.splitlog.max.resubmit", 2); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); TaskBatch batch = new TaskBatch(); @@ -264,19 +259,19 @@ public void testMultipleResubmits() throws Exception { int version = ZKUtil.checkExists(zkw, tasknode); ZKUtil.setData(zkw, tasknode, TaskState.TASK_OWNED.get("worker1")); - waitForCounter(tot_mgr_heartbeat, 0, 1, 1000); - waitForCounter(tot_mgr_resubmit, 0, 1, to + 100); + waitForCounter(tot_mgr_heartbeat, 0, 1, to/2); + waitForCounter(tot_mgr_resubmit, 0, 1, to + to/2); int version1 = ZKUtil.checkExists(zkw, tasknode); assertTrue(version1 > version); ZKUtil.setData(zkw, tasknode, TaskState.TASK_OWNED.get("worker2")); - waitForCounter(tot_mgr_heartbeat, 1, 2, 1000); - waitForCounter(tot_mgr_resubmit, 1, 2, to + 100); + waitForCounter(tot_mgr_heartbeat, 1, 2, to/2); + waitForCounter(tot_mgr_resubmit, 1, 2, to + to/2); int version2 = ZKUtil.checkExists(zkw, tasknode); assertTrue(version2 > version1); ZKUtil.setData(zkw, tasknode, TaskState.TASK_OWNED.get("worker3")); - waitForCounter(tot_mgr_heartbeat, 1, 2, 1000); - waitForCounter(tot_mgr_resubmit_threshold_reached, 0, 1, to + 100); - Thread.sleep(to + 100); + waitForCounter(tot_mgr_heartbeat, 1, 2, to/2); + waitForCounter(tot_mgr_resubmit_threshold_reached, 0, 1, to + to/2); + Thread.sleep(to + to/2); assertEquals(2L, tot_mgr_resubmit.get()); } @@ -284,9 +279,7 @@ public void testMultipleResubmits() throws Exception { public void testRescanCleanup() throws Exception { LOG.info("TestRescanCleanup - ensure RESCAN nodes are cleaned up"); - conf.setInt("hbase.splitlog.manager.timeout", 1000); - conf.setInt("hbase.splitlog.manager.timeoutmonitor.period", 100); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); TaskBatch batch = new TaskBatch(); @@ -294,33 +287,28 @@ public void testRescanCleanup() throws Exception { int version = ZKUtil.checkExists(zkw, tasknode); ZKUtil.setData(zkw, tasknode, TaskState.TASK_OWNED.get("worker1")); - waitForCounter(tot_mgr_heartbeat, 0, 1, 1000); + waitForCounter(tot_mgr_heartbeat, 0, 1, to/2); waitForCounter(new Expr() { @Override public long eval() { return (tot_mgr_resubmit.get() + tot_mgr_resubmit_failed.get()); } }, 0, 1, 5*60000); // wait long enough - if (tot_mgr_resubmit_failed.get() == 0) { - int version1 = ZKUtil.checkExists(zkw, tasknode); - assertTrue(version1 > version); - byte[] taskstate = ZKUtil.getData(zkw, tasknode); - assertTrue(Arrays.equals(TaskState.TASK_UNASSIGNED.get("dummy-master"), - taskstate)); - - waitForCounter(tot_mgr_rescan_deleted, 0, 1, 1000); - } else { - LOG.warn("Could not run test. Lost ZK connection?"); - } + Assert + .assertEquals("Could not run test. Lost ZK connection?", 0, tot_mgr_resubmit_failed.get()); + int version1 = ZKUtil.checkExists(zkw, tasknode); + assertTrue(version1 > version); + byte[] taskstate = ZKUtil.getData(zkw, tasknode); + assertTrue(Arrays.equals(TaskState.TASK_UNASSIGNED.get("dummy-master"), taskstate)); - return; + waitForCounter(tot_mgr_rescan_deleted, 0, 1, to / 2); } @Test public void testTaskDone() throws Exception { LOG.info("TestTaskDone - cleanup task node once in DONE state"); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); TaskBatch batch = new TaskBatch(); String tasknode = submitTaskAndWait(batch, "foo/1"); @@ -330,7 +318,7 @@ public void testTaskDone() throws Exception { batch.wait(); } } - waitForCounter(tot_mgr_task_deleted, 0, 1, 1000); + waitForCounter(tot_mgr_task_deleted, 0, 1, to/2); assertTrue(ZKUtil.checkExists(zkw, tasknode) == -1); } @@ -339,7 +327,7 @@ public void testTaskErr() throws Exception { LOG.info("TestTaskErr - cleanup task node once in ERR state"); conf.setInt("hbase.splitlog.max.resubmit", 0); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); TaskBatch batch = new TaskBatch(); @@ -350,7 +338,7 @@ public void testTaskErr() throws Exception { batch.wait(); } } - waitForCounter(tot_mgr_task_deleted, 0, 1, 1000); + waitForCounter(tot_mgr_task_deleted, 0, 1, to/2); assertTrue(ZKUtil.checkExists(zkw, tasknode) == -1); conf.setInt("hbase.splitlog.max.resubmit", ZKSplitLog.DEFAULT_MAX_RESUBMIT); } @@ -359,14 +347,14 @@ public void testTaskErr() throws Exception { public void testTaskResigned() throws Exception { LOG.info("TestTaskResigned - resubmit task node once in RESIGNED state"); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); TaskBatch batch = new TaskBatch(); String tasknode = submitTaskAndWait(batch, "foo/1"); ZKUtil.setData(zkw, tasknode, TaskState.TASK_RESIGNED.get("worker")); int version = ZKUtil.checkExists(zkw, tasknode); - waitForCounter(tot_mgr_resubmit, 0, 1, 1000); + waitForCounter(tot_mgr_resubmit, 0, 1, to/2); int version1 = ZKUtil.checkExists(zkw, tasknode); assertTrue(version1 > version); @@ -386,15 +374,9 @@ public void testUnassignedTimeout() throws Exception { TaskState.TASK_OWNED.get("dummy-worker"), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); - int to = 1000; - conf.setInt("hbase.splitlog.manager.timeout", to); - conf.setInt("hbase.splitlog.manager.unassigned.timeout", 2 * to); - conf.setInt("hbase.splitlog.manager.timeoutmonitor.period", 100); - - - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); - waitForCounter(tot_mgr_orphan_task_acquired, 0, 1, 100); + waitForCounter(tot_mgr_orphan_task_acquired, 0, 1, to/2); // submit another task which will stay in unassigned mode @@ -411,10 +393,10 @@ public void testUnassignedTimeout() throws Exception { // since we have stopped heartbeating the owned node therefore it should // get resubmitted LOG.info("waiting for manager to resubmit the orphan task"); - waitForCounter(tot_mgr_resubmit, 0, 1, to + 500); + waitForCounter(tot_mgr_resubmit, 0, 1, to + to/2); // now all the nodes are unassigned. manager should post another rescan - waitForCounter(tot_mgr_resubmit_unassigned, 0, 1, 2 * to + 500); + waitForCounter(tot_mgr_resubmit_unassigned, 0, 1, 2 * to + to/2); } @Test @@ -422,7 +404,7 @@ public void testDeadWorker() throws Exception { LOG.info("testDeadWorker"); conf.setLong("hbase.splitlog.max.resubmit", 0); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); TaskBatch batch = new TaskBatch(); @@ -430,10 +412,10 @@ public void testDeadWorker() throws Exception { int version = ZKUtil.checkExists(zkw, tasknode); ZKUtil.setData(zkw, tasknode, TaskState.TASK_OWNED.get("worker1")); - waitForCounter(tot_mgr_heartbeat, 0, 1, 1000); + waitForCounter(tot_mgr_heartbeat, 0, 1, to/2); slm.handleDeadWorker("worker1"); - waitForCounter(tot_mgr_resubmit, 0, 1, 1000); - waitForCounter(tot_mgr_resubmit_dead_server_task, 0, 1, 1000); + waitForCounter(tot_mgr_resubmit, 0, 1, to/2); + waitForCounter(tot_mgr_resubmit_dead_server_task, 0, 1, to + to/2); int version1 = ZKUtil.checkExists(zkw, tasknode); assertTrue(version1 > version); @@ -446,7 +428,7 @@ public void testDeadWorker() throws Exception { @Test public void testEmptyLogDir() throws Exception { LOG.info("testEmptyLogDir"); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); FileSystem fs = TEST_UTIL.getTestFileSystem(); Path emptyLogDirPath = new Path(fs.getWorkingDirectory(), @@ -456,48 +438,29 @@ public void testEmptyLogDir() throws Exception { assertFalse(fs.exists(emptyLogDirPath)); } - @Test(timeout=45000) - public void testVanishingTaskZNode() throws Exception { - LOG.info("testVanishingTaskZNode"); - conf.setInt("hbase.splitlog.manager.unassigned.timeout", 0); - slm = new SplitLogManager(zkw, conf, stopper, "dummy-master", null); + @Test + public void testWorkerCrash() throws Exception { + conf.setInt("hbase.splitlog.max.resubmit", ZKSplitLog.DEFAULT_MAX_RESUBMIT); + slm = new SplitLogManager(zkw, conf, stopper, master, "dummy-master", null); slm.finishInitialization(); - FileSystem fs = TEST_UTIL.getTestFileSystem(); - final Path logDir = new Path(fs.getWorkingDirectory(), - UUID.randomUUID().toString()); - fs.mkdirs(logDir); - Thread thread = null; - try { - Path logFile = new Path(logDir, UUID.randomUUID().toString()); - fs.createNewFile(logFile); - thread = new Thread() { - public void run() { - try { - // this call will block because there are no SplitLogWorkers, - // until the task znode is deleted below. Then the call will - // complete successfully, assuming the log is split. - slm.splitLogDistributed(logDir); - } catch (Exception e) { - LOG.warn("splitLogDistributed failed", e); - } - } - }; - thread.start(); - waitForCounter(tot_mgr_node_create_result, 0, 1, 10000); - String znode = ZKSplitLog.getEncodedNodeName(zkw, logFile.toString()); - // remove the task znode, to finish the distributed log splitting - ZKUtil.deleteNode(zkw, znode); - waitForCounter(tot_mgr_get_data_nonode, 0, 1, 30000); - waitForCounter(tot_mgr_log_split_batch_success, 0, 1, 1000); - assertTrue(fs.exists(logFile)); - } finally { - if (thread != null) { - // interrupt the thread in case the test fails in the middle. - // it has no effect if the thread is already terminated. - thread.interrupt(); - } - fs.delete(logDir, true); - } + TaskBatch batch = new TaskBatch(); + + String tasknode = submitTaskAndWait(batch, "foo/1"); + final ServerName worker1 = new ServerName("worker1,1,1"); + + ZKUtil.setData(zkw, tasknode, TaskState.TASK_OWNED.get(worker1.getServerName())); + if (tot_mgr_heartbeat.get() == 0) waitForCounter(tot_mgr_heartbeat, 0, 1, to / 2); + + // Not yet resubmitted. + Assert.assertEquals(0, tot_mgr_resubmit.get()); + + // This server becomes dead + Mockito.when(sm.isServerOnline(worker1)).thenReturn(false); + + Thread.sleep(1300); // The timeout checker is done every 1000 ms (hardcoded). + + // It has been resubmitted + Assert.assertEquals(1, tot_mgr_resubmit.get()); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java b/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java index c8e523f5e56e..b9381bdff040 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java +++ b/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -28,6 +27,7 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.client.Durability; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; @@ -38,6 +38,7 @@ import org.apache.hadoop.hbase.executor.EventHandler.EventType; import org.apache.hadoop.hbase.master.handler.TotesHRegionInfo; import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.hbase.util.Writables; @@ -47,9 +48,13 @@ import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; +import org.mockito.Mockito; +import org.mockito.internal.util.reflection.Whitebox; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; +import static org.junit.Assert.assertFalse; /** * Test open and close of regions using zk. @@ -65,6 +70,8 @@ public class TestZKBasedOpenCloseRegion { @BeforeClass public static void beforeAllTests() throws Exception { Configuration c = TEST_UTIL.getConfiguration(); + c.setClass(HConstants.REGION_SERVER_IMPL, TestZKBasedOpenCloseRegionRegionServer.class, + HRegionServer.class); c.setBoolean("dfs.support.append", true); c.setInt("hbase.regionserver.info.port", 0); TEST_UTIL.startMiniCluster(2); @@ -90,6 +97,22 @@ public class TestZKBasedOpenCloseRegion { waitUntilAllRegionsAssigned(); } + /** + * Special HRegionServer used in these tests that allows access to + * {@link #addRegionsInTransition(HRegionInfo, String)}. + */ + public static class TestZKBasedOpenCloseRegionRegionServer extends HRegionServer { + public TestZKBasedOpenCloseRegionRegionServer(Configuration conf) + throws IOException, InterruptedException { + super(conf); + } + @Override + public boolean addRegionsInTransition(HRegionInfo region, + String currentAction) throws RegionAlreadyInTransitionException { + return super.addRegionsInTransition(region, currentAction); + } + } + /** * Test we reopen a region once closed. * @throws Exception @@ -190,40 +213,6 @@ public void afterProcess(EventHandler event) { } } - public static class CloseRegionEventListener implements EventHandlerListener { - private static final Log LOG = LogFactory.getLog(CloseRegionEventListener.class); - String regionToClose; - AtomicBoolean closeEventProcessed; - - public CloseRegionEventListener(String regionToClose, - AtomicBoolean closeEventProcessed) { - this.regionToClose = regionToClose; - this.closeEventProcessed = closeEventProcessed; - } - - @Override - public void afterProcess(EventHandler event) { - LOG.info("afterProcess(" + event + ")"); - if(event.getEventType() == EventType.RS_ZK_REGION_CLOSED) { - LOG.info("Finished processing CLOSE REGION"); - TotesHRegionInfo hriCarrier = (TotesHRegionInfo)event; - if (regionToClose.equals(hriCarrier.getHRegionInfo().getRegionNameAsString())) { - LOG.info("Setting closeEventProcessed flag"); - closeEventProcessed.set(true); - } else { - LOG.info("Region to close didn't match"); - } - } - } - - @Override - public void beforeProcess(EventHandler event) { - if(event.getEventType() == EventType.M_RS_CLOSE_REGION) { - LOG.info("Received CLOSE RPC and beginning to process it"); - } - } - } - /** * This test shows how a region won't be able to be assigned to a RS * if it's already "processing" it. @@ -240,8 +229,10 @@ public void testRSAlreadyProcessingRegion() throws Exception { cluster.getLiveRegionServerThreads().get(1).getRegionServer(); HRegionInfo hri = getNonMetaRegion(hr0.getOnlineRegions()); - // fake that hr1 is processing the region - hr1.getRegionsInTransitionInRS().putIfAbsent(hri.getEncodedNameAsBytes(), true); + // Fake that hr1 is processing the region. At top of this test we made a + // regionserver that gave access addRegionsInTransition. Need to cast as + // TestZKBasedOpenCloseRegionRegionServer. + ((TestZKBasedOpenCloseRegionRegionServer) hr1).addRegionsInTransition(hri, "OPEN"); AtomicBoolean reopenEventProcessed = new AtomicBoolean(false); EventHandlerListener openListener = @@ -258,7 +249,7 @@ public void testRSAlreadyProcessingRegion() throws Exception { assertEquals(hr1.getOnlineRegion(hri.getEncodedNameAsBytes()), null); // remove the block and reset the boolean - hr1.getRegionsInTransitionInRS().remove(hri.getEncodedNameAsBytes()); + hr1.removeFromRegionsInTransition(hri); reopenEventProcessed.set(false); // now try moving a region when there is no region in transition. @@ -283,31 +274,30 @@ public void testRSAlreadyProcessingRegion() throws Exception { } - @Test (timeout=300000) public void testCloseRegion() - throws Exception { - LOG.info("Running testCloseRegion"); - MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); - LOG.info("Number of region servers = " + cluster.getLiveRegionServerThreads().size()); - - int rsIdx = 0; - HRegionServer regionServer = TEST_UTIL.getHBaseCluster().getRegionServer(rsIdx); - HRegionInfo hri = getNonMetaRegion(regionServer.getOnlineRegions()); - LOG.debug("Asking RS to close region " + hri.getRegionNameAsString()); - - AtomicBoolean closeEventProcessed = new AtomicBoolean(false); - EventHandlerListener listener = - new CloseRegionEventListener(hri.getRegionNameAsString(), - closeEventProcessed); - cluster.getMaster().executorService.registerListener(EventType.RS_ZK_REGION_CLOSED, listener); - - cluster.getMaster().assignmentManager.unassign(hri); - - while (!closeEventProcessed.get()) { - Threads.sleep(100); + /** + * If region open fails with IOException in openRegion() while doing tableDescriptors.get() + * the region should not add into regionsInTransitionInRS map + * @throws Exception + */ + @Test + public void testRegionOpenFailsDueToIOException() throws Exception { + HRegionInfo REGIONINFO = new HRegionInfo(Bytes.toBytes("t"), + HConstants.EMPTY_START_ROW, HConstants.EMPTY_START_ROW); + HRegionServer regionServer = TEST_UTIL.getHBaseCluster().getRegionServer(0); + TableDescriptors htd = Mockito.mock(TableDescriptors.class); + Object orizinalState = Whitebox.getInternalState(regionServer,"tableDescriptors"); + Whitebox.setInternalState(regionServer, "tableDescriptors", htd); + Mockito.doThrow(new IOException()).when(htd).get((byte[]) Mockito.any()); + try { + regionServer.openRegion(REGIONINFO); + fail("It should throw IOException "); + } catch (IOException e) { } - LOG.info("Done with testCloseRegion"); + Whitebox.setInternalState(regionServer, "tableDescriptors", orizinalState); + assertFalse("Region should not be in RIT", + regionServer.containsKeyInRegionsInTransition(REGIONINFO)); } - + private static void waitUntilAllRegionsAssigned() throws IOException { HTable meta = new HTable(TEST_UTIL.getConfiguration(), @@ -361,7 +351,7 @@ private static int addToEachStartKey(final int expected) throws IOException { // If start key, add 'aaa'. byte [] row = getStartKey(hri); Put p = new Put(row); - p.setWriteToWAL(false); + p.setDurability(Durability.SKIP_WAL); p.add(getTestFamily(), getTestQualifier(), row); t.put(p); rows++; @@ -386,16 +376,6 @@ private static int addToEachStartKey(final int expected) throws IOException { return getTestFamily(); } - public static void main(String args[]) throws Exception { - TestZKBasedOpenCloseRegion.beforeAllTests(); - - TestZKBasedOpenCloseRegion test = new TestZKBasedOpenCloseRegion(); - test.setup(); - test.testCloseRegion(); - - TestZKBasedOpenCloseRegion.afterAllTests(); - } - @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java new file mode 100644 index 000000000000..17c7109c4329 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java @@ -0,0 +1,334 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.cleaner; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.Stoppable; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.After; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; +import org.mockito.invocation.InvocationOnMock; +import org.mockito.stubbing.Answer; + +@Category(SmallTests.class) +public class TestCleanerChore { + + private static final Log LOG = LogFactory.getLog(TestCleanerChore.class); + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @After + public void cleanup() throws Exception { + // delete and recreate the test directory, ensuring a clean test dir between tests + UTIL.cleanupTestDir(); + } + + @Test + public void testSavesFilesOnRequest() throws Exception { + Stoppable stop = new StoppableImplementation(); + Configuration conf = UTIL.getConfiguration(); + Path testDir = UTIL.getDataTestDir(); + FileSystem fs = UTIL.getTestFileSystem(); + String confKey = "hbase.test.cleaner.delegates"; + conf.set(confKey, NeverDelete.class.getName()); + + AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey); + + // create the directory layout in the directory to clean + Path parent = new Path(testDir, "parent"); + Path file = new Path(parent, "someFile"); + fs.mkdirs(parent); + // touch a new file + fs.create(file).close(); + assertTrue("Test file didn't get created.", fs.exists(file)); + + // run the chore + chore.chore(); + + // verify all the files got deleted + assertTrue("File didn't get deleted", fs.exists(file)); + assertTrue("Empty directory didn't get deleted", fs.exists(parent)); + } + + @Test + public void testDeletesEmptyDirectories() throws Exception { + Stoppable stop = new StoppableImplementation(); + Configuration conf = UTIL.getConfiguration(); + Path testDir = UTIL.getDataTestDir(); + FileSystem fs = UTIL.getTestFileSystem(); + String confKey = "hbase.test.cleaner.delegates"; + conf.set(confKey, AlwaysDelete.class.getName()); + + AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey); + + // create the directory layout in the directory to clean + Path parent = new Path(testDir, "parent"); + Path child = new Path(parent, "child"); + Path emptyChild = new Path(parent, "emptyChild"); + Path file = new Path(child, "someFile"); + fs.mkdirs(child); + fs.mkdirs(emptyChild); + // touch a new file + fs.create(file).close(); + // also create a file in the top level directory + Path topFile = new Path(testDir, "topFile"); + fs.create(topFile).close(); + assertTrue("Test file didn't get created.", fs.exists(file)); + assertTrue("Test file didn't get created.", fs.exists(topFile)); + + // run the chore + chore.chore(); + + // verify all the files got deleted + assertFalse("File didn't get deleted", fs.exists(topFile)); + assertFalse("File didn't get deleted", fs.exists(file)); + assertFalse("Empty directory didn't get deleted", fs.exists(child)); + assertFalse("Empty directory didn't get deleted", fs.exists(parent)); + } + + /** + * Test to make sure that we don't attempt to ask the delegate whether or not we should preserve a + * directory. + * @throws Exception on failure + */ + @Test + public void testDoesNotCheckDirectories() throws Exception { + Stoppable stop = new StoppableImplementation(); + Configuration conf = UTIL.getConfiguration(); + Path testDir = UTIL.getDataTestDir(); + FileSystem fs = UTIL.getTestFileSystem(); + String confKey = "hbase.test.cleaner.delegates"; + conf.set(confKey, AlwaysDelete.class.getName()); + + AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey); + // spy on the delegate to ensure that we don't check for directories + AlwaysDelete delegate = (AlwaysDelete) chore.cleanersChain.get(0); + AlwaysDelete spy = Mockito.spy(delegate); + chore.cleanersChain.set(0, spy); + + // create the directory layout in the directory to clean + Path parent = new Path(testDir, "parent"); + Path file = new Path(parent, "someFile"); + fs.mkdirs(parent); + assertTrue("Test parent didn't get created.", fs.exists(parent)); + // touch a new file + fs.create(file).close(); + assertTrue("Test file didn't get created.", fs.exists(file)); + + FileStatus fStat = fs.getFileStatus(parent); + chore.chore(); + // make sure we never checked the directory + Mockito.verify(spy, Mockito.never()).isFileDeletable(fStat); + Mockito.reset(spy); + } + + @Test + public void testStoppedCleanerDoesNotDeleteFiles() throws Exception { + Stoppable stop = new StoppableImplementation(); + Configuration conf = UTIL.getConfiguration(); + Path testDir = UTIL.getDataTestDir(); + FileSystem fs = UTIL.getTestFileSystem(); + String confKey = "hbase.test.cleaner.delegates"; + conf.set(confKey, AlwaysDelete.class.getName()); + + AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey); + + // also create a file in the top level directory + Path topFile = new Path(testDir, "topFile"); + fs.create(topFile).close(); + assertTrue("Test file didn't get created.", fs.exists(topFile)); + + // stop the chore + stop.stop("testing stop"); + + // run the chore + chore.chore(); + + // test that the file still exists + assertTrue("File got deleted while chore was stopped", fs.exists(topFile)); + } + + /** + * While cleaning a directory, all the files in the directory may be deleted, but there may be + * another file added, in which case the directory shouldn't be deleted. + * @throws IOException on failure + */ + @Test + public void testCleanerDoesNotDeleteDirectoryWithLateAddedFiles() throws IOException { + Stoppable stop = new StoppableImplementation(); + Configuration conf = UTIL.getConfiguration(); + final Path testDir = UTIL.getDataTestDir(); + final FileSystem fs = UTIL.getTestFileSystem(); + String confKey = "hbase.test.cleaner.delegates"; + conf.set(confKey, AlwaysDelete.class.getName()); + + AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey); + // spy on the delegate to ensure that we don't check for directories + AlwaysDelete delegate = (AlwaysDelete) chore.cleanersChain.get(0); + AlwaysDelete spy = Mockito.spy(delegate); + chore.cleanersChain.set(0, spy); + + // create the directory layout in the directory to clean + final Path parent = new Path(testDir, "parent"); + Path file = new Path(parent, "someFile"); + fs.mkdirs(parent); + // touch a new file + fs.create(file).close(); + assertTrue("Test file didn't get created.", fs.exists(file)); + final Path addedFile = new Path(parent, "addedFile"); + + // when we attempt to delete the original file, add another file in the same directory + Mockito.doAnswer(new Answer() { + @Override + public Boolean answer(InvocationOnMock invocation) throws Throwable { + fs.create(addedFile).close(); + FSUtils.logFileSystemState(fs, testDir, LOG); + return (Boolean) invocation.callRealMethod(); + } + }).when(spy).isFileDeletable(Mockito.any(FileStatus.class)); + + // run the chore + chore.chore(); + + // make sure all the directories + added file exist, but the original file is deleted + assertTrue("Added file unexpectedly deleted", fs.exists(addedFile)); + assertTrue("Parent directory deleted unexpectedly", fs.exists(parent)); + assertFalse("Original file unexpectedly retained", fs.exists(file)); + Mockito.verify(spy, Mockito.times(1)).isFileDeletable(Mockito.any(FileStatus.class)); + Mockito.reset(spy); + } + + /** + * The cleaner runs in a loop, where it first checks to see all the files under a directory can be + * deleted. If they all can, then we try to delete the directory. However, a file may be added + * that directory to after the original check. This ensures that we don't accidentally delete that + * directory on and don't get spurious IOExceptions. + *

+ * This was from HBASE-7465. + * @throws Exception on failure + */ + @Test + public void testNoExceptionFromDirectoryWithRacyChildren() throws Exception { + Stoppable stop = new StoppableImplementation(); + // need to use a localutil to not break the rest of the test that runs on the local FS, which + // gets hosed when we start to use a minicluster. + HBaseTestingUtility localUtil = new HBaseTestingUtility(); + Configuration conf = localUtil.getConfiguration(); + final Path testDir = UTIL.getDataTestDir(); + final FileSystem fs = UTIL.getTestFileSystem(); + LOG.debug("Writing test data to: " + testDir); + String confKey = "hbase.test.cleaner.delegates"; + conf.set(confKey, AlwaysDelete.class.getName()); + + AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey); + // spy on the delegate to ensure that we don't check for directories + AlwaysDelete delegate = (AlwaysDelete) chore.cleanersChain.get(0); + AlwaysDelete spy = Mockito.spy(delegate); + chore.cleanersChain.set(0, spy); + + // create the directory layout in the directory to clean + final Path parent = new Path(testDir, "parent"); + Path file = new Path(parent, "someFile"); + fs.mkdirs(parent); + // touch a new file + fs.create(file).close(); + assertTrue("Test file didn't get created.", fs.exists(file)); + final Path racyFile = new Path(parent, "addedFile"); + + // when we attempt to delete the original file, add another file in the same directory + Mockito.doAnswer(new Answer() { + @Override + public Boolean answer(InvocationOnMock invocation) throws Throwable { + fs.create(racyFile).close(); + FSUtils.logFileSystemState(fs, testDir, LOG); + return (Boolean) invocation.callRealMethod(); + } + }).when(spy).isFileDeletable(Mockito.any(FileStatus.class)); + + // attempt to delete the directory, which + if (chore.checkAndDeleteDirectory(parent)) { + throw new Exception( + "Reported success deleting directory, should have failed when adding file mid-iteration"); + } + + // make sure all the directories + added file exist, but the original file is deleted + assertTrue("Added file unexpectedly deleted", fs.exists(racyFile)); + assertTrue("Parent directory deleted unexpectedly", fs.exists(parent)); + assertFalse("Original file unexpectedly retained", fs.exists(file)); + Mockito.verify(spy, Mockito.times(1)).isFileDeletable(Mockito.any(FileStatus.class)); + } + + private static class AllValidPaths extends CleanerChore { + + public AllValidPaths(String name, Stoppable s, Configuration conf, FileSystem fs, + Path oldFileDir, String confkey) { + super(name, Integer.MAX_VALUE, s, conf, fs, oldFileDir, confkey); + } + + // all paths are valid + @Override + protected boolean validate(Path file) { + return true; + } + }; + + public static class AlwaysDelete extends BaseHFileCleanerDelegate { + @Override + public boolean isFileDeletable(FileStatus fStat) { + return true; + } + } + + public static class NeverDelete extends BaseHFileCleanerDelegate { + @Override + public boolean isFileDeletable(FileStatus fStat) { + return false; + } + } + + /** + * Simple helper class that just keeps track of whether or not its stopped. + */ + private static class StoppableImplementation implements Stoppable { + private volatile boolean stop; + + @Override + public void stop(String why) { + this.stop = true; + } + + @Override + public boolean isStopped() { + return this.stop; + } + + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java new file mode 100644 index 000000000000..b49cbc2826e7 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java @@ -0,0 +1,241 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.cleaner; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.Server; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.util.EnvironmentEdge; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(MediumTests.class) +public class TestHFileCleaner { + private static final Log LOG = LogFactory.getLog(TestHFileCleaner.class); + + private final static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @BeforeClass + public static void setupCluster() throws Exception { + // have to use a minidfs cluster because the localfs doesn't modify file times correctly + UTIL.startMiniDFSCluster(1); + } + + @AfterClass + public static void shutdownCluster() throws Exception { + UTIL.shutdownMiniDFSCluster(); + } + + @Test + public void testTTLCleaner() throws IOException, InterruptedException { + FileSystem fs = UTIL.getDFSCluster().getFileSystem(); + Path root = UTIL.getDataTestDir(); + Path file = new Path(root, "file"); + fs.createNewFile(file); + long createTime = System.currentTimeMillis(); + assertTrue("Test file not created!", fs.exists(file)); + TimeToLiveHFileCleaner cleaner = new TimeToLiveHFileCleaner(); + // update the time info for the file, so the cleaner removes it + fs.setTimes(file, createTime - 100, -1); + Configuration conf = UTIL.getConfiguration(); + conf.setLong(TimeToLiveHFileCleaner.TTL_CONF_KEY, 100); + cleaner.setConf(conf); + assertTrue("File not set deletable - check mod time:" + getFileStats(file, fs) + + " with create time:" + createTime, cleaner.isFileDeletable(fs.getFileStatus(file))); + } + + /** + * @param file to check + * @return loggable information about the file + */ + private String getFileStats(Path file, FileSystem fs) throws IOException { + FileStatus status = fs.getFileStatus(file); + return "File" + file + ", mtime:" + status.getModificationTime() + ", atime:" + + status.getAccessTime(); + } + + @Test(timeout = 60 *1000) + public void testHFileCleaning() throws Exception { + final EnvironmentEdge originalEdge = EnvironmentEdgeManager.getDelegate(); + String prefix = "someHFileThatWouldBeAUUID"; + Configuration conf = UTIL.getConfiguration(); + // set TTL + long ttl = 2000; + conf.set(HFileCleaner.MASTER_HFILE_CLEANER_PLUGINS, + "org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner"); + conf.setLong(TimeToLiveHFileCleaner.TTL_CONF_KEY, ttl); + Server server = new DummyServer(); + Path archivedHfileDir = new Path(UTIL.getDataTestDir(), HConstants.HFILE_ARCHIVE_DIRECTORY); + FileSystem fs = FileSystem.get(conf); + HFileCleaner cleaner = new HFileCleaner(1000, server, conf, fs, archivedHfileDir); + + // Create 2 invalid files, 1 "recent" file, 1 very new file and 30 old files + final long createTime = System.currentTimeMillis(); + fs.delete(archivedHfileDir, true); + fs.mkdirs(archivedHfileDir); + // Case 1: 1 invalid file, which should be deleted directly + fs.createNewFile(new Path(archivedHfileDir, "dfd-dfd")); + // Case 2: 1 "recent" file, not even deletable for the first log cleaner + // (TimeToLiveLogCleaner), so we are not going down the chain + LOG.debug("Now is: " + createTime); + for (int i = 1; i < 32; i++) { + // Case 3: old files which would be deletable for the first log cleaner + // (TimeToLiveHFileCleaner), + Path fileName = new Path(archivedHfileDir, (prefix + "." + (createTime + i))); + fs.createNewFile(fileName); + // set the creation time past ttl to ensure that it gets removed + fs.setTimes(fileName, createTime - ttl - 1, -1); + LOG.debug("Creating " + getFileStats(fileName, fs)); + } + + // Case 2: 1 newer file, not even deletable for the first log cleaner + // (TimeToLiveLogCleaner), so we are not going down the chain + Path saved = new Path(archivedHfileDir, "thisFileShouldBeSaved.00000000000"); + fs.createNewFile(saved); + // set creation time in the future, so definitely within TTL + fs.setTimes(saved, createTime + (ttl * 2), -1); + LOG.debug("Creating " + getFileStats(saved, fs)); + + assertEquals(33, fs.listStatus(archivedHfileDir).length); + + // set a custom edge manager to handle time checking + EnvironmentEdge setTime = new EnvironmentEdge() { + @Override + public long currentTimeMillis() { + return createTime; + } + }; + EnvironmentEdgeManager.injectEdge(setTime); + + // run the chore + cleaner.chore(); + + for (FileStatus file : fs.listStatus(archivedHfileDir)) { + LOG.debug("Kept hfile: " + file.getPath()); + } + + // ensure we only end up with the saved file + assertEquals("Didn't dev expected number of files in the archive!", 1, + fs.listStatus(archivedHfileDir).length); + + cleaner.interrupt(); + // reset the edge back to the original edge + EnvironmentEdgeManager.injectEdge(originalEdge); + } + + @Test + public void testRemovesEmptyDirectories() throws Exception { + Configuration conf = UTIL.getConfiguration(); + // no cleaner policies = delete all files + conf.setStrings(HFileCleaner.MASTER_HFILE_CLEANER_PLUGINS, ""); + Server server = new DummyServer(); + Path archivedHfileDir = new Path(UTIL.getDataTestDir(), HConstants.HFILE_ARCHIVE_DIRECTORY); + + // setup the cleaner + FileSystem fs = UTIL.getDFSCluster().getFileSystem(); + HFileCleaner cleaner = new HFileCleaner(1000, server, conf, fs, archivedHfileDir); + + // make all the directories for archiving files + Path table = new Path(archivedHfileDir, "table"); + Path region = new Path(table, "regionsomthing"); + Path family = new Path(region, "fam"); + Path file = new Path(family, "file12345"); + fs.mkdirs(family); + if (!fs.exists(family)) throw new RuntimeException("Couldn't create test family:" + family); + fs.create(file).close(); + if (!fs.exists(file)) throw new RuntimeException("Test file didn't get created:" + file); + + // run the chore to cleanup the files (and the directories above it) + cleaner.chore(); + + // make sure all the parent directories get removed + assertFalse("family directory not removed for empty directory", fs.exists(family)); + assertFalse("region directory not removed for empty directory", fs.exists(region)); + assertFalse("table directory not removed for empty directory", fs.exists(table)); + assertTrue("archive directory", fs.exists(archivedHfileDir)); + } + + static class DummyServer implements Server { + + @Override + public Configuration getConfiguration() { + return UTIL.getConfiguration(); + } + + @Override + public ZooKeeperWatcher getZooKeeper() { + try { + return new ZooKeeperWatcher(getConfiguration(), "dummy server", this); + } catch (IOException e) { + e.printStackTrace(); + } + return null; + } + + @Override + public CatalogTracker getCatalogTracker() { + return null; + } + + @Override + public ServerName getServerName() { + return new ServerName("regionserver,60020,000000"); + } + + @Override + public void abort(String why, Throwable e) { + } + + @Override + public boolean isAborted() { + return false; + } + + @Override + public void stop(String why) {} + + @Override + public boolean isStopped() { + return false; + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} + diff --git a/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileLinkCleaner.java b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileLinkCleaner.java new file mode 100644 index 000000000000..c55084db99cd --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileLinkCleaner.java @@ -0,0 +1,174 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.cleaner; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.Server; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.backup.HFileArchiver; +import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.io.HFileLink; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.HFileArchiveUtil; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test the HFileLink Cleaner. + * HFiles with links cannot be deleted until a link is present. + */ +@Category(SmallTests.class) +public class TestHFileLinkCleaner { + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + @Test + public void testHFileLinkCleaning() throws Exception { + Configuration conf = TEST_UTIL.getConfiguration(); + conf.set(HConstants.HBASE_DIR, TEST_UTIL.getDataTestDir().toString()); + conf.set(HFileCleaner.MASTER_HFILE_CLEANER_PLUGINS, HFileLinkCleaner.class.getName()); + Path rootDir = FSUtils.getRootDir(conf); + FileSystem fs = FileSystem.get(conf); + + final String tableName = "test-table"; + final String tableLinkName = "test-link"; + final String hfileName = "1234567890"; + final String familyName = "cf"; + + HRegionInfo hri = new HRegionInfo(Bytes.toBytes(tableName)); + HRegionInfo hriLink = new HRegionInfo(Bytes.toBytes(tableLinkName)); + + Path archiveDir = HFileArchiveUtil.getArchivePath(conf); + Path archiveStoreDir = HFileArchiveUtil.getStoreArchivePath(conf, + tableName, hri.getEncodedName(), familyName); + Path archiveLinkStoreDir = HFileArchiveUtil.getStoreArchivePath(conf, + tableLinkName, hriLink.getEncodedName(), familyName); + + // Create hfile /hbase/table-link/region/cf/getEncodedName.HFILE(conf); + Path familyPath = getFamilyDirPath(archiveDir, tableName, hri.getEncodedName(), familyName); + fs.mkdirs(familyPath); + Path hfilePath = new Path(familyPath, hfileName); + fs.createNewFile(hfilePath); + + // Create link to hfile + Path familyLinkPath = getFamilyDirPath(rootDir, tableLinkName, + hriLink.getEncodedName(), familyName); + fs.mkdirs(familyLinkPath); + HFileLink.create(conf, fs, familyLinkPath, hri, hfileName); + Path linkBackRefDir = HFileLink.getBackReferencesDir(archiveStoreDir, hfileName); + assertTrue(fs.exists(linkBackRefDir)); + FileStatus[] backRefs = fs.listStatus(linkBackRefDir); + assertEquals(1, backRefs.length); + Path linkBackRef = backRefs[0].getPath(); + + // Initialize cleaner + final long ttl = 1000; + conf.setLong(TimeToLiveHFileCleaner.TTL_CONF_KEY, ttl); + Server server = new DummyServer(); + HFileCleaner cleaner = new HFileCleaner(1000, server, conf, fs, archiveDir); + + // Link backref cannot be removed + cleaner.chore(); + assertTrue(fs.exists(linkBackRef)); + assertTrue(fs.exists(hfilePath)); + + // Link backref can be removed + fs.rename(new Path(rootDir, tableLinkName), new Path(archiveDir, tableLinkName)); + cleaner.chore(); + assertFalse("Link should be deleted", fs.exists(linkBackRef)); + + // HFile can be removed + Thread.sleep(ttl * 2); + cleaner.chore(); + assertFalse("HFile should be deleted", fs.exists(hfilePath)); + + // Remove everything + for (int i = 0; i < 4; ++i) { + Thread.sleep(ttl * 2); + cleaner.chore(); + } + assertFalse("HFile should be deleted", fs.exists(new Path(archiveDir, tableName))); + assertFalse("Link should be deleted", fs.exists(new Path(archiveDir, tableLinkName))); + + cleaner.interrupt(); + } + + private static Path getFamilyDirPath (final Path rootDir, final String table, + final String region, final String family) { + return new Path(new Path(new Path(rootDir, table), region), family); + } + + static class DummyServer implements Server { + + @Override + public Configuration getConfiguration() { + return TEST_UTIL.getConfiguration(); + } + + @Override + public ZooKeeperWatcher getZooKeeper() { + try { + return new ZooKeeperWatcher(getConfiguration(), "dummy server", this); + } catch (IOException e) { + e.printStackTrace(); + } + return null; + } + + @Override + public CatalogTracker getCatalogTracker() { + return null; + } + + @Override + public ServerName getServerName() { + return new ServerName("regionserver,60020,000000"); + } + + @Override + public void abort(String why, Throwable e) {} + + @Override + public boolean isAborted() { + return false; + } + + @Override + public void stop(String why) {} + + @Override + public boolean isStopped() { + return false; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java similarity index 98% rename from src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java rename to src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java index ffda68d0f17c..5ae1235beace 100644 --- a/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java +++ b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -17,7 +16,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.hadoop.hbase.master; +package org.apache.hadoop.hbase.master.cleaner; import static org.junit.Assert.assertEquals; @@ -31,6 +30,7 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.master.cleaner.LogCleaner; import org.apache.hadoop.hbase.replication.ReplicationZookeeper; import org.apache.hadoop.hbase.replication.regionserver.Replication; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; diff --git a/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java new file mode 100644 index 000000000000..ffb59e569201 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java @@ -0,0 +1,372 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.cleaner; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.master.snapshot.DisabledTableSnapshotHandler; +import org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.snapshot.HSnapshotDescription; +import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils; +import org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils; +import org.apache.hadoop.hbase.snapshot.UnknownSnapshotException; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.HFileArchiveUtil; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +import com.google.common.collect.Lists; + +/** + * Test the master-related aspects of a snapshot + */ +@Category(MediumTests.class) +public class TestSnapshotFromMaster { + + private static final Log LOG = LogFactory.getLog(TestSnapshotFromMaster.class); + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static final int NUM_RS = 2; + private static Path rootDir; + private static Path snapshots; + private static FileSystem fs; + private static HMaster master; + + // for hfile archiving test. + private static Path archiveDir; + private static final String STRING_TABLE_NAME = "test"; + private static final byte[] TEST_FAM = Bytes.toBytes("fam"); + private static final byte[] TABLE_NAME = Bytes.toBytes(STRING_TABLE_NAME); + // refresh the cache every 1/2 second + private static final long cacheRefreshPeriod = 500; + + /** + * Setup the config for the cluster + */ + @BeforeClass + public static void setupCluster() throws Exception { + setupConf(UTIL.getConfiguration()); + UTIL.startMiniCluster(NUM_RS); + fs = UTIL.getDFSCluster().getFileSystem(); + master = UTIL.getMiniHBaseCluster().getMaster(); + rootDir = master.getMasterFileSystem().getRootDir(); + snapshots = SnapshotDescriptionUtils.getSnapshotsDir(rootDir); + archiveDir = new Path(rootDir, HConstants.HFILE_ARCHIVE_DIRECTORY); + } + + private static void setupConf(Configuration conf) { + // disable the ui + conf.setInt("hbase.regionsever.info.port", -1); + // change the flush size to a small amount, regulating number of store files + conf.setInt("hbase.hregion.memstore.flush.size", 25000); + // so make sure we get a compaction when doing a load, but keep around some + // files in the store + conf.setInt("hbase.hstore.compaction.min", 3); + conf.setInt("hbase.hstore.compactionThreshold", 5); + // block writes if we get to 12 store files + conf.setInt("hbase.hstore.blockingStoreFiles", 12); + // drop the number of attempts for the hbase admin + conf.setInt("hbase.client.retries.number", 1); + // Ensure no extra cleaners on by default (e.g. TimeToLiveHFileCleaner) + conf.set(HFileCleaner.MASTER_HFILE_CLEANER_PLUGINS, ""); + conf.set(HConstants.HBASE_MASTER_LOGCLEANER_PLUGINS, ""); + // Enable snapshot + conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + conf.setLong(SnapshotHFileCleaner.HFILE_CACHE_REFRESH_PERIOD_CONF_KEY, cacheRefreshPeriod); + + // prevent aggressive region split + conf.set(HConstants.HBASE_REGION_SPLIT_POLICY_KEY, + ConstantSizeRegionSplitPolicy.class.getName()); + } + + @Before + public void setup() throws Exception { + UTIL.createTable(TABLE_NAME, TEST_FAM); + master.getSnapshotManagerForTesting().setSnapshotHandlerForTesting(STRING_TABLE_NAME, null); + } + + @After + public void tearDown() throws Exception { + UTIL.deleteTable(TABLE_NAME); + SnapshotTestingUtils.deleteAllSnapshots(UTIL.getHBaseAdmin()); + SnapshotTestingUtils.deleteArchiveDirectory(UTIL); + } + + @AfterClass + public static void cleanupTest() throws Exception { + try { + UTIL.shutdownMiniCluster(); + } catch (Exception e) { + // NOOP; + } + } + + /** + * Test that the contract from the master for checking on a snapshot are valid. + *

+ *

    + *
  1. If a snapshot fails with an error, we expect to get the source error.
    2. + *
  2. If there is no snapshot name supplied, we should get an error.
    3. + *
  3. If asking about a snapshot has hasn't occurred, you should get an error.
    4. + *
+ */ + @Test(timeout = 60000) + public void testIsDoneContract() throws Exception { + + String snapshotName = "asyncExpectedFailureTest"; + + // check that we get an exception when looking up snapshot where one hasn't happened + SnapshotTestingUtils.expectSnapshotDoneException(master, new HSnapshotDescription(), + UnknownSnapshotException.class); + + // and that we get the same issue, even if we specify a name + SnapshotDescription desc = SnapshotDescription.newBuilder() + .setName(snapshotName).setTable(STRING_TABLE_NAME).build(); + SnapshotTestingUtils.expectSnapshotDoneException(master, new HSnapshotDescription(desc), + UnknownSnapshotException.class); + + // set a mock handler to simulate a snapshot + DisabledTableSnapshotHandler mockHandler = Mockito.mock(DisabledTableSnapshotHandler.class); + Mockito.when(mockHandler.getException()).thenReturn(null); + Mockito.when(mockHandler.getSnapshot()).thenReturn(desc); + Mockito.when(mockHandler.isFinished()).thenReturn(new Boolean(true)); + Mockito.when(mockHandler.getCompletionTimestamp()) + .thenReturn(EnvironmentEdgeManager.currentTimeMillis()); + + master.getSnapshotManagerForTesting() + .setSnapshotHandlerForTesting(STRING_TABLE_NAME, mockHandler); + + // if we do a lookup without a snapshot name, we should fail - you should always know your name + SnapshotTestingUtils.expectSnapshotDoneException(master, new HSnapshotDescription(), + UnknownSnapshotException.class); + + // then do the lookup for the snapshot that it is done + boolean isDone = master.isSnapshotDone(new HSnapshotDescription(desc)); + assertTrue("Snapshot didn't complete when it should have.", isDone); + + // now try the case where we are looking for a snapshot we didn't take + desc = SnapshotDescription.newBuilder().setName("Not A Snapshot").build(); + SnapshotTestingUtils.expectSnapshotDoneException(master, new HSnapshotDescription(desc), + UnknownSnapshotException.class); + + // then create a snapshot to the fs and make sure that we can find it when checking done + snapshotName = "completed"; + Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir); + desc = desc.toBuilder().setName(snapshotName).build(); + SnapshotDescriptionUtils.writeSnapshotInfo(desc, snapshotDir, fs); + + isDone = master.isSnapshotDone(new HSnapshotDescription(desc)); + assertTrue("Completed, on-disk snapshot not found", isDone); + } + + @Test + public void testGetCompletedSnapshots() throws Exception { + // first check when there are no snapshots + List snapshots = master.getCompletedSnapshots(); + assertEquals("Found unexpected number of snapshots", 0, snapshots.size()); + + // write one snapshot to the fs + String snapshotName = "completed"; + Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir); + SnapshotDescription snapshot = SnapshotDescription.newBuilder().setName(snapshotName).build(); + SnapshotDescriptionUtils.writeSnapshotInfo(snapshot, snapshotDir, fs); + + // check that we get one snapshot + snapshots = master.getCompletedSnapshots(); + assertEquals("Found unexpected number of snapshots", 1, snapshots.size()); + List expected = Lists.newArrayList(new HSnapshotDescription(snapshot)); + assertEquals("Returned snapshots don't match created snapshots", expected, snapshots); + + // write a second snapshot + snapshotName = "completed_two"; + snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir); + snapshot = SnapshotDescription.newBuilder().setName(snapshotName).build(); + SnapshotDescriptionUtils.writeSnapshotInfo(snapshot, snapshotDir, fs); + expected.add(new HSnapshotDescription(snapshot)); + + // check that we get one snapshot + snapshots = master.getCompletedSnapshots(); + assertEquals("Found unexpected number of snapshots", 2, snapshots.size()); + assertEquals("Returned snapshots don't match created snapshots", expected, snapshots); + } + + @Test + public void testDeleteSnapshot() throws Exception { + + String snapshotName = "completed"; + SnapshotDescription snapshot = SnapshotDescription.newBuilder().setName(snapshotName).build(); + + try { + master.deleteSnapshot(new HSnapshotDescription(snapshot)); + fail("Master didn't throw exception when attempting to delete snapshot that doesn't exist"); + } catch (IOException e) { + LOG.debug("Correctly failed delete of non-existant snapshot:" + e.getMessage()); + } + + // write one snapshot to the fs + Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir); + SnapshotDescriptionUtils.writeSnapshotInfo(snapshot, snapshotDir, fs); + + // then delete the existing snapshot,which shouldn't cause an exception to be thrown + master.deleteSnapshot(new HSnapshotDescription(snapshot)); + } + + /** + * Test that the snapshot hfile archive cleaner works correctly. HFiles that are in snapshots + * should be retained, while those that are not in a snapshot should be deleted. + * @throws Exception on failure + */ + @Test + public void testSnapshotHFileArchiving() throws Exception { + HBaseAdmin admin = UTIL.getHBaseAdmin(); + // make sure we don't fail on listing snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + // load the table + UTIL.loadTable(new HTable(UTIL.getConfiguration(), TABLE_NAME), TEST_FAM); + + // disable the table so we can take a snapshot + admin.disableTable(TABLE_NAME); + + // take a snapshot of the table + String snapshotName = "snapshot"; + byte[] snapshotNameBytes = Bytes.toBytes(snapshotName); + admin.snapshot(snapshotNameBytes, TABLE_NAME); + + Configuration conf = master.getConfiguration(); + LOG.info("After snapshot File-System state"); + FSUtils.logFileSystemState(fs, rootDir, LOG); + + // ensure we only have one snapshot + SnapshotTestingUtils.assertOneSnapshotThatMatches(admin, snapshotNameBytes, TABLE_NAME); + + // renable the table so we can compact the regions + admin.enableTable(TABLE_NAME); + + // compact the files so we get some archived files for the table we just snapshotted + List regions = UTIL.getHBaseCluster().getRegions(TABLE_NAME); + for (HRegion region : regions) { + region.waitForFlushesAndCompactions(); // enable can trigger a compaction, wait for it. + region.compactStores(); + } + LOG.info("After compaction File-System state"); + FSUtils.logFileSystemState(fs, rootDir, LOG); + + // make sure the cleaner has run + LOG.debug("Running hfile cleaners"); + ensureHFileCleanersRun(); + LOG.info("After cleaners File-System state: " + rootDir); + FSUtils.logFileSystemState(fs, rootDir, LOG); + + // get the snapshot files for the table + Path snapshotTable = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir); + Path[] snapshotHFiles = SnapshotTestingUtils.listHFiles(fs, snapshotTable); + // check that the files in the archive contain the ones that we need for the snapshot + LOG.debug("Have snapshot hfiles:"); + for (Path file : snapshotHFiles) { + LOG.debug(file); + } + // get the archived files for the table + Collection files = getArchivedHFiles(archiveDir, rootDir, fs, STRING_TABLE_NAME); + + // and make sure that there is a proper subset + for (Path file : snapshotHFiles) { + assertTrue("Archived hfiles " + files + " is missing snapshot file:" + file, + files.contains(file.getName())); + } + + // delete the existing snapshot + admin.deleteSnapshot(snapshotNameBytes); + SnapshotTestingUtils.assertNoSnapshots(admin); + + // make sure that we don't keep around the hfiles that aren't in a snapshot + // make sure we wait long enough to refresh the snapshot hfile + List delegates = UTIL.getMiniHBaseCluster().getMaster() + .getHFileCleaner().cleanersChain; + for (BaseHFileCleanerDelegate delegate: delegates) { + if (delegate instanceof SnapshotHFileCleaner) { + ((SnapshotHFileCleaner)delegate).getFileCacheForTesting().triggerCacheRefreshForTesting(); + } + } + // run the cleaner again + LOG.debug("Running hfile cleaners"); + ensureHFileCleanersRun(); + LOG.info("After delete snapshot cleaners run File-System state"); + FSUtils.logFileSystemState(fs, rootDir, LOG); + + files = getArchivedHFiles(archiveDir, rootDir, fs, STRING_TABLE_NAME); + assertEquals("Still have some hfiles in the archive, when their snapshot has been deleted.", 0, + files.size()); + } + + /** + * @return all the HFiles for a given table that have been archived + * @throws IOException on expected failure + */ + private final Collection getArchivedHFiles(Path archiveDir, Path rootDir, + FileSystem fs, String tableName) throws IOException { + Path tableArchive = new Path(archiveDir, tableName); + Path[] archivedHFiles = SnapshotTestingUtils.listHFiles(fs, tableArchive); + List files = new ArrayList(archivedHFiles.length); + LOG.debug("Have archived hfiles: " + tableArchive); + for (Path file : archivedHFiles) { + LOG.debug(file); + files.add(file.getName()); + } + // sort the archived files + + Collections.sort(files); + return files; + } + + /** + * Make sure the {@link HFileCleaner HFileCleaners} run at least once + */ + private static void ensureHFileCleanersRun() { + UTIL.getHBaseCluster().getMaster().getHFileCleaner().chore(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/handler/TestCreateTableHandler.java b/src/test/java/org/apache/hadoop/hbase/master/handler/TestCreateTableHandler.java new file mode 100644 index 000000000000..938d9e0e9d8c --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/handler/TestCreateTableHandler.java @@ -0,0 +1,177 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.handler; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException; +import org.apache.hadoop.hbase.Server; +import org.apache.hadoop.hbase.TableExistsException; +import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.master.AssignmentManager; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.master.MasterFileSystem; +import org.apache.hadoop.hbase.master.ServerManager; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(MediumTests.class) +public class TestCreateTableHandler { + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final Log LOG = LogFactory.getLog(TestCreateTableHandler.class); + private static final byte[] FAMILYNAME = Bytes.toBytes("fam"); + private static boolean throwException = false; + + @Before + public void setUp() throws Exception { + TEST_UTIL.startMiniCluster(1); + } + + @After + public void tearDown() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + throwException = true; + } + + @Test + public void testCreateTableCalledTwiceAndFirstOneInProgress() throws Exception { + final byte[] tableName = Bytes.toBytes("testCreateTableCalledTwiceAndFirstOneInProgress"); + final MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + final HMaster m = cluster.getMaster(); + final HTableDescriptor desc = new HTableDescriptor(tableName); + desc.addFamily(new HColumnDescriptor(FAMILYNAME)); + final HRegionInfo[] hRegionInfos = new HRegionInfo[] { new HRegionInfo(desc.getName(), null, + null) }; + CustomCreateTableHandler handler = new CustomCreateTableHandler(m, m.getMasterFileSystem(), + m.getServerManager(), desc, cluster.getConfiguration(), hRegionInfos, + m.getCatalogTracker(), m.getAssignmentManager()); + throwException = true; + handler.process(); + throwException = false; + CustomCreateTableHandler handler1 = new CustomCreateTableHandler(m, m.getMasterFileSystem(), + m.getServerManager(), desc, cluster.getConfiguration(), hRegionInfos, + m.getCatalogTracker(), m.getAssignmentManager()); + handler1.process(); + for (int i = 0; i < 100; i++) { + if (!TEST_UTIL.getHBaseAdmin().isTableAvailable(tableName)) { + Thread.sleep(200); + } + } + assertTrue(TEST_UTIL.getHBaseAdmin().isTableEnabled(tableName)); + } + + @Test (timeout=300000) + public void testCreateTableWithSplitRegion() throws Exception { + final byte[] tableName = Bytes.toBytes("testCreateTableWithSplitRegion"); + final MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + final HMaster m = cluster.getMaster(); + final HTableDescriptor desc = new HTableDescriptor(tableName); + desc.addFamily(new HColumnDescriptor(FAMILYNAME)); + byte[] splitPoint = Bytes.toBytes("split-point"); + long ts = System.currentTimeMillis(); + HRegionInfo d1 = new HRegionInfo(tableName, null, splitPoint, false, ts); + HRegionInfo d2 = new HRegionInfo(tableName, splitPoint, null, false, ts + 1); + HRegionInfo parent = new HRegionInfo(tableName, null, null, true, ts + 2); + parent.setOffline(true); + + Path tempdir = m.getMasterFileSystem().getTempDir(); + FileSystem fs = m.getMasterFileSystem().getFileSystem(); + Path tempTableDir = FSUtils.getTablePath(tempdir, tableName); + fs.delete(tempTableDir, true); // Clean up temp table dir if exists + + final HRegionInfo[] hRegionInfos = new HRegionInfo[] {d1, d2, parent}; + CreateTableHandler handler = new CreateTableHandler(m, m.getMasterFileSystem(), + m.getServerManager(), desc, cluster.getConfiguration(), hRegionInfos, + m.getCatalogTracker(), m.getAssignmentManager()); + handler.process(); + for (int i = 0; i < 200; i++) { + if (!TEST_UTIL.getHBaseAdmin().isTableAvailable(tableName)) { + Thread.sleep(300); + } + } + assertTrue(TEST_UTIL.getHBaseAdmin().isTableEnabled(tableName)); + assertTrue(TEST_UTIL.getHBaseAdmin().isTableAvailable(tableName)); + List regions = m.getAssignmentManager().getRegionsOfTable(tableName); + assertFalse("Split parent should not be assigned", regions.contains(parent)); + } + + @Test (timeout=60000) + public void testMasterRestartAfterEnablingNodeIsCreated() throws Exception { + byte[] tableName = Bytes.toBytes("testMasterRestartAfterEnablingNodeIsCreated"); + final MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + final HMaster m = cluster.getMaster(); + final HTableDescriptor desc = new HTableDescriptor(tableName); + desc.addFamily(new HColumnDescriptor(FAMILYNAME)); + final HRegionInfo[] hRegionInfos = new HRegionInfo[] { new HRegionInfo(desc.getName(), null, + null) }; + CustomCreateTableHandler handler = new CustomCreateTableHandler(m, m.getMasterFileSystem(), + m.getServerManager(), desc, cluster.getConfiguration(), hRegionInfos, + m.getCatalogTracker(), m.getAssignmentManager()); + throwException = true; + handler.process(); + abortAndStartNewMaster(cluster); + assertTrue(cluster.getLiveMasterThreads().size() == 1); + } + + private void abortAndStartNewMaster(final MiniHBaseCluster cluster) throws IOException { + cluster.abortMaster(0); + cluster.waitOnMaster(0); + LOG.info("Starting new master"); + cluster.startMaster(); + LOG.info("Waiting for master to become active."); + cluster.waitForActiveAndReadyMaster(); + } + + private static class CustomCreateTableHandler extends CreateTableHandler { + public CustomCreateTableHandler(Server server, MasterFileSystem fileSystemManager, + ServerManager sm, HTableDescriptor hTableDescriptor, Configuration conf, + HRegionInfo[] newRegions, CatalogTracker ct, AssignmentManager am) + throws NotAllMetaRegionsOnlineException, TableExistsException, IOException { + super(server, fileSystemManager, sm, hTableDescriptor, conf, newRegions, ct, am); + } + + @Override + protected List handleCreateHdfsRegions(Path tableRootDir, String tableName) + throws IOException { + if (throwException) { + throw new IOException("Test throws exceptions."); + } + return super.handleCreateHdfsRegions(tableRootDir, tableName); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/handler/TestTableDeleteFamilyHandler.java b/src/test/java/org/apache/hadoop/hbase/master/handler/TestTableDeleteFamilyHandler.java new file mode 100644 index 000000000000..ad56bd4c1d77 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/handler/TestTableDeleteFamilyHandler.java @@ -0,0 +1,159 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.handler; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; + +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(LargeTests.class) +public class TestTableDeleteFamilyHandler { + + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final String TABLENAME = "column_family_handlers"; + private static final byte[][] FAMILIES = new byte[][] { Bytes.toBytes("cf1"), + Bytes.toBytes("cf2"), Bytes.toBytes("cf3") }; + + /** + * Start up a mini cluster and put a small table of empty regions into it. + * + * @throws Exception + */ + @BeforeClass + public static void beforeAllTests() throws Exception { + + TEST_UTIL.getConfiguration().setBoolean("dfs.support.append", true); + TEST_UTIL.startMiniCluster(2); + + // Create a table of three families. This will assign a region. + TEST_UTIL.createTable(Bytes.toBytes(TABLENAME), FAMILIES); + HTable t = new HTable(TEST_UTIL.getConfiguration(), TABLENAME); + + // Create multiple regions in all the three column families + TEST_UTIL.createMultiRegions(t, FAMILIES[0]); + + // Load the table with data for all families + TEST_UTIL.loadTable(t, FAMILIES); + + TEST_UTIL.flush(); + + t.close(); + } + + @AfterClass + public static void afterAllTests() throws Exception { + TEST_UTIL.deleteTable(Bytes.toBytes(TABLENAME)); + TEST_UTIL.shutdownMiniCluster(); + } + + @Before + public void setup() throws IOException, InterruptedException { + TEST_UTIL.ensureSomeRegionServersAvailable(2); + } + + @Test + public void deleteColumnFamilyWithMultipleRegions() throws Exception { + + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + HTableDescriptor beforehtd = admin.getTableDescriptor(Bytes + .toBytes(TABLENAME)); + + FileSystem fs = TEST_UTIL.getDFSCluster().getFileSystem(); + + // 1 - Check if table exists in descriptor + assertTrue(admin.isTableAvailable(TABLENAME)); + + // 2 - Check if all three families exist in descriptor + assertEquals(3, beforehtd.getColumnFamilies().length); + HColumnDescriptor[] families = beforehtd.getColumnFamilies(); + for (int i = 0; i < families.length; i++) { + + assertTrue(families[i].getNameAsString().equals("cf" + (i + 1))); + } + + // 3 - Check if table exists in FS + Path tableDir = new Path(TEST_UTIL.getDefaultRootDirPath().toString() + "/" + + TABLENAME); + assertTrue(fs.exists(tableDir)); + + // 4 - Check if all the 3 column families exist in FS + FileStatus[] fileStatus = fs.listStatus(tableDir); + for (int i = 0; i < fileStatus.length; i++) { + if (fileStatus[i].isDir() == true) { + FileStatus[] cf = fs.listStatus(fileStatus[i].getPath()); + int k = 1; + for (int j = 0; j < cf.length; j++) { + if (cf[j].isDir() == true + && cf[j].getPath().getName().startsWith(".") == false) { + assertTrue(cf[j].getPath().getName().equals("cf" + k)); + k++; + } + } + } + } + + // TEST - Disable and delete the column family + admin.disableTable(TABLENAME); + admin.deleteColumn(TABLENAME, "cf2"); + + // 5 - Check if only 2 column families exist in the descriptor + HTableDescriptor afterhtd = admin.getTableDescriptor(Bytes + .toBytes(TABLENAME)); + assertEquals(2, afterhtd.getColumnFamilies().length); + HColumnDescriptor[] newFamilies = afterhtd.getColumnFamilies(); + assertTrue(newFamilies[0].getNameAsString().equals("cf1")); + assertTrue(newFamilies[1].getNameAsString().equals("cf3")); + + // 6 - Check if the second column family is gone from the FS + fileStatus = fs.listStatus(tableDir); + for (int i = 0; i < fileStatus.length; i++) { + if (fileStatus[i].isDir() == true) { + FileStatus[] cf = fs.listStatus(fileStatus[i].getPath()); + for (int j = 0; j < cf.length; j++) { + if (cf[j].isDir() == true) { + assertFalse(cf[j].getPath().getName().equals("cf2")); + } + } + } + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/handler/TestTableDescriptorModification.java b/src/test/java/org/apache/hadoop/hbase/master/handler/TestTableDescriptorModification.java new file mode 100644 index 000000000000..8b6e8b991dcc --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/handler/TestTableDescriptorModification.java @@ -0,0 +1,160 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.handler; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.Set; + +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.master.MasterFileSystem; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSTableDescriptors; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Verify that the HTableDescriptor is updated after + * addColumn(), deleteColumn() and modifyTable() operations. + */ +@Category(LargeTests.class) +public class TestTableDescriptorModification { + + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final byte[] TABLE_NAME = Bytes.toBytes("table"); + private static final byte[] FAMILY_0 = Bytes.toBytes("cf0"); + private static final byte[] FAMILY_1 = Bytes.toBytes("cf1"); + + /** + * Start up a mini cluster and put a small table of empty regions into it. + * + * @throws Exception + */ + @BeforeClass + public static void beforeAllTests() throws Exception { + TEST_UTIL.startMiniCluster(1); + } + + @AfterClass + public static void afterAllTests() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + @Test + public void testModifyTable() throws IOException { + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + // Create a table with one family + HTableDescriptor baseHtd = new HTableDescriptor(TABLE_NAME); + baseHtd.addFamily(new HColumnDescriptor(FAMILY_0)); + admin.createTable(baseHtd); + admin.disableTable(TABLE_NAME); + try { + // Verify the table descriptor + verifyTableDescriptor(TABLE_NAME, FAMILY_0); + + // Modify the table adding another family and verify the descriptor + HTableDescriptor modifiedHtd = new HTableDescriptor(TABLE_NAME); + modifiedHtd.addFamily(new HColumnDescriptor(FAMILY_0)); + modifiedHtd.addFamily(new HColumnDescriptor(FAMILY_1)); + admin.modifyTable(TABLE_NAME, modifiedHtd); + verifyTableDescriptor(TABLE_NAME, FAMILY_0, FAMILY_1); + } finally { + admin.deleteTable(TABLE_NAME); + } + } + + @Test + public void testAddColumn() throws IOException { + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + // Create a table with two families + HTableDescriptor baseHtd = new HTableDescriptor(TABLE_NAME); + baseHtd.addFamily(new HColumnDescriptor(FAMILY_0)); + admin.createTable(baseHtd); + admin.disableTable(TABLE_NAME); + try { + // Verify the table descriptor + verifyTableDescriptor(TABLE_NAME, FAMILY_0); + + // Modify the table removing one family and verify the descriptor + admin.addColumn(TABLE_NAME, new HColumnDescriptor(FAMILY_1)); + verifyTableDescriptor(TABLE_NAME, FAMILY_0, FAMILY_1); + } finally { + admin.deleteTable(TABLE_NAME); + } + } + + @Test + public void testDeleteColumn() throws IOException { + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + // Create a table with two families + HTableDescriptor baseHtd = new HTableDescriptor(TABLE_NAME); + baseHtd.addFamily(new HColumnDescriptor(FAMILY_0)); + baseHtd.addFamily(new HColumnDescriptor(FAMILY_1)); + admin.createTable(baseHtd); + admin.disableTable(TABLE_NAME); + try { + // Verify the table descriptor + verifyTableDescriptor(TABLE_NAME, FAMILY_0, FAMILY_1); + + // Modify the table removing one family and verify the descriptor + admin.deleteColumn(TABLE_NAME, FAMILY_1); + verifyTableDescriptor(TABLE_NAME, FAMILY_0); + } finally { + admin.deleteTable(TABLE_NAME); + } + } + + private void verifyTableDescriptor(final byte[] tableName, final byte[]... families) + throws IOException { + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + + // Verify descriptor from master + HTableDescriptor htd = admin.getTableDescriptor(tableName); + verifyTableDescriptor(htd, tableName, families); + + // Verify descriptor from HDFS + MasterFileSystem mfs = TEST_UTIL.getMiniHBaseCluster().getMaster().getMasterFileSystem(); + Path tableDir = HTableDescriptor.getTableDir(mfs.getRootDir(), tableName); + htd = FSTableDescriptors.getTableDescriptor(mfs.getFileSystem(), tableDir); + verifyTableDescriptor(htd, tableName, families); + } + + private void verifyTableDescriptor(final HTableDescriptor htd, + final byte[] tableName, final byte[]... families) { + Set htdFamilies = htd.getFamiliesKeys(); + assertTrue(Bytes.equals(tableName, htd.getName())); + assertEquals(families.length, htdFamilies.size()); + for (byte[] familyName: families) { + assertTrue("Expected family " + Bytes.toString(familyName), htdFamilies.contains(familyName)); + } + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/master/metrics/TestMasterStatistics.java b/src/test/java/org/apache/hadoop/hbase/master/metrics/TestMasterStatistics.java new file mode 100644 index 000000000000..77dd9c2309c1 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/metrics/TestMasterStatistics.java @@ -0,0 +1,145 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.metrics; + +import static org.junit.Assert.*; + +import java.lang.management.ManagementFactory; + +import javax.management.MBeanServer; +import javax.management.ObjectName; + +import org.apache.hadoop.hbase.MediumTests; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Tests {@link MasterMetrics} and access to it through the + * {@link MasterStatistics} management bean. + * + * Note: this test must always be run in separate fork (process) + * because it changes static contents of metrics subsystem and + * is affected itself by that static contents. For that reason + * the test put into {@link MediumTests} Category. + */ +@Category(MediumTests.class) +public class TestMasterStatistics { + + @Before + @SuppressWarnings("deprecation") + public void ensureNullContext() throws Exception { + // Clean up the factory attributes to instantiate the NullContext, + // regardless if the resource "/hadoop-metrics.properties" is present + // in the class-path: + org.apache.hadoop.metrics.ContextFactory factory = + org.apache.hadoop.metrics.ContextFactory.getFactory(); + String[] attributeNames = factory.getAttributeNames(); + for (String attributeName: attributeNames) { + factory.removeAttribute(attributeName); + } + // ensure the attributes are cleaned up: + attributeNames = factory.getAttributeNames(); + assertEquals(0, attributeNames.length); + // Get the "hbase" context and ensure it is NullContext: + org.apache.hadoop.metrics.MetricsContext context + = org.apache.hadoop.metrics.MetricsUtil.getContext("hbase"); + assertTrue(context instanceof org.apache.hadoop.metrics.spi.NullContext); + assertTrue(!context.isMonitoring()); + } + + @Test + public void testMasterStatistics() throws Exception { + // No timer updates started here since NullContext is used, see #ensureNullContext(). + // (NullContext never starts the updater thread). + MasterMetrics masterMetrics = new MasterMetrics("foo"); + + try { + final MBeanServer server = ManagementFactory.getPlatformMBeanServer(); + final ObjectName objectName = new ObjectName( + "hadoop:name=MasterStatistics,service=Master"); + + masterMetrics.doUpdates(null); + + masterMetrics.resetAllMinMax(); + + masterMetrics.incrementRequests(10); + Thread.sleep(1001); + + masterMetrics.addSnapshot(1L); + masterMetrics.addSnapshotClone(2L); + masterMetrics.addSnapshotRestore(3L); + + // 3 times added split, average = (5+3+4)/3 = 4 + masterMetrics.addSplit(4L, 5L); + masterMetrics.addSplit(2L, 3L); + masterMetrics.addSplit(13L, 4L); + + masterMetrics.doUpdates(null); + + final float f = masterMetrics.getRequests(); + // f = 10/T, where T >= 1 sec. So, we assert that 0 < f <= 10: + if (f <= 0.0f || f > 10.0f) { + fail("Unexpected rate value: " + f); + } + Object attribute = server.getAttribute(objectName, "cluster_requests"); + float f2 = ((Float) attribute).floatValue(); + assertEquals("The value obtained through bean server should be equal to the one " + + "obtained directly.", f, f2, 1e-4); + + // NB: these 3 metrics are not pushed upon masterMetrics.doUpdates(), + // so they always return null: + attribute = server.getAttribute(objectName, "snapshotTimeNumOps"); + assertEquals(Integer.valueOf(0), attribute); + attribute = server.getAttribute(objectName, "snapshotRestoreTimeNumOps"); + assertEquals(Integer.valueOf(0), attribute); + attribute = server.getAttribute(objectName, "snapshotCloneTimeNumOps"); + assertEquals(Integer.valueOf(0), attribute); + + attribute = server.getAttribute(objectName, "splitSizeNumOps"); + assertEquals(Integer.valueOf(3), attribute); + attribute = server.getAttribute(objectName, "splitSizeAvgTime"); + assertEquals(Long.valueOf(4), attribute); + } finally { + masterMetrics.shutdown(); + } + } + + @Test + public void testHBaseInfoBean() throws Exception { + MasterMetrics masterMetrics = new MasterMetrics("foo"); + try { + final MBeanServer server = ManagementFactory.getPlatformMBeanServer(); + // Test Info bean: + final ObjectName objectName2 = new ObjectName( + "hadoop:name=Info,service=HBase"); + Object attribute; + attribute = server.getAttribute(objectName2, "revision"); + assertNotNull(attribute); + attribute = server.getAttribute(objectName2, "version"); + assertNotNull(attribute); + attribute = server.getAttribute(objectName2, "hdfsUrl"); + assertNotNull(attribute); + attribute = server.getAttribute(objectName2, "user"); + assertNotNull(attribute); + } finally { + masterMetrics.shutdown(); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java b/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java new file mode 100644 index 000000000000..8bf4e96cbc6b --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java @@ -0,0 +1,329 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.snapshot; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.*; +import java.util.concurrent.atomic.AtomicInteger; + +import com.google.common.collect.Iterables; +import com.google.common.collect.ObjectArrays; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils; +import org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil; +import org.apache.hadoop.hbase.snapshot.TakeSnapshotUtils; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test that we correctly reload the cache, filter directories, etc. + */ +@Category(MediumTests.class) +public class TestSnapshotFileCache { + + private static final Log LOG = LogFactory.getLog(TestSnapshotFileCache.class); + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static FileSystem fs; + private static Path rootDir; + + @BeforeClass + public static void startCluster() throws Exception { + UTIL.startMiniDFSCluster(1); + fs = UTIL.getDFSCluster().getFileSystem(); + rootDir = UTIL.getDefaultRootDirPath(); + } + + @AfterClass + public static void stopCluster() throws Exception { + UTIL.shutdownMiniDFSCluster(); + } + + @After + public void cleanupFiles() throws Exception { + // cleanup the snapshot directory + Path snapshotDir = SnapshotDescriptionUtils.getSnapshotsDir(rootDir); + fs.delete(snapshotDir, true); + } + + @Test(timeout = 10000000) + public void testLoadAndDelete() throws Exception { + // don't refresh the cache unless we tell it to + long period = Long.MAX_VALUE; + Path snapshotDir = SnapshotDescriptionUtils.getSnapshotsDir(rootDir); + SnapshotFileCache cache = new SnapshotFileCache(fs, rootDir, period, 10000000, + "test-snapshot-file-cache-refresh", new SnapshotFiles()); + + Path snapshot = new Path(snapshotDir, "snapshot"); + Path region = new Path(snapshot, "7e91021"); + Path family = new Path(region, "fam"); + Path file1 = new Path(family, "file1"); + Path file2 = new Path(family, "file2"); + + + // create two hfiles under the snapshot + fs.createNewFile(file1); + fs.createNewFile(file2); + + FSUtils.logFileSystemState(fs, rootDir, LOG); + + // then make sure the cache finds them + Iterable nonSnapshotFiles = cache.getUnreferencedFiles( + Arrays.asList(FSUtils.listStatus(fs, family)) + ); + assertFalse("Cache didn't find:" + file1, Iterables.contains(nonSnapshotFiles, file1)); + assertFalse("Cache didn't find:" + file2, Iterables.contains(nonSnapshotFiles, file2)); + String not = "file-shouldn't-be-found"; + assertFalse("Cache found '" + not + "', but it shouldn't have.", Iterables.contains(nonSnapshotFiles, not)); + + // make sure we get a little bit of separation in the modification times + // its okay if we sleep a little longer (b/c of GC pause), as long as we sleep a little + Thread.sleep(10); + + LOG.debug("Deleting snapshot."); + // then delete the snapshot and make sure that we can still find the files + if (!fs.delete(snapshot, true)) { + throw new IOException("Couldn't delete " + snapshot + " for an unknown reason."); + } + FSUtils.logFileSystemState(fs, rootDir, LOG); + + + LOG.debug("Checking to see if file is deleted."); + nonSnapshotFiles = cache.getUnreferencedFiles( + nonSnapshotFiles + ); + + assertFalse("Cache didn't find:" + file1, Iterables.contains(nonSnapshotFiles, file1)); + assertFalse("Cache didn't find:" + file2, Iterables.contains(nonSnapshotFiles, file2)); + + // then trigger a refresh + cache.triggerCacheRefreshForTesting(); + + nonSnapshotFiles = cache.getUnreferencedFiles( + nonSnapshotFiles + ); + // and not it shouldn't find those files + assertFalse("Cache found '" + file1 + "', but it shouldn't have.", + Iterables.contains(nonSnapshotFiles, file1)); + assertFalse("Cache found '" + file2 + "', but it shouldn't have.", + Iterables.contains(nonSnapshotFiles, file2)); + + fs.delete(snapshotDir, true); + } + + @Test + public void testWeNeverCacheTmpDirAndLoadIt() throws Exception { + + final AtomicInteger count = new AtomicInteger(0); + // don't refresh the cache unless we tell it to + long period = Long.MAX_VALUE; + Path snapshotDir = SnapshotDescriptionUtils.getSnapshotsDir(rootDir); + SnapshotFileCache cache = new SnapshotFileCache(fs, rootDir, period, 10000000, + "test-snapshot-file-cache-refresh", new SnapshotFiles()) { + @Override + List getSnapshotsInProgress() throws IOException { + List result = super.getSnapshotsInProgress(); + count.incrementAndGet(); + return result; + } + }; + + // create a file in a 'completed' snapshot + Path snapshot = new Path(snapshotDir, "snapshot"); + Path region = new Path(snapshot, "7e91021"); + Path family = new Path(region, "fam"); + Path file1 = new Path(family, "file1"); + fs.createNewFile(file1); + + FileStatus[] completedFiles = FSUtils.listStatus(fs, family); + + // create an 'in progress' snapshot + SnapshotDescription desc = SnapshotDescription.newBuilder().setName("working").build(); + snapshot = SnapshotDescriptionUtils.getWorkingSnapshotDir(desc, rootDir); + region = new Path(snapshot, "7e91021"); + family = new Path(region, "fam"); + Path file2 = new Path(family, "file2"); + fs.createNewFile(file2); + cache.triggerCacheRefreshForTesting(); + + Iterable deletableFiles = cache.getUnreferencedFiles(Arrays.asList( + ObjectArrays.concat(completedFiles, FSUtils.listStatus(fs, family), FileStatus.class)) + ); + assertTrue(Iterables.isEmpty(deletableFiles)); + assertEquals(1, count.get()); // we check the tmp directory + + Path file3 = new Path(family, "file3"); + fs.create(file3); + deletableFiles = cache.getUnreferencedFiles(Arrays.asList( + ObjectArrays.concat(completedFiles, FSUtils.listStatus(fs, family), FileStatus.class)) + ); + assertTrue(Iterables.isEmpty(deletableFiles)); + assertEquals(2, count.get()); // we check the tmp directory + + } + + @Test + public void testLoadsTmpDir() throws Exception { + // don't refresh the cache unless we tell it to + long period = Long.MAX_VALUE; + Path snapshotDir = SnapshotDescriptionUtils.getSnapshotsDir(rootDir); + SnapshotFileCache cache = new SnapshotFileCache(fs, rootDir, period, 10000000, + "test-snapshot-file-cache-refresh", new SnapshotFiles()); + + // create a file in a 'completed' snapshot + Path snapshot = new Path(snapshotDir, "snapshot"); + Path region = new Path(snapshot, "7e91021"); + Path family = new Path(region, "fam"); + Path file1 = new Path(family, "file1"); + fs.createNewFile(file1); + + // create an 'in progress' snapshot + SnapshotDescription desc = SnapshotDescription.newBuilder().setName("working").build(); + snapshot = SnapshotDescriptionUtils.getWorkingSnapshotDir(desc, rootDir); + region = new Path(snapshot, "7e91021"); + family = new Path(region, "fam"); + Path file2 = new Path(family, "file2"); + fs.createNewFile(file2); + + FSUtils.logFileSystemState(fs, rootDir, LOG); + + // then make sure the cache finds both files + Iterable nonSnapshotFiles = cache.getUnreferencedFiles( + Arrays.asList(FSUtils.listStatus(fs, family)) + ); + assertFalse("Cache didn't find:" + file1, Iterables.contains(nonSnapshotFiles, file1)); + assertFalse("Cache didn't find:" + file2, Iterables.contains(nonSnapshotFiles, file2)); + } + + @Test + public void testJustFindLogsDirectory() throws Exception { + // don't refresh the cache unless we tell it to + long period = Long.MAX_VALUE; + Path snapshotDir = SnapshotDescriptionUtils.getSnapshotsDir(rootDir); + SnapshotFileCache cache = new SnapshotFileCache(fs, rootDir, period, 10000000, + "test-snapshot-file-cache-refresh", new SnapshotFileCache.SnapshotFileInspector() { + public Collection filesUnderSnapshot(final Path snapshotDir) + throws IOException { + return SnapshotReferenceUtil.getHLogNames(fs, snapshotDir); + } + }); + + // create a file in a 'completed' snapshot + Path snapshot = new Path(snapshotDir, "snapshot"); + Path region = new Path(snapshot, "7e91021"); + Path family = new Path(region, "fam"); + Path file1 = new Path(family, "file1"); + fs.createNewFile(file1); + + // and another file in the logs directory + Path logs = TakeSnapshotUtils.getSnapshotHLogsDir(snapshot, "server"); + Path log = new Path(logs, "me.hbase.com%2C58939%2C1350424310315.1350424315552"); + fs.createNewFile(log); + + FSUtils.logFileSystemState(fs, rootDir, LOG); + + Iterable nonSnapshotFiles = cache.getUnreferencedFiles( + Arrays.asList(FSUtils.listStatus(fs, family)) + ); + // then make sure the cache only finds the log files + assertFalse("Cache found '" + file1 + "', but it shouldn't have.", + Iterables.contains(nonSnapshotFiles, file1)); + assertFalse("Cache didn't find:" + log, Iterables.contains(nonSnapshotFiles, log)); + } + + @Test + public void testReloadModifiedDirectory() throws IOException { + // don't refresh the cache unless we tell it to + long period = Long.MAX_VALUE; + Path snapshotDir = SnapshotDescriptionUtils.getSnapshotsDir(rootDir); + SnapshotFileCache cache = new SnapshotFileCache(fs, rootDir, period, 10000000, + "test-snapshot-file-cache-refresh", new SnapshotFiles()); + + Path snapshot = new Path(snapshotDir, "snapshot"); + Path region = new Path(snapshot, "7e91021"); + Path family = new Path(region, "fam"); + Path file1 = new Path(family, "file1"); + Path file2 = new Path(family, "file2"); + + // create two hfiles under the snapshot + fs.createNewFile(file1); + fs.createNewFile(file2); + + FSUtils.logFileSystemState(fs, rootDir, LOG); + + Iterable nonSnapshotFiles = cache.getUnreferencedFiles( + Arrays.asList(FSUtils.listStatus(fs, family)) + ); + assertFalse("Cache didn't find " + file1, Iterables.contains(nonSnapshotFiles, file1)); + + // now delete the snapshot and add a file with a different name + fs.delete(snapshot, true); + Path file3 = new Path(family, "new_file"); + fs.createNewFile(file3); + + FSUtils.logFileSystemState(fs, rootDir, LOG); + nonSnapshotFiles = cache.getUnreferencedFiles( + Arrays.asList(FSUtils.listStatus(fs, family)) + ); + assertFalse("Cache didn't find new file:" + file3, Iterables.contains(nonSnapshotFiles, file3)); + } + + @Test + public void testSnapshotTempDirReload() throws IOException { + long period = Long.MAX_VALUE; + Path snapshotDir = new Path(SnapshotDescriptionUtils.getSnapshotsDir(rootDir), + SnapshotDescriptionUtils.SNAPSHOT_TMP_DIR_NAME); + SnapshotFileCache cache = new SnapshotFileCache(fs, rootDir, period, 10000000, + "test-snapshot-file-cache-refresh", new SnapshotFiles()); + + // Add a new snapshot + Path snapshot1 = new Path(snapshotDir, "snapshot1"); + Path file1 = new Path(new Path(new Path(snapshot1, "7e91021"), "fam"), "file1"); + fs.createNewFile(file1); + assertTrue(cache.getSnapshotsInProgress().contains(file1.getName())); + + // Add another snapshot + Path snapshot2 = new Path(snapshotDir, "snapshot2"); + Path file2 = new Path(new Path(new Path(snapshot2, "7e91021"), "fam2"), "file2"); + fs.createNewFile(file2); + assertTrue(cache.getSnapshotsInProgress().contains((file2.getName()))); + } + + class SnapshotFiles implements SnapshotFileCache.SnapshotFileInspector { + public Collection filesUnderSnapshot(final Path snapshotDir) throws IOException { + Collection files = new HashSet(); + files.addAll(SnapshotReferenceUtil.getHLogNames(fs, snapshotDir)); + files.addAll(SnapshotReferenceUtil.getHFileNames(fs, snapshotDir)); + return files; + } + }; +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotHFileCleaner.java b/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotHFileCleaner.java new file mode 100644 index 000000000000..0b6f1cd6ca7f --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotHFileCleaner.java @@ -0,0 +1,89 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.snapshot; + +import static org.junit.Assert.assertFalse; + +import java.io.IOException; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.AfterClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test that the snapshot hfile cleaner finds hfiles referenced in a snapshot + */ +@Category(SmallTests.class) +public class TestSnapshotHFileCleaner { + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + @AfterClass + public static void cleanup() throws IOException { + Configuration conf = TEST_UTIL.getConfiguration(); + Path rootDir = FSUtils.getRootDir(conf); + FileSystem fs = FileSystem.get(conf); + // cleanup + fs.delete(rootDir, true); + } + + @Test + public void testFindsSnapshotFilesWhenCleaning() throws IOException { + Configuration conf = TEST_UTIL.getConfiguration(); + FSUtils.setRootDir(conf, TEST_UTIL.getDataTestDir()); + Path rootDir = FSUtils.getRootDir(conf); + Path archivedHfileDir = new Path(TEST_UTIL.getDataTestDir(), HConstants.HFILE_ARCHIVE_DIRECTORY); + + FileSystem fs = FileSystem.get(conf); + SnapshotHFileCleaner cleaner = new SnapshotHFileCleaner(); + cleaner.setConf(conf); + + // write an hfile to the snapshot directory + String snapshotName = "snapshot"; + byte[] snapshot = Bytes.toBytes(snapshotName); + String table = "table"; + byte[] tableName = Bytes.toBytes(table); + Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir); + HRegionInfo mockRegion = new HRegionInfo(tableName); + Path regionSnapshotDir = new Path(snapshotDir, mockRegion.getEncodedName()); + Path familyDir = new Path(regionSnapshotDir, "family"); + // create a reference to a supposedly valid hfile + String hfile = "fd1e73e8a96c486090c5cec07b4894c4"; + Path refFile = new Path(familyDir, hfile); + + // make sure the reference file exists + fs.create(refFile); + + // create the hfile in the archive + fs.mkdirs(archivedHfileDir); + fs.createNewFile(new Path(archivedHfileDir, hfile)); + + // make sure that the file isn't deletable + assertFalse(cleaner.isFileDeletable(fs.getFileStatus(refFile))); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotLogCleaner.java b/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotLogCleaner.java new file mode 100644 index 000000000000..2c2a5aa0eb67 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotLogCleaner.java @@ -0,0 +1,85 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.snapshot; + +import static org.junit.Assert.assertFalse; + +import java.io.IOException; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.AfterClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test that the snapshot log cleaner finds logs referenced in a snapshot + */ +@Category(SmallTests.class) +public class TestSnapshotLogCleaner { + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + @AfterClass + public static void cleanup() throws IOException { + Configuration conf = TEST_UTIL.getConfiguration(); + Path rootDir = FSUtils.getRootDir(conf); + FileSystem fs = FileSystem.get(conf); + // cleanup + fs.delete(rootDir, true); + } + + @Test + public void testFindsSnapshotFilesWhenCleaning() throws IOException { + Configuration conf = TEST_UTIL.getConfiguration(); + FSUtils.setRootDir(conf, TEST_UTIL.getDataTestDir()); + Path rootDir = FSUtils.getRootDir(conf); + FileSystem fs = FileSystem.get(conf); + SnapshotLogCleaner cleaner = new SnapshotLogCleaner(); + cleaner.setConf(conf); + + // write an hfile to the snapshot directory + String snapshotName = "snapshot"; + byte[] snapshot = Bytes.toBytes(snapshotName); + Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir); + Path snapshotLogDir = new Path(snapshotDir, HConstants.HREGION_LOGDIR_NAME); + String timestamp = "1339643343027"; + String hostFromMaster = "localhost%2C59648%2C1339643336601"; + + Path hostSnapshotLogDir = new Path(snapshotLogDir, hostFromMaster); + String snapshotlogfile = hostFromMaster + "." + timestamp + ".hbase"; + + // add the reference to log in the snapshot + fs.create(new Path(hostSnapshotLogDir, snapshotlogfile)); + + // now check to see if that log file would get deleted. + Path oldlogDir = new Path(rootDir, ".oldlogs"); + Path logFile = new Path(oldlogDir, snapshotlogfile); + fs.create(logFile); + + // make sure that the file isn't deletable + assertFalse(cleaner.isFileDeletable(fs.getFileStatus(logFile))); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotManager.java b/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotManager.java new file mode 100644 index 000000000000..47c57bf3b63e --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotManager.java @@ -0,0 +1,162 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master.snapshot; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.IOException; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.executor.ExecutorService; +import org.apache.hadoop.hbase.master.MasterFileSystem; +import org.apache.hadoop.hbase.master.MasterServices; +import org.apache.hadoop.hbase.master.cleaner.HFileCleaner; +import org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner; +import org.apache.hadoop.hbase.master.metrics.MasterMetrics; +import org.apache.hadoop.hbase.procedure.ProcedureCoordinator; +import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils; +import org.apache.zookeeper.KeeperException; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +/** + * Test basic snapshot manager functionality + */ +@Category(SmallTests.class) +public class TestSnapshotManager { + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + MasterServices services = Mockito.mock(MasterServices.class); + MasterMetrics metrics = Mockito.mock(MasterMetrics.class); + ProcedureCoordinator coordinator = Mockito.mock(ProcedureCoordinator.class); + ExecutorService pool = Mockito.mock(ExecutorService.class); + MasterFileSystem mfs = Mockito.mock(MasterFileSystem.class); + FileSystem fs; + { + try { + fs = UTIL.getTestFileSystem(); + } catch (IOException e) { + throw new RuntimeException("Couldn't get test filesystem", e); + } + } + + private SnapshotManager getNewManager() throws IOException, KeeperException { + return getNewManager(UTIL.getConfiguration()); + } + + private SnapshotManager getNewManager(final Configuration conf) + throws IOException, KeeperException { + Mockito.reset(services); + Mockito.when(services.getConfiguration()).thenReturn(conf); + Mockito.when(services.getMasterFileSystem()).thenReturn(mfs); + Mockito.when(mfs.getFileSystem()).thenReturn(fs); + Mockito.when(mfs.getRootDir()).thenReturn(UTIL.getDataTestDir()); + return new SnapshotManager(services, metrics, coordinator, pool); + } + + @Test + public void testInProcess() throws KeeperException, IOException { + String tableName = "testTable"; + SnapshotManager manager = getNewManager(); + TakeSnapshotHandler handler = Mockito.mock(TakeSnapshotHandler.class); + assertFalse("Manager is in process when there is no current handler", + manager.isTakingSnapshot(tableName)); + manager.setSnapshotHandlerForTesting(tableName, handler); + Mockito.when(handler.isFinished()).thenReturn(false); + assertTrue("Manager isn't in process when handler is running", + manager.isTakingSnapshot(tableName)); + Mockito.when(handler.isFinished()).thenReturn(true); + assertFalse("Manager is process when handler isn't running", + manager.isTakingSnapshot(tableName)); + } + + /** + * Verify the snapshot support based on the configuration. + */ + @Test + public void testSnapshotSupportConfiguration() throws Exception { + // No configuration (no cleaners, not enabled): snapshot feature disabled + Configuration conf = new Configuration(); + SnapshotManager manager = getNewManager(conf); + assertFalse("Snapshot should be disabled with no configuration", isSnapshotSupported(manager)); + + // force snapshot feature to be enabled + conf = new Configuration(); + conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + manager = getNewManager(conf); + assertTrue("Snapshot should be enabled", isSnapshotSupported(manager)); + + // force snapshot feature to be disabled + conf = new Configuration(); + conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, false); + manager = getNewManager(conf); + assertFalse("Snapshot should be disabled", isSnapshotSupported(manager)); + + // force snapshot feature to be disabled, even if cleaners are present + conf = new Configuration(); + conf.setStrings(HFileCleaner.MASTER_HFILE_CLEANER_PLUGINS, + SnapshotHFileCleaner.class.getName(), HFileLinkCleaner.class.getName()); + conf.set(HConstants.HBASE_MASTER_LOGCLEANER_PLUGINS, SnapshotLogCleaner.class.getName()); + conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, false); + manager = getNewManager(conf); + assertFalse("Snapshot should be disabled", isSnapshotSupported(manager)); + + // cleaners are present, but missing snapshot enabled property + conf = new Configuration(); + conf.setStrings(HFileCleaner.MASTER_HFILE_CLEANER_PLUGINS, + SnapshotHFileCleaner.class.getName(), HFileLinkCleaner.class.getName()); + conf.set(HConstants.HBASE_MASTER_LOGCLEANER_PLUGINS, SnapshotLogCleaner.class.getName()); + manager = getNewManager(conf); + assertTrue("Snapshot should be enabled, because cleaners are present", + isSnapshotSupported(manager)); + + // Create a "test snapshot" + Path rootDir = UTIL.getDataTestDir(); + Path testSnapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir( + "testSnapshotSupportConfiguration", rootDir); + fs.mkdirs(testSnapshotDir); + try { + // force snapshot feature to be disabled, but snapshots are present + conf = new Configuration(); + conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, false); + manager = getNewManager(conf); + fail("Master should not start when snapshot is disabled, but snapshots are present"); + } catch (UnsupportedOperationException e) { + // expected + } finally { + fs.delete(testSnapshotDir, true); + } + } + + private boolean isSnapshotSupported(final SnapshotManager manager) { + try { + manager.checkSnapshotSupport(); + return true; + } catch (UnsupportedOperationException e) { + return false; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java b/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java new file mode 100644 index 000000000000..a0ba248289c3 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java @@ -0,0 +1,50 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.metrics; + +import java.util.List; + +import junit.framework.Assert; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.util.Pair; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(SmallTests.class) +public class TestExactCounterMetric { + + @Test + public void testBasic() { + final ExactCounterMetric counter = new ExactCounterMetric("testCounter", null); + for (int i = 1; i <= 10; i++) { + for (int j = 0; j < i; j++) { + counter.update(i + ""); + } + } + + List> topFive = counter.getTop(5); + Long i = 10L; + for (Pair entry : topFive) { + Assert.assertEquals(i + "", entry.getFirst()); + Assert.assertEquals(i, entry.getSecond()); + i--; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java b/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java new file mode 100644 index 000000000000..9d09486ee118 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java @@ -0,0 +1,68 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.metrics; + +import junit.framework.Assert; + +import com.yammer.metrics.stats.ExponentiallyDecayingSample; +import com.yammer.metrics.stats.Snapshot; + +import org.apache.hadoop.hbase.SmallTests; + +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(SmallTests.class) +public class TestExponentiallyDecayingSample { + + @Test + public void testBasic() { + final ExponentiallyDecayingSample sample = + new ExponentiallyDecayingSample(100, 0.99); + + for (int i = 0; i < 1000; i++) { + sample.update(i); + } + Assert.assertEquals(100, sample.size()); + + final Snapshot snapshot = sample.getSnapshot(); + Assert.assertEquals(100, snapshot.size()); + + for (double i : snapshot.getValues()) { + Assert.assertTrue(i >= 0.0 && i < 1000.0); + } + } + + @Test + public void testTooBig() throws Exception { + final ExponentiallyDecayingSample sample = + new ExponentiallyDecayingSample(100, 0.99); + for (int i = 0; i < 10; i++) { + sample.update(i); + } + Assert.assertEquals(10, sample.size()); + + final Snapshot snapshot = sample.getSnapshot(); + Assert.assertEquals(10, sample.size()); + + for (double i : snapshot.getValues()) { + Assert.assertTrue(i >= 0.0 && i < 1000.0); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java b/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java new file mode 100644 index 000000000000..0d853fa487cd --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java @@ -0,0 +1,112 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.metrics; + +import static org.mockito.Matchers.anyFloat; +import static org.mockito.Matchers.anyLong; +import static org.mockito.Matchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; + +import java.util.Random; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram; +import org.apache.hadoop.metrics.MetricsRecord; +import org.junit.Assert; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import com.yammer.metrics.stats.Snapshot; + +@SuppressWarnings("deprecation") +@Category(SmallTests.class) +public class TestMetricsHistogram { + + @Test + public void testBasicUniform() { + MetricsHistogram h = new MetricsHistogram("testHistogram", null); + + for (int i = 0; i < 100; i++) { + h.update(i); + } + + Assert.assertEquals(100, h.getCount()); + Assert.assertEquals(0, h.getMin()); + Assert.assertEquals(99, h.getMax()); + Assert.assertEquals(49.5d, h.getMean(), 0.01); + } + + @Test + public void testSnapshotPercentiles() { + final MetricsHistogram h = new MetricsHistogram("testHistogram", null); + final long[] data = genRandomData(h); + + final Snapshot s = h.getSnapshot(); + + assertPercentile(data, 50, s.getMedian()); + assertPercentile(data, 75, s.get75thPercentile()); + assertPercentile(data, 95, s.get95thPercentile()); + assertPercentile(data, 98, s.get98thPercentile()); + assertPercentile(data, 99, s.get99thPercentile()); + assertPercentile(data, 99.9, s.get999thPercentile()); + } + + @Test + public void testPushMetric() { + final MetricsHistogram h = new MetricsHistogram("testHistogram", null); + genRandomData(h); + + MetricsRecord mr = mock(MetricsRecord.class); + h.pushMetric(mr); + + verify(mr).setMetric("testHistogram_num_ops", 10000L); + verify(mr).setMetric(eq("testHistogram_min"), anyLong()); + verify(mr).setMetric(eq("testHistogram_max"), anyLong()); + verify(mr).setMetric(eq("testHistogram_mean"), anyFloat()); + verify(mr).setMetric(eq("testHistogram_std_dev"), anyFloat()); + verify(mr).setMetric(eq("testHistogram_median"), anyFloat()); + verify(mr).setMetric(eq("testHistogram_75th_percentile"), anyFloat()); + verify(mr).setMetric(eq("testHistogram_95th_percentile"), anyFloat()); + verify(mr).setMetric(eq("testHistogram_99th_percentile"), anyFloat()); + } + + private void assertPercentile(long[] data, double percentile, double value) { + int count = 0; + for (long v : data) { + if (v < value) { + count++; + } + } + Assert.assertEquals("Wrong " + percentile + " percentile", + (int)(percentile / 100), count / data.length); + } + + private long[] genRandomData(final MetricsHistogram h) { + final Random r = new Random(); + final long[] data = new long[10000]; + + for (int i = 0; i < data.length; i++) { + data[i] = (long) (r.nextGaussian() * 10000); + h.update(data[i]); + } + + return data; + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsMBeanBase.java b/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsMBeanBase.java index a6c771e9fd83..d040e2a2535f 100644 --- a/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsMBeanBase.java +++ b/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsMBeanBase.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -24,8 +23,13 @@ import java.util.List; import java.util.Map; +import javax.management.AttributeNotFoundException; import javax.management.MBeanAttributeInfo; +import javax.management.MBeanException; import javax.management.MBeanInfo; +import javax.management.ReflectionException; +import org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram; +import com.yammer.metrics.stats.Snapshot; import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.metrics.MetricsContext; @@ -35,6 +39,9 @@ import org.apache.hadoop.metrics.util.MetricsRegistry; import org.apache.hadoop.metrics.util.MetricsTimeVaryingRate; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + import junit.framework.TestCase; import org.junit.experimental.categories.Category; @@ -114,6 +121,52 @@ public void testGetMBeanInfo() { "varyRateNumOps", "java.lang.Integer", "test"); } + public void testMetricsMBeanBaseHistogram() + throws ReflectionException, AttributeNotFoundException, MBeanException { + MetricsRegistry mr = new MetricsRegistry(); + MetricsHistogram histo = mock(MetricsHistogram.class); + Snapshot snap = mock(Snapshot.class); + + //Set up the mocks + String histoName = "MockHisto"; + when(histo.getName()).thenReturn(histoName); + when(histo.getCount()).thenReturn(20l); + when(histo.getMin()).thenReturn(1l); + when(histo.getMax()).thenReturn(999l); + when(histo.getMean()).thenReturn(500.2); + when(histo.getStdDev()).thenReturn(1.2); + when(histo.getSnapshot()).thenReturn(snap); + + when(snap.getMedian()).thenReturn(490.0); + when(snap.get75thPercentile()).thenReturn(550.0); + when(snap.get95thPercentile()).thenReturn(900.0); + when(snap.get99thPercentile()).thenReturn(990.0); + + mr.add("myTestHisto", histo); + + MetricsMBeanBase mBeanBase = new MetricsMBeanBase(mr, "test"); + + assertEquals(new Long(20), mBeanBase + .getAttribute(histoName + MetricsHistogram.NUM_OPS_METRIC_NAME)); + assertEquals(new Long(1), mBeanBase + .getAttribute(histoName + MetricsHistogram.MIN_METRIC_NAME)); + assertEquals(new Long(999), mBeanBase + .getAttribute(histoName + MetricsHistogram.MAX_METRIC_NAME)); + assertEquals(new Float(500.2), mBeanBase + .getAttribute(histoName + MetricsHistogram.MEAN_METRIC_NAME)); + assertEquals(new Float(1.2), mBeanBase + .getAttribute(histoName + MetricsHistogram.STD_DEV_METRIC_NAME)); + + assertEquals(new Float(490.0), mBeanBase + .getAttribute(histoName + MetricsHistogram.MEDIAN_METRIC_NAME)); + assertEquals(new Float(550.0), mBeanBase + .getAttribute(histoName + MetricsHistogram.SEVENTY_FIFTH_PERCENTILE_METRIC_NAME)); + assertEquals(new Float(900.0), mBeanBase + .getAttribute(histoName + MetricsHistogram.NINETY_FIFTH_PERCENTILE_METRIC_NAME)); + assertEquals(new Float(990.0), mBeanBase + .getAttribute(histoName + MetricsHistogram.NINETY_NINETH_PERCENTILE_METRIC_NAME)); + } + protected void assertAttribute(MBeanAttributeInfo attr, String name, String type, String description) { diff --git a/src/test/java/org/apache/hadoop/hbase/monitoring/TestMemoryBoundedLogMessageBuffer.java b/src/test/java/org/apache/hadoop/hbase/monitoring/TestMemoryBoundedLogMessageBuffer.java index ac76887569c6..9c46307629a9 100644 --- a/src/test/java/org/apache/hadoop/hbase/monitoring/TestMemoryBoundedLogMessageBuffer.java +++ b/src/test/java/org/apache/hadoop/hbase/monitoring/TestMemoryBoundedLogMessageBuffer.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java b/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java index 98de0247b998..3b35c2be1eaa 100644 --- a/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java +++ b/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java b/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java new file mode 100644 index 000000000000..a69dbbb35391 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java @@ -0,0 +1,518 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.mttr; + +import com.google.common.base.Objects; +import org.apache.commons.lang.RandomStringUtils; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.commons.math.stat.descriptive.DescriptiveStatistics; +import org.apache.hadoop.hbase.ClusterStatus; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.IntegrationTestingUtility; +import org.apache.hadoop.hbase.IntegrationTests; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.filter.KeyOnlyFilter; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.ChaosMonkey; +import org.apache.hadoop.hbase.util.LoadTestTool; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +import static junit.framework.Assert.assertEquals; + +/** + * Integration test that should benchmark how fast HBase can recover from failures. This test starts + * different threads: + *
    + *
  1. + * Load Test Tool.
    + * This runs so that all RegionServers will have some load and HLogs will be full. + *
    2. + *
  2. + * Scan thread.
    + * This thread runs a very short scan over and over again recording how log it takes to respond. + * The longest response is assumed to be the time it took to recover. + *
    3. + *
  3. + * Put thread.
    + * This thread just like the scan thread except it does a very small put. + *
    4. + *
  4. + * Admin thread.
    + * This thread will continually go to the master to try and get the cluster status. Just like the + * put and scan threads, the time to respond is recorded. + *
    5. + *
  5. + * Chaos Monkey thread.
    + * This thread runs a ChaosMonkey.Action. + *
    6. + *
+ *

+ * The ChaosMonkey actions currently run are: + *

    + *
  • Restart the RegionServer holding meta.
    • + *
  • Restart the RegionServer holding the table the scan and put threads are targeting.
    • + *
  • Move the Regions of the table used by the scan and put threads.
    • + *
  • Restart the master.
    • + *
+ *

+ * At the end of the test a log line is output on the INFO level containing the timing data that was + * collected. + */ + +@Category(IntegrationTests.class) +public class IntegrationTestMTTR { + /** + * Constants. + */ + private static final byte[] FAMILY = Bytes.toBytes("d"); + private static final Log LOG = LogFactory.getLog(IntegrationTestMTTR.class); + private static final long SLEEP_TIME = 60 * 1000l; + + /** + * Configurable table names. + */ + private static String tableName; + private static byte[] tableNameBytes; + private static String loadTableName; + private static byte[] loadTableNameBytes; + + /** + * Util to get at the cluster. + */ + private static IntegrationTestingUtility util; + + /** + * Executor for test threads. + */ + private static ExecutorService executorService; + + /** + * All of the chaos monkey actions used. + */ + private static ChaosMonkey.Action restartRSAction; + private static ChaosMonkey.Action restartMetaAction; + private static ChaosMonkey.Action moveRegionAction; + private static ChaosMonkey.Action restartMasterAction; + + /** + * The load test tool used to create load and make sure that HLogs aren't empty. + */ + private static LoadTestTool loadTool; + + + @BeforeClass + public static void setUp() throws Exception { + // Set up the integration test util + if (util == null) { + util = new IntegrationTestingUtility(); + } + + // Make sure there are three servers. + util.initializeCluster(3); + + // Set up the load test tool. + loadTool = new LoadTestTool(); + loadTool.setConf(util.getConfiguration()); + + // Create executor with enough threads to restart rs's, + // run scans, puts, admin ops and load test tool. + executorService = Executors.newFixedThreadPool(8); + + // Set up the tables needed. + setupTables(); + + // Set up the actions. + setupActions(); + } + + private static void setupActions() throws IOException { + // Set up the action that will restart a region server holding a region from our table + // because this table should only have one region we should be good. + restartRSAction = new ChaosMonkey.RestartRsHoldingTable(SLEEP_TIME, tableName); + + // Set up the action that will kill the region holding meta. + restartMetaAction = new ChaosMonkey.RestartRsHoldingMeta(SLEEP_TIME); + + // Set up the action that will move the regions of our table. + moveRegionAction = new ChaosMonkey.MoveRegionsOfTable(SLEEP_TIME, tableName); + + // Kill the master + restartMasterAction = new ChaosMonkey.RestartActiveMaster(1000); + + // Give the action the access to the cluster. + ChaosMonkey.ActionContext actionContext = new ChaosMonkey.ActionContext(util); + restartRSAction.init(actionContext); + restartMetaAction.init(actionContext); + moveRegionAction.init(actionContext); + restartMasterAction.init(actionContext); + } + + private static void setupTables() throws IOException { + // Get the table name. + tableName = util.getConfiguration() + .get("hbase.IntegrationTestMTTR.tableName", "IntegrationTestMTTR"); + tableNameBytes = Bytes.toBytes(tableName); + + loadTableName = util.getConfiguration() + .get("hbase.IntegrationTestMTTR.loadTableName", "IntegrationTestMTTRLoadTestTool"); + loadTableNameBytes = Bytes.toBytes(loadTableName); + + if (util.getHBaseAdmin().tableExists(tableNameBytes)) { + util.deleteTable(tableNameBytes); + } + + if (util.getHBaseAdmin().tableExists(loadTableName)) { + util.deleteTable(loadTableNameBytes); + } + + // Create the table. If this fails then fail everything. + HTableDescriptor tableDescriptor = new HTableDescriptor(tableNameBytes); + + // Make the max file size huge so that splits don't happen during the test. + tableDescriptor.setMaxFileSize(Long.MAX_VALUE); + + HColumnDescriptor descriptor = new HColumnDescriptor(FAMILY); + descriptor.setMaxVersions(1); + tableDescriptor.addFamily(descriptor); + util.getHBaseAdmin().createTable(tableDescriptor); + + // Setup the table for LoadTestTool + int ret = loadTool.run(new String[]{"-tn", loadTableName, "-init_only"}); + assertEquals("Failed to initialize LoadTestTool", 0, ret); + } + + @AfterClass + public static void after() throws IOException { + // Clean everything up. + util.restoreCluster(); + util = null; + + // Stop the threads so that we know everything is complete. + executorService.shutdown(); + executorService = null; + + // Clean up the actions. + moveRegionAction = null; + restartMetaAction = null; + restartRSAction = null; + restartMasterAction = null; + + loadTool = null; + } + + @Test + public void testRestartRsHoldingTable() throws Exception { + run(new ActionCallable(restartRSAction), "RestartRsHoldingTable"); + } + + @Test + public void testKillRsHoldingMeta() throws Exception { + run(new ActionCallable(restartMetaAction), "KillRsHoldingMeta"); + } + + @Test + public void testMoveRegion() throws Exception { + run(new ActionCallable(moveRegionAction), "MoveRegion"); + } + + @Test + public void testRestartMaster() throws Exception { + run(new ActionCallable(restartMasterAction), "RestartMaster"); + } + + public void run(Callable monkeyCallable, String testName) throws Exception { + int maxIters = util.getHBaseClusterInterface().isDistributedCluster() ? 10 : 3; + + // Array to keep track of times. + ArrayList resultPuts = new ArrayList(maxIters); + ArrayList resultScan = new ArrayList(maxIters); + ArrayList resultAdmin = new ArrayList(maxIters); + long start = System.nanoTime(); + + // We're going to try this multiple times + for (int fullIterations = 0; fullIterations < maxIters; fullIterations++) { + // Create and start executing a callable that will kill the servers + Future monkeyFuture = executorService.submit(monkeyCallable); + + // Pass that future to the timing Callables. + Future putFuture = executorService.submit(new PutCallable(monkeyFuture)); + Future scanFuture = executorService.submit(new ScanCallable(monkeyFuture)); + Future adminFuture = executorService.submit(new AdminCallable(monkeyFuture)); + + Future loadFuture = executorService.submit(new LoadCallable(monkeyFuture)); + + monkeyFuture.get(); + loadFuture.get(); + + // Get the values from the futures. + TimingResult putTime = putFuture.get(); + TimingResult scanTime = scanFuture.get(); + TimingResult adminTime = adminFuture.get(); + + // Store the times to display later. + resultPuts.add(putTime); + resultScan.add(scanTime); + resultAdmin.add(adminTime); + + // Wait some time for everything to settle down. + Thread.sleep(5000l); + } + + long runtimeMs = TimeUnit.MILLISECONDS.convert(System.nanoTime() - start, TimeUnit.NANOSECONDS); + + Objects.ToStringHelper helper = Objects.toStringHelper("MTTRResults") + .add("putResults", resultPuts) + .add("scanResults", resultScan) + .add("adminResults", resultAdmin) + .add("totalRuntimeMs", runtimeMs) + .add("name", testName); + + // Log the info + LOG.info(helper.toString()); + } + + /** + * Class to store results of TimingCallable. + * + * Stores times and trace id. + */ + private class TimingResult { + DescriptiveStatistics stats = new DescriptiveStatistics(); + + /** + * Add a result to this aggregate result. + * @param time Time in nanoseconds + * @param span Span. To be kept if the time taken was over 1 second + */ + public void addResult(long time) { + stats.addValue(TimeUnit.MILLISECONDS.convert(time, TimeUnit.NANOSECONDS)); + } + + public String toString() { + Objects.ToStringHelper helper = Objects.toStringHelper(this) + .add("numResults", stats.getN()) + .add("minTime", stats.getMin()) + .add("meanTime", stats.getMean()) + .add("maxTime", stats.getMax()) + .add("25th", stats.getPercentile(25)) + .add("50th", stats.getPercentile(50)) + .add("75th", stats.getPercentile(75)) + .add("90th", stats.getPercentile(90)) + .add("95th", stats.getPercentile(95)) + .add("99th", stats.getPercentile(99)) + .add("99.9th", stats.getPercentile(99.9)) + .add("99.99th", stats.getPercentile(99.99)); + return helper.toString(); + } + } + + /** + * Base class for actions that need to record the time needed to recover from a failure. + */ + public abstract class TimingCallable implements Callable { + protected final Future future; + + public TimingCallable(Future f) { + future = f; + } + + @Override + public TimingResult call() throws Exception { + TimingResult result = new TimingResult(); + int numAfterDone = 0; + // Keep trying until the rs is back up and we've gotten a put through + while (numAfterDone < 10) { + long start = System.nanoTime(); + try { + boolean actionResult = doAction(); + if (actionResult && future.isDone()) { + numAfterDone ++; + } + } catch (Exception e) { + numAfterDone = 0; + } + result.addResult(System.nanoTime() - start); + } + return result; + } + + protected abstract boolean doAction() throws Exception; + + protected String getSpanName() { + return this.getClass().getSimpleName(); + } + } + + /** + * Callable that will keep putting small amounts of data into a table + * until the future supplied returns. It keeps track of the max time. + */ + public class PutCallable extends TimingCallable { + + private final HTable table; + + public PutCallable(Future f) throws IOException { + super(f); + this.table = new HTable(util.getConfiguration(), tableNameBytes); + } + + @Override + protected boolean doAction() throws Exception { + Put p = new Put(Bytes.toBytes(RandomStringUtils.randomAlphanumeric(5))); + p.add(FAMILY, Bytes.toBytes("\0"), Bytes.toBytes(RandomStringUtils.randomAscii(5))); + table.put(p); + table.flushCommits(); + return true; + } + + @Override + protected String getSpanName() { + return "MTTR Put Test"; + } + } + + /** + * Callable that will keep scanning for small amounts of data until the + * supplied future returns. Returns the max time taken to scan. + */ + public class ScanCallable extends TimingCallable { + private final HTable table; + + public ScanCallable(Future f) throws IOException { + super(f); + this.table = new HTable(util.getConfiguration(), tableNameBytes); + } + + @Override + protected boolean doAction() throws Exception { + ResultScanner rs = null; + try { + Scan s = new Scan(); + s.setBatch(2); + s.addFamily(FAMILY); + s.setFilter(new KeyOnlyFilter()); + s.setMaxVersions(1); + + rs = table.getScanner(s); + Result result = rs.next(); + return rs != null && result != null && result.size() > 0; + } finally { + if (rs != null) { + rs.close(); + } + } + } + @Override + protected String getSpanName() { + return "MTTR Scan Test"; + } + } + + /** + * Callable that will keep going to the master for cluster status. Returns the max time taken. + */ + public class AdminCallable extends TimingCallable { + + public AdminCallable(Future f) throws IOException { + super(f); + } + + @Override + protected boolean doAction() throws Exception { + HBaseAdmin admin = new HBaseAdmin(util.getConfiguration()); + ClusterStatus status = admin.getClusterStatus(); + return status != null; + } + + @Override + protected String getSpanName() { + return "MTTR Admin Test"; + } + } + + + public class ActionCallable implements Callable { + private final ChaosMonkey.Action action; + + public ActionCallable(ChaosMonkey.Action action) { + this.action = action; + } + + @Override + public Boolean call() throws Exception { + this.action.perform(); + return true; + } + } + + /** + * Callable used to make sure the cluster has some load on it. + * This callable uses LoadTest tool to + */ + public class LoadCallable implements Callable { + + private final Future future; + + public LoadCallable(Future f) { + future = f; + } + + @Override + public Boolean call() throws Exception { + int colsPerKey = 10; + int recordSize = 500; + int numServers = util.getHBaseClusterInterface().getInitialClusterStatus().getServersSize(); + int numKeys = numServers * 5000; + int writeThreads = 10; + + + // Loop until the chaos monkey future is done. + // But always go in just in case some action completes quickly + do { + int ret = loadTool.run(new String[]{ + "-tn", loadTableName, + "-write", String.format("%d:%d:%d", colsPerKey, recordSize, writeThreads), + "-num_keys", String.valueOf(numKeys), + "-skip_init" + }); + assertEquals("Load failed", 0, ret); + } while (!future.isDone()); + + return true; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedure.java b/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedure.java new file mode 100644 index 000000000000..4026f394da5c --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedure.java @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.procedure; + +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.errorhandling.ForeignException; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Demonstrate how Procedure handles single members, multiple members, and errors semantics + */ +@Category(SmallTests.class) +public class TestProcedure { + + ProcedureCoordinator coord; + + @Before + public void setup() { + coord = mock(ProcedureCoordinator.class); + final ProcedureCoordinatorRpcs comms = mock(ProcedureCoordinatorRpcs.class); + when(coord.getRpcs()).thenReturn(comms); // make it not null + } + + class LatchedProcedure extends Procedure { + CountDownLatch startedAcquireBarrier = new CountDownLatch(1); + CountDownLatch startedDuringBarrier = new CountDownLatch(1); + CountDownLatch completedProcedure = new CountDownLatch(1); + + public LatchedProcedure(ProcedureCoordinator coord, ForeignExceptionDispatcher monitor, + long wakeFreq, long timeout, String opName, byte[] data, + List expectedMembers) { + super(coord, monitor, wakeFreq, timeout, opName, data, expectedMembers); + } + + @Override + public void sendGlobalBarrierStart() { + startedAcquireBarrier.countDown(); + } + + @Override + public void sendGlobalBarrierReached() { + startedDuringBarrier.countDown(); + } + + @Override + public void sendGlobalBarrierComplete() { + completedProcedure.countDown(); + } + }; + + /** + * With a single member, verify ordered execution. The Coordinator side is run in a separate + * thread so we can only trigger from members and wait for particular state latches. + */ + @Test(timeout = 60000) + public void testSingleMember() throws Exception { + // The member + List members = new ArrayList(); + members.add("member"); + LatchedProcedure proc = new LatchedProcedure(coord, new ForeignExceptionDispatcher(), 100, + Integer.MAX_VALUE, "op", null, members); + final LatchedProcedure procspy = spy(proc); + // coordinator: start the barrier procedure + new Thread() { + public void run() { + procspy.call(); + } + }.start(); + + // coordinator: wait for the barrier to be acquired, then send start barrier + proc.startedAcquireBarrier.await(); + + // we only know that {@link Procedure#sendStartBarrier()} was called, and others are blocked. + verify(procspy).sendGlobalBarrierStart(); + verify(procspy, never()).sendGlobalBarrierReached(); + verify(procspy, never()).sendGlobalBarrierComplete(); + verify(procspy, never()).barrierAcquiredByMember(anyString()); + + // member: trigger global barrier acquisition + proc.barrierAcquiredByMember(members.get(0)); + + // coordinator: wait for global barrier to be acquired. + proc.acquiredBarrierLatch.await(); + verify(procspy).sendGlobalBarrierStart(); // old news + + // since two threads, we cannot guarantee that {@link Procedure#sendSatsifiedBarrier()} was + // or was not called here. + + // member: trigger global barrier release + proc.barrierReleasedByMember(members.get(0)); + + // coordinator: wait for procedure to be completed + proc.completedProcedure.await(); + verify(procspy).sendGlobalBarrierReached(); + verify(procspy).sendGlobalBarrierComplete(); + verify(procspy, never()).receive(any(ForeignException.class)); + } + + @Test(timeout=60000) + public void testMultipleMember() throws Exception { + // 2 members + List members = new ArrayList(); + members.add("member1"); + members.add("member2"); + + LatchedProcedure proc = new LatchedProcedure(coord, new ForeignExceptionDispatcher(), 100, + Integer.MAX_VALUE, "op", null, members); + final LatchedProcedure procspy = spy(proc); + // start the barrier procedure + new Thread() { + public void run() { + procspy.call(); + } + }.start(); + + // coordinator: wait for the barrier to be acquired, then send start barrier + procspy.startedAcquireBarrier.await(); + + // we only know that {@link Procedure#sendStartBarrier()} was called, and others are blocked. + verify(procspy).sendGlobalBarrierStart(); + verify(procspy, never()).sendGlobalBarrierReached(); + verify(procspy, never()).sendGlobalBarrierComplete(); + verify(procspy, never()).barrierAcquiredByMember(anyString()); // no externals + + // member0: [1/2] trigger global barrier acquisition. + procspy.barrierAcquiredByMember(members.get(0)); + + // coordinator not satisified. + verify(procspy).sendGlobalBarrierStart(); + verify(procspy, never()).sendGlobalBarrierReached(); + verify(procspy, never()).sendGlobalBarrierComplete(); + + // member 1: [2/2] trigger global barrier acquisition. + procspy.barrierAcquiredByMember(members.get(1)); + + // coordinator: wait for global barrier to be acquired. + procspy.startedDuringBarrier.await(); + verify(procspy).sendGlobalBarrierStart(); // old news + + // member 1, 2: trigger global barrier release + procspy.barrierReleasedByMember(members.get(0)); + procspy.barrierReleasedByMember(members.get(1)); + + // coordinator wait for procedure to be completed + procspy.completedProcedure.await(); + verify(procspy).sendGlobalBarrierReached(); + verify(procspy).sendGlobalBarrierComplete(); + verify(procspy, never()).receive(any(ForeignException.class)); + } + + @Test(timeout = 60000) + public void testErrorPropagation() throws Exception { + List members = new ArrayList(); + members.add("member"); + Procedure proc = new Procedure(coord, new ForeignExceptionDispatcher(), 100, + Integer.MAX_VALUE, "op", null, members); + final Procedure procspy = spy(proc); + + ForeignException cause = new ForeignException("SRC", "External Exception"); + proc.receive(cause); + + // start the barrier procedure + Thread t = new Thread() { + public void run() { + procspy.call(); + } + }; + t.start(); + t.join(); + + verify(procspy, never()).sendGlobalBarrierStart(); + verify(procspy, never()).sendGlobalBarrierReached(); + verify(procspy).sendGlobalBarrierComplete(); + } + + @Test(timeout = 60000) + public void testBarrieredErrorPropagation() throws Exception { + List members = new ArrayList(); + members.add("member"); + LatchedProcedure proc = new LatchedProcedure(coord, new ForeignExceptionDispatcher(), 100, + Integer.MAX_VALUE, "op", null, members); + final LatchedProcedure procspy = spy(proc); + + // start the barrier procedure + Thread t = new Thread() { + public void run() { + procspy.call(); + } + }; + t.start(); + + // now test that we can put an error in before the commit phase runs + procspy.startedAcquireBarrier.await(); + ForeignException cause = new ForeignException("SRC", "External Exception"); + procspy.receive(cause); + procspy.barrierAcquiredByMember(members.get(0)); + t.join(); + + // verify state of all the object + verify(procspy).sendGlobalBarrierStart(); + verify(procspy).sendGlobalBarrierComplete(); + verify(procspy, never()).sendGlobalBarrierReached(); + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureCoordinator.java b/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureCoordinator.java new file mode 100644 index 000000000000..d0b62b29bc29 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureCoordinator.java @@ -0,0 +1,349 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.procedure; + +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyListOf; +import static org.mockito.Matchers.anyString; +import static org.mockito.Matchers.eq; +import static org.mockito.Mockito.atLeastOnce; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.inOrder; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.ThreadPoolExecutor; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.errorhandling.ForeignException; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.junit.After; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.InOrder; +import org.mockito.invocation.InvocationOnMock; +import org.mockito.stubbing.Answer; + +import com.google.common.collect.Lists; + +/** + * Test Procedure coordinator operation. + *

+ * This only works correctly when we do class level parallelization of tests. If we do method + * level serialization this class will likely throw all kinds of errors. + */ +@Category(SmallTests.class) +public class TestProcedureCoordinator { + // general test constants + private static final long WAKE_FREQUENCY = 1000; + private static final long TIMEOUT = 100000; + private static final long POOL_KEEP_ALIVE = 1; + private static final String nodeName = "node"; + private static final String procName = "some op"; + private static final byte[] procData = new byte[0]; + private static final List expected = Lists.newArrayList("remote1", "remote2"); + + // setup the mocks + private final ProcedureCoordinatorRpcs controller = mock(ProcedureCoordinatorRpcs.class); + private final Procedure task = mock(Procedure.class); + private final ForeignExceptionDispatcher monitor = mock(ForeignExceptionDispatcher.class); + + // handle to the coordinator for each test + private ProcedureCoordinator coordinator; + + @After + public void resetTest() throws IOException { + // reset all the mocks used for the tests + reset(controller, task, monitor); + // close the open coordinator, if it was used + if (coordinator != null) coordinator.close(); + } + + private ProcedureCoordinator buildNewCoordinator() { + ThreadPoolExecutor pool = ProcedureCoordinator.defaultPool(nodeName, 1, POOL_KEEP_ALIVE); + return spy(new ProcedureCoordinator(controller, pool)); + } + + /** + * Currently we can only handle one procedure at a time. This makes sure we handle that and + * reject submitting more. + */ + @Test + public void testThreadPoolSize() throws Exception { + ProcedureCoordinator coordinator = buildNewCoordinator(); + Procedure proc = new Procedure(coordinator, monitor, + WAKE_FREQUENCY, TIMEOUT, procName, procData, expected); + Procedure procSpy = spy(proc); + + Procedure proc2 = new Procedure(coordinator, monitor, + WAKE_FREQUENCY, TIMEOUT, procName +"2", procData, expected); + Procedure procSpy2 = spy(proc2); + when(coordinator.createProcedure(any(ForeignExceptionDispatcher.class), eq(procName), eq(procData), anyListOf(String.class))) + .thenReturn(procSpy, procSpy2); + + coordinator.startProcedure(procSpy.getErrorMonitor(), procName, procData, expected); + // null here means second procedure failed to start. + assertNull("Coordinator successfully ran two tasks at once with a single thread pool.", + coordinator.startProcedure(proc2.getErrorMonitor(), "another op", procData, expected)); + } + + /** + * Check handling a connection failure correctly if we get it during the acquiring phase + */ + @Test(timeout = 60000) + public void testUnreachableControllerDuringPrepare() throws Exception { + coordinator = buildNewCoordinator(); + // setup the proc + List expected = Arrays.asList("cohort"); + Procedure proc = new Procedure(coordinator, WAKE_FREQUENCY, + TIMEOUT, procName, procData, expected); + final Procedure procSpy = spy(proc); + + when(coordinator.createProcedure(any(ForeignExceptionDispatcher.class), eq(procName), eq(procData), anyListOf(String.class))) + .thenReturn(procSpy); + + // use the passed controller responses + IOException cause = new IOException("Failed to reach comms during acquire"); + doThrow(cause).when(controller) + .sendGlobalBarrierAcquire(eq(procSpy), eq(procData), anyListOf(String.class)); + + // run the operation + proc = coordinator.startProcedure(proc.getErrorMonitor(), procName, procData, expected); + // and wait for it to finish + proc.waitForCompleted(); + verify(procSpy, atLeastOnce()).receive(any(ForeignException.class)); + verify(coordinator, times(1)).rpcConnectionFailure(anyString(), eq(cause)); + verify(controller, times(1)).sendGlobalBarrierAcquire(procSpy, procData, expected); + verify(controller, never()).sendGlobalBarrierReached(any(Procedure.class), + anyListOf(String.class)); + } + + /** + * Check handling a connection failure correctly if we get it during the barrier phase + */ + @Test(timeout = 60000) + public void testUnreachableControllerDuringCommit() throws Exception { + coordinator = buildNewCoordinator(); + + // setup the task and spy on it + List expected = Arrays.asList("cohort"); + final Procedure spy = spy(new Procedure(coordinator, + WAKE_FREQUENCY, TIMEOUT, procName, procData, expected)); + + when(coordinator.createProcedure(any(ForeignExceptionDispatcher.class), eq(procName), eq(procData), anyListOf(String.class))) + .thenReturn(spy); + + // use the passed controller responses + IOException cause = new IOException("Failed to reach controller during prepare"); + doAnswer(new AcquireBarrierAnswer(procName, new String[] { "cohort" })) + .when(controller).sendGlobalBarrierAcquire(eq(spy), eq(procData), anyListOf(String.class)); + doThrow(cause).when(controller).sendGlobalBarrierReached(eq(spy), anyListOf(String.class)); + + // run the operation + Procedure task = coordinator.startProcedure(spy.getErrorMonitor(), procName, procData, expected); + // and wait for it to finish + task.waitForCompleted(); + verify(spy, atLeastOnce()).receive(any(ForeignException.class)); + verify(coordinator, times(1)).rpcConnectionFailure(anyString(), eq(cause)); + verify(controller, times(1)).sendGlobalBarrierAcquire(eq(spy), + eq(procData), anyListOf(String.class)); + verify(controller, times(1)).sendGlobalBarrierReached(any(Procedure.class), + anyListOf(String.class)); + } + + @Test(timeout = 60000) + public void testNoCohort() throws Exception { + runSimpleProcedure(); + } + + @Test(timeout = 60000) + public void testSingleCohortOrchestration() throws Exception { + runSimpleProcedure("one"); + } + + @Test(timeout = 60000) + public void testMultipleCohortOrchestration() throws Exception { + runSimpleProcedure("one", "two", "three", "four"); + } + + public void runSimpleProcedure(String... members) throws Exception { + coordinator = buildNewCoordinator(); + Procedure task = new Procedure(coordinator, monitor, WAKE_FREQUENCY, + TIMEOUT, procName, procData, Arrays.asList(members)); + final Procedure spy = spy(task); + runCoordinatedProcedure(spy, members); + } + + /** + * Test that if nodes join the barrier early we still correctly handle the progress + */ + @Test(timeout = 60000) + public void testEarlyJoiningBarrier() throws Exception { + final String[] cohort = new String[] { "one", "two", "three", "four" }; + coordinator = buildNewCoordinator(); + final ProcedureCoordinator ref = coordinator; + Procedure task = new Procedure(coordinator, monitor, WAKE_FREQUENCY, + TIMEOUT, procName, procData, Arrays.asList(cohort)); + final Procedure spy = spy(task); + + AcquireBarrierAnswer prepare = new AcquireBarrierAnswer(procName, cohort) { + public void doWork() { + // then do some fun where we commit before all nodes have prepared + // "one" commits before anyone else is done + ref.memberAcquiredBarrier(this.opName, this.cohort[0]); + ref.memberFinishedBarrier(this.opName, this.cohort[0]); + // but "two" takes a while + ref.memberAcquiredBarrier(this.opName, this.cohort[1]); + // "three"jumps ahead + ref.memberAcquiredBarrier(this.opName, this.cohort[2]); + ref.memberFinishedBarrier(this.opName, this.cohort[2]); + // and "four" takes a while + ref.memberAcquiredBarrier(this.opName, this.cohort[3]); + } + }; + + BarrierAnswer commit = new BarrierAnswer(procName, cohort) { + @Override + public void doWork() { + ref.memberFinishedBarrier(opName, this.cohort[1]); + ref.memberFinishedBarrier(opName, this.cohort[3]); + } + }; + runCoordinatedOperation(spy, prepare, commit, cohort); + } + + /** + * Just run a procedure with the standard name and data, with not special task for the mock + * coordinator (it works just like a regular coordinator). For custom behavior see + * {@link #runCoordinatedOperation(Procedure, AcquireBarrierAnswer, BarrierAnswer, String[])} + * . + * @param spy Spy on a real {@link Procedure} + * @param cohort expected cohort members + * @throws Exception on failure + */ + public void runCoordinatedProcedure(Procedure spy, String... cohort) throws Exception { + runCoordinatedOperation(spy, new AcquireBarrierAnswer(procName, cohort), + new BarrierAnswer(procName, cohort), cohort); + } + + public void runCoordinatedOperation(Procedure spy, AcquireBarrierAnswer prepare, + String... cohort) throws Exception { + runCoordinatedOperation(spy, prepare, new BarrierAnswer(procName, cohort), cohort); + } + + public void runCoordinatedOperation(Procedure spy, BarrierAnswer commit, + String... cohort) throws Exception { + runCoordinatedOperation(spy, new AcquireBarrierAnswer(procName, cohort), commit, cohort); + } + + public void runCoordinatedOperation(Procedure spy, AcquireBarrierAnswer prepareOperation, + BarrierAnswer commitOperation, String... cohort) throws Exception { + List expected = Arrays.asList(cohort); + when(coordinator.createProcedure(any(ForeignExceptionDispatcher.class), eq(procName), eq(procData), anyListOf(String.class))) + .thenReturn(spy); + + // use the passed controller responses + doAnswer(prepareOperation).when(controller).sendGlobalBarrierAcquire(spy, procData, expected); + doAnswer(commitOperation).when(controller) + .sendGlobalBarrierReached(eq(spy), anyListOf(String.class)); + + // run the operation + Procedure task = coordinator.startProcedure(spy.getErrorMonitor(), procName, procData, expected); + // and wait for it to finish + task.waitForCompleted(); + + // make sure we mocked correctly + prepareOperation.ensureRan(); + // we never got an exception + InOrder inorder = inOrder(spy, controller); + inorder.verify(spy).sendGlobalBarrierStart(); + inorder.verify(controller).sendGlobalBarrierAcquire(task, procData, expected); + inorder.verify(spy).sendGlobalBarrierReached(); + inorder.verify(controller).sendGlobalBarrierReached(eq(task), anyListOf(String.class)); + } + + private abstract class OperationAnswer implements Answer { + private boolean ran = false; + + public void ensureRan() { + assertTrue("Prepare mocking didn't actually run!", ran); + } + + @Override + public final Void answer(InvocationOnMock invocation) throws Throwable { + this.ran = true; + doWork(); + return null; + } + + protected abstract void doWork() throws Throwable; + } + + /** + * Just tell the current coordinator that each of the nodes has prepared + */ + private class AcquireBarrierAnswer extends OperationAnswer { + protected final String[] cohort; + protected final String opName; + + public AcquireBarrierAnswer(String opName, String... cohort) { + this.cohort = cohort; + this.opName = opName; + } + + @Override + public void doWork() { + if (cohort == null) return; + for (String member : cohort) { + TestProcedureCoordinator.this.coordinator.memberAcquiredBarrier(opName, member); + } + } + } + + /** + * Just tell the current coordinator that each of the nodes has committed + */ + private class BarrierAnswer extends OperationAnswer { + protected final String[] cohort; + protected final String opName; + + public BarrierAnswer(String opName, String... cohort) { + this.cohort = cohort; + this.opName = opName; + } + + @Override + public void doWork() { + if (cohort == null) return; + for (String member : cohort) { + TestProcedureCoordinator.this.coordinator.memberFinishedBarrier(opName, member); + } + } + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureMember.java b/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureMember.java new file mode 100644 index 000000000000..29ca89c23b74 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureMember.java @@ -0,0 +1,444 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.procedure; + +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyString; +import static org.mockito.Matchers.eq; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.inOrder; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyZeroInteractions; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.util.concurrent.ThreadPoolExecutor; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.errorhandling.ForeignException; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.apache.hadoop.hbase.errorhandling.TimeoutException; +import org.apache.hadoop.hbase.procedure.Subprocedure.SubprocedureImpl; +import org.junit.After; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.InOrder; +import org.mockito.Mockito; +import org.mockito.invocation.InvocationOnMock; +import org.mockito.stubbing.Answer; + +/** + * Test the procedure member, and it's error handling mechanisms. + */ +@Category(SmallTests.class) +public class TestProcedureMember { + private static final long WAKE_FREQUENCY = 100; + private static final long TIMEOUT = 100000; + private static final long POOL_KEEP_ALIVE = 1; + + private final String op = "some op"; + private final byte[] data = new byte[0]; + private final ForeignExceptionDispatcher mockListener = Mockito + .spy(new ForeignExceptionDispatcher()); + private final SubprocedureFactory mockBuilder = mock(SubprocedureFactory.class); + private final ProcedureMemberRpcs mockMemberComms = Mockito + .mock(ProcedureMemberRpcs.class); + private ProcedureMember member; + private ForeignExceptionDispatcher dispatcher; + Subprocedure spySub; + + /** + * Reset all the mock objects + */ + @After + public void resetTest() { + reset(mockListener, mockBuilder, mockMemberComms); + if (member != null) + try { + member.close(); + } catch (IOException e) { + e.printStackTrace(); + } + } + + /** + * Build a member using the class level mocks + * @return member to use for tests + */ + private ProcedureMember buildCohortMember() { + String name = "node"; + ThreadPoolExecutor pool = ProcedureMember.defaultPool(name, 1, POOL_KEEP_ALIVE); + return new ProcedureMember(mockMemberComms, pool, mockBuilder); + } + + /** + * Setup a procedure member that returns the spied-upon {@link Subprocedure}. + */ + private void buildCohortMemberPair() throws IOException { + dispatcher = new ForeignExceptionDispatcher(); + String name = "node"; + ThreadPoolExecutor pool = ProcedureMember.defaultPool(name, 1, POOL_KEEP_ALIVE); + member = new ProcedureMember(mockMemberComms, pool, mockBuilder); + when(mockMemberComms.getMemberName()).thenReturn("membername"); // needed for generating exception + Subprocedure subproc = new EmptySubprocedure(member, dispatcher); + spySub = spy(subproc); + when(mockBuilder.buildSubprocedure(op, data)).thenReturn(spySub); + addCommitAnswer(); + } + + + /** + * Add a 'in barrier phase' response to the mock controller when it gets a acquired notification + */ + private void addCommitAnswer() throws IOException { + doAnswer(new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + member.receivedReachedGlobalBarrier(op); + return null; + } + }).when(mockMemberComms).sendMemberAcquired(any(Subprocedure.class)); + } + + /** + * Test the normal sub procedure execution case. + */ + @Test(timeout = 60000) + public void testSimpleRun() throws Exception { + member = buildCohortMember(); + EmptySubprocedure subproc = new EmptySubprocedure(member, mockListener); + EmptySubprocedure spy = spy(subproc); + when(mockBuilder.buildSubprocedure(op, data)).thenReturn(spy); + + // when we get a prepare, then start the commit phase + addCommitAnswer(); + + // run the operation + // build a new operation + Subprocedure subproc1 = member.createSubprocedure(op, data); + member.submitSubprocedure(subproc1); + // and wait for it to finish + subproc.waitForLocallyCompleted(); + + // make sure everything ran in order + InOrder order = inOrder(mockMemberComms, spy); + order.verify(spy).acquireBarrier(); + order.verify(mockMemberComms).sendMemberAcquired(eq(spy)); + order.verify(spy).insideBarrier(); + order.verify(mockMemberComms).sendMemberCompleted(eq(spy)); + order.verify(mockMemberComms, never()).sendMemberAborted(eq(spy), + any(ForeignException.class)); + } + + /** + * Make sure we call cleanup etc, when we have an exception during + * {@link Subprocedure#acquireBarrier()}. + */ + @Test(timeout = 60000) + public void testMemberPrepareException() throws Exception { + buildCohortMemberPair(); + + // mock an exception on Subprocedure's prepare + doAnswer( + new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + throw new IOException("Forced IOException in member acquireBarrier"); + } + }).when(spySub).acquireBarrier(); + + // run the operation + // build a new operation + Subprocedure subproc = member.createSubprocedure(op, data); + member.submitSubprocedure(subproc); + // if the operation doesn't die properly, then this will timeout + member.closeAndWait(TIMEOUT); + + // make sure everything ran in order + InOrder order = inOrder(mockMemberComms, spySub); + order.verify(spySub).acquireBarrier(); + // Later phases not run + order.verify(mockMemberComms, never()).sendMemberAcquired(eq(spySub)); + order.verify(spySub, never()).insideBarrier(); + order.verify(mockMemberComms, never()).sendMemberCompleted(eq(spySub)); + // error recovery path exercised + order.verify(spySub).cancel(anyString(), any(Exception.class)); + order.verify(spySub).cleanup(any(Exception.class)); + } + + /** + * Make sure we call cleanup etc, when we have an exception during prepare. + */ + @Test(timeout = 60000) + public void testSendMemberAcquiredCommsFailure() throws Exception { + buildCohortMemberPair(); + + // mock an exception on Subprocedure's prepare + doAnswer( + new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + throw new IOException("Forced IOException in memeber prepare"); + } + }).when(mockMemberComms).sendMemberAcquired(any(Subprocedure.class)); + + // run the operation + // build a new operation + Subprocedure subproc = member.createSubprocedure(op, data); + member.submitSubprocedure(subproc); + // if the operation doesn't die properly, then this will timeout + member.closeAndWait(TIMEOUT); + + // make sure everything ran in order + InOrder order = inOrder(mockMemberComms, spySub); + order.verify(spySub).acquireBarrier(); + order.verify(mockMemberComms).sendMemberAcquired(eq(spySub)); + + // Later phases not run + order.verify(spySub, never()).insideBarrier(); + order.verify(mockMemberComms, never()).sendMemberCompleted(eq(spySub)); + // error recovery path exercised + order.verify(spySub).cancel(anyString(), any(Exception.class)); + order.verify(spySub).cleanup(any(Exception.class)); + } + + /** + * Fail correctly if coordinator aborts the procedure. The subprocedure will not interrupt a + * running {@link Subprocedure#prepare} -- prepare needs to finish first, and the the abort + * is checked. Thus, the {@link Subprocedure#prepare} should succeed but later get rolled back + * via {@link Subprocedure#cleanup}. + */ + @Test(timeout = 60000) + public void testCoordinatorAbort() throws Exception { + buildCohortMemberPair(); + + // mock that another node timed out or failed to prepare + final TimeoutException oate = new TimeoutException("bogus timeout", 1,2,0); + doAnswer( + new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + // inject a remote error (this would have come from an external thread) + spySub.cancel("bogus message", oate); + // sleep the wake frequency since that is what we promised + Thread.sleep(WAKE_FREQUENCY); + return null; + } + }).when(spySub).waitForReachedGlobalBarrier(); + + // run the operation + // build a new operation + Subprocedure subproc = member.createSubprocedure(op, data); + member.submitSubprocedure(subproc); + // if the operation doesn't die properly, then this will timeout + member.closeAndWait(TIMEOUT); + + // make sure everything ran in order + InOrder order = inOrder(mockMemberComms, spySub); + order.verify(spySub).acquireBarrier(); + order.verify(mockMemberComms).sendMemberAcquired(eq(spySub)); + // Later phases not run + order.verify(spySub, never()).insideBarrier(); + order.verify(mockMemberComms, never()).sendMemberCompleted(eq(spySub)); + // error recovery path exercised + order.verify(spySub).cancel(anyString(), any(Exception.class)); + order.verify(spySub).cleanup(any(Exception.class)); + } + + /** + * Handle failures if a member's commit phase fails. + * + * NOTE: This is the core difference that makes this different from traditional 2PC. In true + * 2PC the transaction is committed just before the coordinator sends commit messages to the + * member. Members are then responsible for reading its TX log. This implementation actually + * rolls back, and thus breaks the normal TX guarantees. + */ + @Test(timeout = 60000) + public void testMemberCommitException() throws Exception { + buildCohortMemberPair(); + + // mock an exception on Subprocedure's prepare + doAnswer( + new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + throw new IOException("Forced IOException in memeber prepare"); + } + }).when(spySub).insideBarrier(); + + // run the operation + // build a new operation + Subprocedure subproc = member.createSubprocedure(op, data); + member.submitSubprocedure(subproc); + // if the operation doesn't die properly, then this will timeout + member.closeAndWait(TIMEOUT); + + // make sure everything ran in order + InOrder order = inOrder(mockMemberComms, spySub); + order.verify(spySub).acquireBarrier(); + order.verify(mockMemberComms).sendMemberAcquired(eq(spySub)); + order.verify(spySub).insideBarrier(); + + // Later phases not run + order.verify(mockMemberComms, never()).sendMemberCompleted(eq(spySub)); + // error recovery path exercised + order.verify(spySub).cancel(anyString(), any(Exception.class)); + order.verify(spySub).cleanup(any(Exception.class)); + } + + /** + * Handle Failures if a member's commit phase succeeds but notification to coordinator fails + * + * NOTE: This is the core difference that makes this different from traditional 2PC. In true + * 2PC the transaction is committed just before the coordinator sends commit messages to the + * member. Members are then responsible for reading its TX log. This implementation actually + * rolls back, and thus breaks the normal TX guarantees. + */ + @Test(timeout = 60000) + public void testMemberCommitCommsFailure() throws Exception { + buildCohortMemberPair(); + final TimeoutException oate = new TimeoutException("bogus timeout",1,2,0); + doAnswer( + new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + // inject a remote error (this would have come from an external thread) + spySub.cancel("commit comms fail", oate); + // sleep the wake frequency since that is what we promised + Thread.sleep(WAKE_FREQUENCY); + return null; + } + }).when(mockMemberComms).sendMemberCompleted(any(Subprocedure.class)); + + // run the operation + // build a new operation + Subprocedure subproc = member.createSubprocedure(op, data); + member.submitSubprocedure(subproc); + // if the operation doesn't die properly, then this will timeout + member.closeAndWait(TIMEOUT); + + // make sure everything ran in order + InOrder order = inOrder(mockMemberComms, spySub); + order.verify(spySub).acquireBarrier(); + order.verify(mockMemberComms).sendMemberAcquired(eq(spySub)); + order.verify(spySub).insideBarrier(); + order.verify(mockMemberComms).sendMemberCompleted(eq(spySub)); + // error recovery path exercised + order.verify(spySub).cancel(anyString(), any(Exception.class)); + order.verify(spySub).cleanup(any(Exception.class)); + } + + /** + * Fail correctly on getting an external error while waiting for the prepared latch + * @throws Exception on failure + */ + @Test(timeout = 1000) + public void testPropagateConnectionErrorBackToManager() throws Exception { + // setup the operation + member = buildCohortMember(); + ProcedureMember memberSpy = spy(member); + + // setup the commit and the spy + final ForeignExceptionDispatcher dispatcher = new ForeignExceptionDispatcher(); + ForeignExceptionDispatcher dispSpy = spy(dispatcher); + Subprocedure commit = new EmptySubprocedure(member, dispatcher); + Subprocedure spy = spy(commit); + when(mockBuilder.buildSubprocedure(op, data)).thenReturn(spy); + + // fail during the prepare phase + doThrow(new ForeignException("SRC", "prepare exception")).when(spy).acquireBarrier(); + // and throw a connection error when we try to tell the controller about it + doThrow(new IOException("Controller is down!")).when(mockMemberComms) + .sendMemberAborted(eq(spy), any(ForeignException.class)); + + + // run the operation + // build a new operation + Subprocedure subproc = memberSpy.createSubprocedure(op, data); + memberSpy.submitSubprocedure(subproc); + // if the operation doesn't die properly, then this will timeout + memberSpy.closeAndWait(TIMEOUT); + + // make sure everything ran in order + InOrder order = inOrder(mockMemberComms, spy, dispSpy); + // make sure we acquire. + order.verify(spy).acquireBarrier(); + order.verify(mockMemberComms, never()).sendMemberAcquired(spy); + + // TODO Need to do another refactor to get this to propagate to the coordinator. + // make sure we pass a remote exception back the controller +// order.verify(mockMemberComms).sendMemberAborted(eq(spy), +// any(ExternalException.class)); +// order.verify(dispSpy).receiveError(anyString(), +// any(ExternalException.class), any()); + } + + /** + * Test that the cohort member correctly doesn't attempt to start a task when the builder cannot + * correctly build a new task for the requested operation + * @throws Exception on failure + */ + @Test + public void testNoTaskToBeRunFromRequest() throws Exception { + ThreadPoolExecutor pool = mock(ThreadPoolExecutor.class); + when(mockBuilder.buildSubprocedure(op, data)).thenReturn(null) + .thenThrow(new IllegalStateException("Wrong state!"), new IllegalArgumentException("can't understand the args")); + member = new ProcedureMember(mockMemberComms, pool, mockBuilder); + // builder returns null + // build a new operation + Subprocedure subproc = member.createSubprocedure(op, data); + member.submitSubprocedure(subproc); + // throws an illegal state exception + try { + // build a new operation + Subprocedure subproc2 = member.createSubprocedure(op, data); + member.submitSubprocedure(subproc2); + } catch (IllegalStateException ise) { + } + // throws an illegal argument exception + try { + // build a new operation + Subprocedure subproc3 = member.createSubprocedure(op, data); + member.submitSubprocedure(subproc3); + } catch (IllegalArgumentException iae) { + } + + // no request should reach the pool + verifyZeroInteractions(pool); + // get two abort requests + // TODO Need to do another refactor to get this to propagate to the coordinator. + // verify(mockMemberComms, times(2)).sendMemberAborted(any(Subprocedure.class), any(ExternalException.class)); + } + + /** + * Helper {@link Procedure} who's phase for each step is just empty + */ + public class EmptySubprocedure extends SubprocedureImpl { + public EmptySubprocedure(ProcedureMember member, ForeignExceptionDispatcher dispatcher) { + super( member, op, dispatcher, + // TODO 1000000 is an arbitrary number that I picked. + WAKE_FREQUENCY, TIMEOUT); + } + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedure.java b/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedure.java new file mode 100644 index 000000000000..6e098c0e45de --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedure.java @@ -0,0 +1,408 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.procedure; + +import static org.junit.Assert.assertEquals; +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyListOf; +import static org.mockito.Matchers.eq; +import static org.mockito.Mockito.atMost; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.Abortable; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.errorhandling.ForeignException; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.apache.hadoop.hbase.errorhandling.TimeoutException; +import org.apache.hadoop.hbase.procedure.Subprocedure.SubprocedureImpl; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; +import org.mockito.internal.matchers.ArrayEquals; +import org.mockito.invocation.InvocationOnMock; +import org.mockito.stubbing.Answer; +import org.mockito.verification.VerificationMode; + +import com.google.common.collect.Lists; + +/** + * Cluster-wide testing of a distributed three-phase commit using a 'real' zookeeper cluster + */ +@Category(MediumTests.class) +public class TestZKProcedure { + + private static final Log LOG = LogFactory.getLog(TestZKProcedure.class); + private static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static final String COORDINATOR_NODE_NAME = "coordinator"; + private static final long KEEP_ALIVE = 100; // seconds + private static final int POOL_SIZE = 1; + private static final long TIMEOUT = 10000; // when debugging make this larger for debugging + private static final long WAKE_FREQUENCY = 500; + private static final String opName = "op"; + private static final byte[] data = new byte[] { 1, 2 }; // TODO what is this used for? + private static final VerificationMode once = Mockito.times(1); + + @BeforeClass + public static void setupTest() throws Exception { + UTIL.startMiniZKCluster(); + } + + @AfterClass + public static void cleanupTest() throws Exception { + UTIL.shutdownMiniZKCluster(); + } + + private static ZooKeeperWatcher newZooKeeperWatcher() throws IOException { + return new ZooKeeperWatcher(UTIL.getConfiguration(), "testing utility", new Abortable() { + @Override + public void abort(String why, Throwable e) { + throw new RuntimeException( + "Unexpected abort in distributed three phase commit test:" + why, e); + } + + @Override + public boolean isAborted() { + return false; + } + }); + } + + @Test + public void testEmptyMemberSet() throws Exception { + runCommit(); + } + + @Test + public void testSingleMember() throws Exception { + runCommit("one"); + } + + @Test + public void testMultipleMembers() throws Exception { + runCommit("one", "two", "three", "four" ); + } + + private void runCommit(String... members) throws Exception { + // make sure we just have an empty list + if (members == null) { + members = new String[0]; + } + List expected = Arrays.asList(members); + + // setup the constants + ZooKeeperWatcher coordZkw = newZooKeeperWatcher(); + String opDescription = "coordination test - " + members.length + " cohort members"; + + // start running the controller + ZKProcedureCoordinatorRpcs coordinatorComms = new ZKProcedureCoordinatorRpcs( + coordZkw, opDescription, COORDINATOR_NODE_NAME); + ThreadPoolExecutor pool = ProcedureCoordinator.defaultPool(COORDINATOR_NODE_NAME, POOL_SIZE, KEEP_ALIVE); + ProcedureCoordinator coordinator = new ProcedureCoordinator(coordinatorComms, pool) { + @Override + public Procedure createProcedure(ForeignExceptionDispatcher fed, String procName, byte[] procArgs, + List expectedMembers) { + return Mockito.spy(super.createProcedure(fed, procName, procArgs, expectedMembers)); + } + }; + + // build and start members + // NOTE: There is a single subprocedure builder for all members here. + SubprocedureFactory subprocFactory = Mockito.mock(SubprocedureFactory.class); + List> procMembers = new ArrayList>( + members.length); + // start each member + for (String member : members) { + ZooKeeperWatcher watcher = newZooKeeperWatcher(); + ZKProcedureMemberRpcs comms = new ZKProcedureMemberRpcs(watcher, opDescription); + ThreadPoolExecutor pool2 = ProcedureMember.defaultPool(member, 1, KEEP_ALIVE); + ProcedureMember procMember = new ProcedureMember(comms, pool2, subprocFactory); + procMembers.add(new Pair(procMember, comms)); + comms.start(member, procMember); + } + + // setup mock member subprocedures + final List subprocs = new ArrayList(); + for (int i = 0; i < procMembers.size(); i++) { + ForeignExceptionDispatcher cohortMonitor = new ForeignExceptionDispatcher(); + Subprocedure commit = Mockito + .spy(new SubprocedureImpl(procMembers.get(i).getFirst(), opName, cohortMonitor, + WAKE_FREQUENCY, TIMEOUT)); + subprocs.add(commit); + } + + // link subprocedure to buildNewOperation invocation. + final AtomicInteger i = new AtomicInteger(0); // NOTE: would be racy if not an AtomicInteger + Mockito.when(subprocFactory.buildSubprocedure(Mockito.eq(opName), + (byte[]) Mockito.argThat(new ArrayEquals(data)))).thenAnswer( + new Answer() { + @Override + public Subprocedure answer(InvocationOnMock invocation) throws Throwable { + int index = i.getAndIncrement(); + LOG.debug("Task size:" + subprocs.size() + ", getting:" + index); + Subprocedure commit = subprocs.get(index); + return commit; + } + }); + + // setup spying on the coordinator +// Procedure proc = Mockito.spy(procBuilder.createProcedure(coordinator, opName, data, expected)); +// Mockito.when(procBuilder.build(coordinator, opName, data, expected)).thenReturn(proc); + + // start running the operation + Procedure task = coordinator.startProcedure(new ForeignExceptionDispatcher(), opName, data, expected); +// assertEquals("Didn't mock coordinator task", proc, task); + + // verify all things ran as expected +// waitAndVerifyProc(proc, once, once, never(), once, false); + waitAndVerifyProc(task, once, once, never(), once, false); + verifyCohortSuccessful(expected, subprocFactory, subprocs, once, once, never(), once, false); + + // close all the things + closeAll(coordinator, coordinatorComms, procMembers); + } + + /** + * Test a distributed commit with multiple cohort members, where one of the cohort members has a + * timeout exception during the prepare stage. + */ + @Test + public void testMultiCohortWithMemberTimeoutDuringPrepare() throws Exception { + String opDescription = "error injection coordination"; + String[] cohortMembers = new String[] { "one", "two", "three" }; + List expected = Lists.newArrayList(cohortMembers); + // error constants + final int memberErrorIndex = 2; + final CountDownLatch coordinatorReceivedErrorLatch = new CountDownLatch(1); + + // start running the coordinator and its controller + ZooKeeperWatcher coordinatorWatcher = newZooKeeperWatcher(); + ZKProcedureCoordinatorRpcs coordinatorController = new ZKProcedureCoordinatorRpcs( + coordinatorWatcher, opDescription, COORDINATOR_NODE_NAME); + ThreadPoolExecutor pool = ProcedureCoordinator.defaultPool(COORDINATOR_NODE_NAME, POOL_SIZE, KEEP_ALIVE); + ProcedureCoordinator coordinator = spy(new ProcedureCoordinator(coordinatorController, pool)); + + // start a member for each node + SubprocedureFactory subprocFactory = Mockito.mock(SubprocedureFactory.class); + List> members = new ArrayList>( + expected.size()); + for (String member : expected) { + ZooKeeperWatcher watcher = newZooKeeperWatcher(); + ZKProcedureMemberRpcs controller = new ZKProcedureMemberRpcs(watcher, opDescription); + ThreadPoolExecutor pool2 = ProcedureMember.defaultPool(member, 1, KEEP_ALIVE); + ProcedureMember mem = new ProcedureMember(controller, pool2, subprocFactory); + members.add(new Pair(mem, controller)); + controller.start(member, mem); + } + + // setup mock subprocedures + final List cohortTasks = new ArrayList(); + final int[] elem = new int[1]; + for (int i = 0; i < members.size(); i++) { + ForeignExceptionDispatcher cohortMonitor = new ForeignExceptionDispatcher(); + ProcedureMember comms = members.get(i).getFirst(); + Subprocedure commit = Mockito + .spy(new SubprocedureImpl(comms, opName, cohortMonitor, WAKE_FREQUENCY, TIMEOUT)); + // This nasty bit has one of the impls throw a TimeoutException + Mockito.doAnswer(new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + int index = elem[0]; + if (index == memberErrorIndex) { + LOG.debug("Sending error to coordinator"); + ForeignException remoteCause = new ForeignException("TIMER", + new TimeoutException("subprocTimeout" , 1, 2, 0)); + Subprocedure r = ((Subprocedure) invocation.getMock()); + LOG.error("Remote commit failure, not propagating error:" + remoteCause); + r.monitor.receive(remoteCause); + // don't complete the error phase until the coordinator has gotten the error + // notification (which ensures that we never progress past prepare) + try { + Procedure.waitForLatch(coordinatorReceivedErrorLatch, new ForeignExceptionDispatcher(), + WAKE_FREQUENCY, "coordinator received error"); + } catch (InterruptedException e) { + LOG.debug("Wait for latch interrupted, done:" + (coordinatorReceivedErrorLatch.getCount() == 0)); + // reset the interrupt status on the thread + Thread.currentThread().interrupt(); + } + } + elem[0] = ++index; + return null; + } + }).when(commit).acquireBarrier(); + cohortTasks.add(commit); + } + + // pass out a task per member + final int[] i = new int[] { 0 }; + Mockito.when( + subprocFactory.buildSubprocedure(Mockito.eq(opName), + (byte[]) Mockito.argThat(new ArrayEquals(data)))).thenAnswer( + new Answer() { + @Override + public Subprocedure answer(InvocationOnMock invocation) throws Throwable { + int index = i[0]; + Subprocedure commit = cohortTasks.get(index); + index++; + i[0] = index; + return commit; + } + }); + + // setup spying on the coordinator + ForeignExceptionDispatcher coordinatorTaskErrorMonitor = Mockito + .spy(new ForeignExceptionDispatcher()); + Procedure coordinatorTask = Mockito.spy(new Procedure(coordinator, + coordinatorTaskErrorMonitor, WAKE_FREQUENCY, TIMEOUT, + opName, data, expected)); + when(coordinator.createProcedure(any(ForeignExceptionDispatcher.class), eq(opName), eq(data), anyListOf(String.class))) + .thenReturn(coordinatorTask); + // count down the error latch when we get the remote error + Mockito.doAnswer(new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + // pass on the error to the master + invocation.callRealMethod(); + // then count down the got error latch + coordinatorReceivedErrorLatch.countDown(); + return null; + } + }).when(coordinatorTask).receive(Mockito.any(ForeignException.class)); + + // ---------------------------- + // start running the operation + // ---------------------------- + + Procedure task = coordinator.startProcedure(coordinatorTaskErrorMonitor, opName, data, expected); + assertEquals("Didn't mock coordinator task", coordinatorTask, task); + + // wait for the task to complete + try { + task.waitForCompleted(); + } catch (ForeignException fe) { + // this may get caught or may not + } + + // ------------- + // verification + // ------------- + // always expect prepared, never committed, and possible to have cleanup and finish (racy since + // error case) + waitAndVerifyProc(coordinatorTask, once, never(), once, atMost(1), true); + verifyCohortSuccessful(expected, subprocFactory, cohortTasks, once, never(), once, + once, true); + + // close all the open things + closeAll(coordinator, coordinatorController, members); + } + + /** + * Wait for the coordinator task to complete, and verify all the mocks + * @param task to wait on + * @throws Exception on unexpected failure + */ + private void waitAndVerifyProc(Procedure proc, VerificationMode prepare, + VerificationMode commit, VerificationMode cleanup, VerificationMode finish, boolean opHasError) + throws Exception { + boolean caughtError = false; + try { + proc.waitForCompleted(); + } catch (ForeignException fe) { + caughtError = true; + } + // make sure that the task called all the expected phases + Mockito.verify(proc, prepare).sendGlobalBarrierStart(); + Mockito.verify(proc, commit).sendGlobalBarrierReached(); + Mockito.verify(proc, finish).sendGlobalBarrierComplete(); + assertEquals("Operation error state was unexpected", opHasError, proc.getErrorMonitor() + .hasException()); + assertEquals("Operation error state was unexpected", opHasError, caughtError); + + } + + /** + * Wait for the coordinator task to complete, and verify all the mocks + * @param task to wait on + * @throws Exception on unexpected failure + */ + private void waitAndVerifySubproc(Subprocedure op, VerificationMode prepare, + VerificationMode commit, VerificationMode cleanup, VerificationMode finish, boolean opHasError) + throws Exception { + boolean caughtError = false; + try { + op.waitForLocallyCompleted(); + } catch (ForeignException fe) { + caughtError = true; + } + // make sure that the task called all the expected phases + Mockito.verify(op, prepare).acquireBarrier(); + Mockito.verify(op, commit).insideBarrier(); + // We cannot guarantee that cleanup has run so we don't check it. + + assertEquals("Operation error state was unexpected", opHasError, op.getErrorCheckable() + .hasException()); + assertEquals("Operation error state was unexpected", opHasError, caughtError); + + } + + private void verifyCohortSuccessful(List cohortNames, + SubprocedureFactory subprocFactory, Iterable cohortTasks, + VerificationMode prepare, VerificationMode commit, VerificationMode cleanup, + VerificationMode finish, boolean opHasError) throws Exception { + + // make sure we build the correct number of cohort members + Mockito.verify(subprocFactory, Mockito.times(cohortNames.size())).buildSubprocedure( + Mockito.eq(opName), (byte[]) Mockito.argThat(new ArrayEquals(data))); + // verify that we ran each of the operations cleanly + int j = 0; + for (Subprocedure op : cohortTasks) { + LOG.debug("Checking mock:" + (j++)); + waitAndVerifySubproc(op, prepare, commit, cleanup, finish, opHasError); + } + } + + private void closeAll( + ProcedureCoordinator coordinator, + ZKProcedureCoordinatorRpcs coordinatorController, + List> cohort) + throws IOException { + // make sure we close all the resources + for (Pair member : cohort) { + member.getFirst().close(); + member.getSecond().close(); + } + coordinator.close(); + coordinatorController.close(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedureControllers.java b/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedureControllers.java new file mode 100644 index 000000000000..6065cb6e0a1b --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedureControllers.java @@ -0,0 +1,427 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.procedure; + +import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.errorhandling.ForeignException; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.apache.hadoop.hbase.protobuf.ProtobufUtil; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.hadoop.hbase.zookeeper.ZKUtil; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; +import org.mockito.invocation.InvocationOnMock; +import org.mockito.stubbing.Answer; +import org.mockito.verification.VerificationMode; + +import com.google.common.collect.Lists; + +/** + * Test zookeeper-based, procedure controllers + */ +@Category(MediumTests.class) +public class TestZKProcedureControllers { + + static final Log LOG = LogFactory.getLog(TestZKProcedureControllers.class); + private final static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static final String COHORT_NODE_NAME = "expected"; + private static final String CONTROLLER_NODE_NAME = "controller"; + private static final VerificationMode once = Mockito.times(1); + + @BeforeClass + public static void setupTest() throws Exception { + UTIL.startMiniZKCluster(); + } + + @AfterClass + public static void cleanupTest() throws Exception { + UTIL.shutdownMiniZKCluster(); + } + + /** + * Smaller test to just test the actuation on the cohort member + * @throws Exception on failure + */ + @Test(timeout = 60000) + public void testSimpleZKCohortMemberController() throws Exception { + ZooKeeperWatcher watcher = HBaseTestingUtility.getZooKeeperWatcher(UTIL); + final String operationName = "instanceTest"; + + final Subprocedure sub = Mockito.mock(Subprocedure.class); + Mockito.when(sub.getName()).thenReturn(operationName); + + final byte[] data = new byte[] { 1, 2, 3 }; + final CountDownLatch prepared = new CountDownLatch(1); + final CountDownLatch committed = new CountDownLatch(1); + + final ForeignExceptionDispatcher monitor = spy(new ForeignExceptionDispatcher()); + final ZKProcedureMemberRpcs controller = new ZKProcedureMemberRpcs( + watcher, "testSimple"); + + // mock out cohort member callbacks + final ProcedureMember member = Mockito + .mock(ProcedureMember.class); + Mockito.doReturn(sub).when(member).createSubprocedure(operationName, data); + Mockito.doAnswer(new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + controller.sendMemberAcquired(sub); + prepared.countDown(); + return null; + } + }).when(member).submitSubprocedure(sub); + Mockito.doAnswer(new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + controller.sendMemberCompleted(sub); + committed.countDown(); + return null; + } + }).when(member).receivedReachedGlobalBarrier(operationName); + + // start running the listener + controller.start(COHORT_NODE_NAME, member); + + // set a prepare node from a 'coordinator' + String prepare = ZKProcedureUtil.getAcquireBarrierNode(controller.getZkController(), operationName); + ZKUtil.createSetData(watcher, prepare, ProtobufUtil.prependPBMagic(data)); + // wait for the operation to be prepared + prepared.await(); + + // create the commit node so we update the operation to enter the commit phase + String commit = ZKProcedureUtil.getReachedBarrierNode(controller.getZkController(), operationName); + LOG.debug("Found prepared, posting commit node:" + commit); + ZKUtil.createAndFailSilent(watcher, commit); + LOG.debug("Commit node:" + commit + ", exists:" + ZKUtil.checkExists(watcher, commit)); + committed.await(); + + verify(monitor, never()).receive(Mockito.any(ForeignException.class)); + // XXX: broken due to composition. +// verify(member, never()).getManager().controllerConnectionFailure(Mockito.anyString(), +// Mockito.any(IOException.class)); + // cleanup after the test + ZKUtil.deleteNodeRecursively(watcher, controller.getZkController().getBaseZnode()); + assertEquals("Didn't delete prepare node", -1, ZKUtil.checkExists(watcher, prepare)); + assertEquals("Didn't delete commit node", -1, ZKUtil.checkExists(watcher, commit)); + } + + @Test(timeout = 60000) + public void testZKCoordinatorControllerWithNoCohort() throws Exception { + final String operationName = "no cohort controller test"; + final byte[] data = new byte[] { 1, 2, 3 }; + + runMockCommitWithOrchestratedControllers(startCoordinatorFirst, operationName, data); + runMockCommitWithOrchestratedControllers(startCohortFirst, operationName, data); + } + + @Test(timeout = 60000) + public void testZKCoordinatorControllerWithSingleMemberCohort() throws Exception { + final String operationName = "single member controller test"; + final byte[] data = new byte[] { 1, 2, 3 }; + + runMockCommitWithOrchestratedControllers(startCoordinatorFirst, operationName, data, "cohort"); + runMockCommitWithOrchestratedControllers(startCohortFirst, operationName, data, "cohort"); + } + + @Test(timeout = 60000) + public void testZKCoordinatorControllerMultipleCohort() throws Exception { + final String operationName = "multi member controller test"; + final byte[] data = new byte[] { 1, 2, 3 }; + + runMockCommitWithOrchestratedControllers(startCoordinatorFirst, operationName, data, "cohort", + "cohort2", "cohort3"); + runMockCommitWithOrchestratedControllers(startCohortFirst, operationName, data, "cohort", + "cohort2", "cohort3"); + } + + private void runMockCommitWithOrchestratedControllers(StartControllers controllers, + String operationName, byte[] data, String... cohort) throws Exception { + ZooKeeperWatcher watcher = HBaseTestingUtility.getZooKeeperWatcher(UTIL); + List expected = Lists.newArrayList(cohort); + + final Subprocedure sub = Mockito.mock(Subprocedure.class); + Mockito.when(sub.getName()).thenReturn(operationName); + + CountDownLatch prepared = new CountDownLatch(expected.size()); + CountDownLatch committed = new CountDownLatch(expected.size()); + // mock out coordinator so we can keep track of zk progress + ProcedureCoordinator coordinator = setupMockCoordinator(operationName, + prepared, committed); + + ProcedureMember member = Mockito.mock(ProcedureMember.class); + + Pair> pair = controllers + .start(watcher, operationName, coordinator, CONTROLLER_NODE_NAME, member, expected); + ZKProcedureCoordinatorRpcs controller = pair.getFirst(); + List cohortControllers = pair.getSecond(); + // start the operation + Procedure p = Mockito.mock(Procedure.class); + Mockito.when(p.getName()).thenReturn(operationName); + + controller.sendGlobalBarrierAcquire(p, data, expected); + + // post the prepare node for each expected node + for (ZKProcedureMemberRpcs cc : cohortControllers) { + cc.sendMemberAcquired(sub); + } + + // wait for all the notifications to reach the coordinator + prepared.await(); + // make sure we got the all the nodes and no more + Mockito.verify(coordinator, times(expected.size())).memberAcquiredBarrier(Mockito.eq(operationName), + Mockito.anyString()); + + // kick off the commit phase + controller.sendGlobalBarrierReached(p, expected); + + // post the committed node for each expected node + for (ZKProcedureMemberRpcs cc : cohortControllers) { + cc.sendMemberCompleted(sub); + } + + // wait for all commit notifications to reach the coordinator + committed.await(); + // make sure we got the all the nodes and no more + Mockito.verify(coordinator, times(expected.size())).memberFinishedBarrier(Mockito.eq(operationName), + Mockito.anyString()); + + controller.resetMembers(p); + + // verify all behavior + verifyZooKeeperClean(operationName, watcher, controller.getZkProcedureUtil()); + verifyCohort(member, cohortControllers.size(), operationName, data); + verifyCoordinator(operationName, coordinator, expected); + } + + // TODO Broken by composition. +// @Test +// public void testCoordinatorControllerHandlesEarlyPrepareNodes() throws Exception { +// runEarlyPrepareNodes(startCoordinatorFirst, "testEarlyPreparenodes", new byte[] { 1, 2, 3 }, +// "cohort1", "cohort2"); +// runEarlyPrepareNodes(startCohortFirst, "testEarlyPreparenodes", new byte[] { 1, 2, 3 }, +// "cohort1", "cohort2"); +// } + + public void runEarlyPrepareNodes(StartControllers controllers, String operationName, byte[] data, + String... cohort) throws Exception { + ZooKeeperWatcher watcher = HBaseTestingUtility.getZooKeeperWatcher(UTIL); + List expected = Lists.newArrayList(cohort); + + final Subprocedure sub = Mockito.mock(Subprocedure.class); + Mockito.when(sub.getName()).thenReturn(operationName); + + final CountDownLatch prepared = new CountDownLatch(expected.size()); + final CountDownLatch committed = new CountDownLatch(expected.size()); + // mock out coordinator so we can keep track of zk progress + ProcedureCoordinator coordinator = setupMockCoordinator(operationName, + prepared, committed); + + ProcedureMember member = Mockito.mock(ProcedureMember.class); + Procedure p = Mockito.mock(Procedure.class); + Mockito.when(p.getName()).thenReturn(operationName); + + Pair> pair = controllers + .start(watcher, operationName, coordinator, CONTROLLER_NODE_NAME, member, expected); + ZKProcedureCoordinatorRpcs controller = pair.getFirst(); + List cohortControllers = pair.getSecond(); + + // post 1/2 the prepare nodes early + for (int i = 0; i < cohortControllers.size() / 2; i++) { + cohortControllers.get(i).sendMemberAcquired(sub); + } + + // start the operation + controller.sendGlobalBarrierAcquire(p, data, expected); + + // post the prepare node for each expected node + for (ZKProcedureMemberRpcs cc : cohortControllers) { + cc.sendMemberAcquired(sub); + } + + // wait for all the notifications to reach the coordinator + prepared.await(); + // make sure we got the all the nodes and no more + Mockito.verify(coordinator, times(expected.size())).memberAcquiredBarrier(Mockito.eq(operationName), + Mockito.anyString()); + + // kick off the commit phase + controller.sendGlobalBarrierReached(p, expected); + + // post the committed node for each expected node + for (ZKProcedureMemberRpcs cc : cohortControllers) { + cc.sendMemberCompleted(sub); + } + + // wait for all commit notifications to reach the coordiantor + committed.await(); + // make sure we got the all the nodes and no more + Mockito.verify(coordinator, times(expected.size())).memberFinishedBarrier(Mockito.eq(operationName), + Mockito.anyString()); + + controller.resetMembers(p); + + // verify all behavior + verifyZooKeeperClean(operationName, watcher, controller.getZkProcedureUtil()); + verifyCohort(member, cohortControllers.size(), operationName, data); + verifyCoordinator(operationName, coordinator, expected); + } + + /** + * @return a mock {@link ProcedureCoordinator} that just counts down the + * prepared and committed latch for called to the respective method + */ + private ProcedureCoordinator setupMockCoordinator(String operationName, + final CountDownLatch prepared, final CountDownLatch committed) { + ProcedureCoordinator coordinator = Mockito + .mock(ProcedureCoordinator.class); + Mockito.mock(ProcedureCoordinator.class); + Mockito.doAnswer(new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + prepared.countDown(); + return null; + } + }).when(coordinator).memberAcquiredBarrier(Mockito.eq(operationName), Mockito.anyString()); + Mockito.doAnswer(new Answer() { + @Override + public Void answer(InvocationOnMock invocation) throws Throwable { + committed.countDown(); + return null; + } + }).when(coordinator).memberFinishedBarrier(Mockito.eq(operationName), Mockito.anyString()); + return coordinator; + } + + /** + * Verify that the prepare, commit and abort nodes for the operation are removed from zookeeper + */ + private void verifyZooKeeperClean(String operationName, ZooKeeperWatcher watcher, + ZKProcedureUtil controller) throws Exception { + String prepare = ZKProcedureUtil.getAcquireBarrierNode(controller, operationName); + String commit = ZKProcedureUtil.getReachedBarrierNode(controller, operationName); + String abort = ZKProcedureUtil.getAbortNode(controller, operationName); + assertEquals("Didn't delete prepare node", -1, ZKUtil.checkExists(watcher, prepare)); + assertEquals("Didn't delete commit node", -1, ZKUtil.checkExists(watcher, commit)); + assertEquals("Didn't delete abort node", -1, ZKUtil.checkExists(watcher, abort)); + } + + /** + * Verify the cohort controller got called once per expected node to start the operation + */ + private void verifyCohort(ProcedureMember member, int cohortSize, + String operationName, byte[] data) { +// verify(member, Mockito.times(cohortSize)).submitSubprocedure(Mockito.eq(operationName), +// (byte[]) Mockito.argThat(new ArrayEquals(data))); + verify(member, Mockito.times(cohortSize)).submitSubprocedure(Mockito.any(Subprocedure.class)); + + } + + /** + * Verify that the coordinator only got called once for each expected node + */ + private void verifyCoordinator(String operationName, + ProcedureCoordinator coordinator, List expected) { + // verify that we got all the expected nodes + for (String node : expected) { + verify(coordinator, once).memberAcquiredBarrier(operationName, node); + verify(coordinator, once).memberFinishedBarrier(operationName, node); + } + } + + /** + * Specify how the controllers that should be started (not spy/mockable) for the test. + */ + private abstract class StartControllers { + public abstract Pair> start( + ZooKeeperWatcher watcher, String operationName, + ProcedureCoordinator coordinator, String controllerName, + ProcedureMember member, List cohortNames) throws Exception; + } + + private final StartControllers startCoordinatorFirst = new StartControllers() { + + @Override + public Pair> start( + ZooKeeperWatcher watcher, String operationName, + ProcedureCoordinator coordinator, String controllerName, + ProcedureMember member, List expected) throws Exception { + // start the controller + ZKProcedureCoordinatorRpcs controller = new ZKProcedureCoordinatorRpcs( + watcher, operationName, CONTROLLER_NODE_NAME); + controller.start(coordinator); + + // make a cohort controller for each expected node + + List cohortControllers = new ArrayList(); + for (String nodeName : expected) { + ZKProcedureMemberRpcs cc = new ZKProcedureMemberRpcs(watcher, operationName); + cc.start(nodeName, member); + cohortControllers.add(cc); + } + return new Pair>( + controller, cohortControllers); + } + }; + + /** + * Check for the possible race condition where a cohort member starts after the controller and + * therefore could miss a new operation + */ + private final StartControllers startCohortFirst = new StartControllers() { + + @Override + public Pair> start( + ZooKeeperWatcher watcher, String operationName, + ProcedureCoordinator coordinator, String controllerName, + ProcedureMember member, List expected) throws Exception { + + // make a cohort controller for each expected node + List cohortControllers = new ArrayList(); + for (String nodeName : expected) { + ZKProcedureMemberRpcs cc = new ZKProcedureMemberRpcs(watcher, operationName); + cc.start(nodeName, member); + cohortControllers.add(cc); + } + + // start the controller + ZKProcedureCoordinatorRpcs controller = new ZKProcedureCoordinatorRpcs( + watcher, operationName, CONTROLLER_NODE_NAME); + controller.start(coordinator); + + return new Pair>( + controller, cohortControllers); + } + }; +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java b/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java new file mode 100644 index 000000000000..c5d9cc9cbfca --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java @@ -0,0 +1,46 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.hbase.master.cleaner.BaseHFileCleanerDelegate; + +/** + * HFile archive cleaner that just tells you if it has been run already or not (and allows resets) - + * always attempts to delete the passed file. + *

+ * Just a helper class for testing to make sure the cleaner has been run. + */ +public class CheckedArchivingHFileCleaner extends BaseHFileCleanerDelegate { + + private static boolean checked; + + @Override + public boolean isFileDeletable(FileStatus fStat) { + checked = true; + return true; + } + + public static boolean getChecked() { + return checked; + } + + public static void resetCheck() { + checked = false; + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java b/src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java index cbcbffa54b79..bfa9f0508abb 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -189,6 +188,8 @@ public boolean run(String[] args) throws IOException { .withCompression(compr) .withBloomType(bloomType) .withMaxKeyCount(numKV) + .withChecksumType(HFile.DEFAULT_CHECKSUM_TYPE) + .withBytesPerChecksum(HFile.DEFAULT_BYTES_PER_CHECKSUM) .build(); rand = new Random(); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java b/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java index e4d849331cb7..e6ff17305eca 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java @@ -61,7 +61,6 @@ import org.apache.hadoop.hbase.io.hfile.HFileDataBlockEncoderImpl; import org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter; import org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder; -import org.apache.hadoop.hbase.regionserver.StoreScanner.ScanType; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.LoadTestTool; import org.apache.hadoop.hbase.util.MD5Hash; @@ -356,6 +355,8 @@ public void runMergeWorkload() throws IOException { .withDataBlockEncoder(dataBlockEncoder) .withBloomType(bloomType) .withMaxKeyCount(maxKeyCount) + .withChecksumType(HFile.DEFAULT_CHECKSUM_TYPE) + .withBytesPerChecksum(HFile.DEFAULT_BYTES_PER_CHECKSUM) .build(); StatisticsPrinter statsPrinter = new StatisticsPrinter(); @@ -406,7 +407,7 @@ private void performMerge(List scanners, Store store, Scan scan = new Scan(); // Include deletes - scanner = new StoreScanner(store, scan, scanners, + scanner = new StoreScanner(store, store.scanInfo, scan, scanners, ScanType.MAJOR_COMPACT, Long.MIN_VALUE, Long.MIN_VALUE); ArrayList kvs = new ArrayList(); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/KeyValueScanFixture.java b/src/test/java/org/apache/hadoop/hbase/regionserver/KeyValueScanFixture.java index 08fb91e22926..53c55be8d16b 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/KeyValueScanFixture.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/KeyValueScanFixture.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/NoOpScanPolicyObserver.java b/src/test/java/org/apache/hadoop/hbase/regionserver/NoOpScanPolicyObserver.java new file mode 100644 index 000000000000..c933ade81c94 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/NoOpScanPolicyObserver.java @@ -0,0 +1,79 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.Collections; +import java.util.List; +import java.util.NavigableSet; + +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor; +import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver; +import org.apache.hadoop.hbase.coprocessor.ObserverContext; +import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment; + +/** + * RegionObserver that just reimplements the default behavior, + * in order to validate that all the necessary APIs for this are public + * This observer is also used in {@link TestFromClientSideWithCoprocessor} and + * {@link TestCompactionWithCoprocessor} to make sure that a wide range + * of functionality still behaves as expected. + */ +public class NoOpScanPolicyObserver extends BaseRegionObserver { + /** + * Reimplement the default behavior + */ + @Override + public InternalScanner preFlushScannerOpen(final ObserverContext c, + Store store, KeyValueScanner memstoreScanner, InternalScanner s) throws IOException { + Store.ScanInfo oldSI = store.getScanInfo(); + Store.ScanInfo scanInfo = new Store.ScanInfo(store.getFamily(), oldSI.getTtl(), + oldSI.getTimeToPurgeDeletes(), oldSI.getComparator()); + Scan scan = new Scan(); + scan.setMaxVersions(oldSI.getMaxVersions()); + return new StoreScanner(store, scanInfo, scan, Collections.singletonList(memstoreScanner), + ScanType.MINOR_COMPACT, store.getHRegion().getSmallestReadPoint(), + HConstants.OLDEST_TIMESTAMP); + } + + /** + * Reimplement the default behavior + */ + @Override + public InternalScanner preCompactScannerOpen(final ObserverContext c, + Store store, List scanners, ScanType scanType, long earliestPutTs, + InternalScanner s) throws IOException { + // this demonstrates how to override the scanners default behavior + Store.ScanInfo oldSI = store.getScanInfo(); + Store.ScanInfo scanInfo = new Store.ScanInfo(store.getFamily(), oldSI.getTtl(), + oldSI.getTimeToPurgeDeletes(), oldSI.getComparator()); + Scan scan = new Scan(); + scan.setMaxVersions(oldSI.getMaxVersions()); + return new StoreScanner(store, scanInfo, scan, scanners, scanType, store.getHRegion() + .getSmallestReadPoint(), earliestPutTs); + } + + @Override + public KeyValueScanner preStoreScannerOpen(final ObserverContext c, + Store store, final Scan scan, final NavigableSet targetCols, KeyValueScanner s) + throws IOException { + return new StoreScanner(store, store.getScanInfo(), scan, targetCols); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/OOMERegionServer.java b/src/test/java/org/apache/hadoop/hbase/regionserver/OOMERegionServer.java index cac29897efcf..636caf8b1da1 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/OOMERegionServer.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/OOMERegionServer.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java index 1640f296b0fc..6cbb2bc3f197 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java @@ -40,7 +40,6 @@ import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.util.Bytes; -import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; import org.apache.hadoop.hbase.util.EnvironmentEdgeManagerTestHelper; import org.junit.experimental.categories.Category; @@ -55,11 +54,9 @@ public class TestAtomicOperation extends HBaseTestCase { HRegion region = null; private HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); - private final String DIR = TEST_UTIL.getDataTestDir("TestIncrement").toString(); + private final String DIR = TEST_UTIL.getDataTestDir("TestAtomicOperation").toString(); - private final int MAX_VERSIONS = 2; - // Test names static final byte[] tableName = Bytes.toBytes("testtable");; static final byte[] qual1 = Bytes.toBytes("qual1"); @@ -258,10 +255,10 @@ public void testRowMutationMultiThreads() throws IOException { LOG.info("Starting test testRowMutationMultiThreads"); initHRegion(tableName, getName(), fam1); - // create 100 threads, each will alternate between adding and + // create 10 threads, each will alternate between adding and // removing a column - int numThreads = 100; - int opsPerThread = 1000; + int numThreads = 10; + int opsPerThread = 500; AtomicOperation[] all = new AtomicOperation[numThreads]; AtomicLong timeStamps = new AtomicLong(0); @@ -275,9 +272,14 @@ public void run() { for (int i=0; i mrm = new ArrayList(); @@ -386,6 +393,7 @@ public void run() { RegionScanner rs = region.getScanner(s); List r = new ArrayList(); while(rs.next(r)); + rs.close(); if (r.size() != 1) { LOG.debug(r); failures.incrementAndGet(); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestBatchHRegionLockingAndWrites.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestBatchHRegionLockingAndWrites.java new file mode 100644 index 000000000000..b50e06ecc1c4 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestBatchHRegionLockingAndWrites.java @@ -0,0 +1,106 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import com.google.common.collect.Lists; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.client.Mutation; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.io.HeapSize; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.HashedBytes; +import org.apache.hadoop.hbase.util.Pair; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import java.io.IOException; +import java.util.List; + +import static org.junit.Assert.assertEquals; + + +@Category(SmallTests.class) +public class TestBatchHRegionLockingAndWrites { + private static final String FAMILY = "a"; + + @Test + @SuppressWarnings("unchecked") + public void testRedundantRowKeys() throws Exception { + + final int batchSize = 100000; + + String tableName = getClass().getSimpleName(); + Configuration conf = HBaseConfiguration.create(); + conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class); + MockHRegion region = (MockHRegion) TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf, Bytes.toBytes("a")); + + List> someBatch = Lists.newArrayList(); + int i = 0; + while (i < batchSize) { + if (i % 2 == 0) { + someBatch.add(new Pair(new Put(Bytes.toBytes(0)), null)); + } else { + someBatch.add(new Pair(new Put(Bytes.toBytes(1)), null)); + } + i++; + } + long start = System.nanoTime(); + region.batchMutate(someBatch.toArray(new Pair[0])); + long duration = System.nanoTime() - start; + System.out.println("Batch mutate took: " + duration + "ns"); + assertEquals(2, region.getAcquiredLockCount()); + } + + @Test + public void testGettingTheLockMatchesMyRow() throws Exception { + MockHRegion region = getMockHRegion(); + HashedBytes rowKey = new HashedBytes(Bytes.toBytes(1)); + assertEquals(Integer.valueOf(2), region.getLock(null, rowKey, false)); + assertEquals(Integer.valueOf(2), region.getLock(2, rowKey, false)); + } + + private MockHRegion getMockHRegion() throws IOException { + String tableName = getClass().getSimpleName(); + Configuration conf = HBaseConfiguration.create(); + conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class); + return (MockHRegion) TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf, Bytes.toBytes(FAMILY)); + } + + private static class MockHRegion extends HRegion { + private int acqioredLockCount = 0; + + public MockHRegion(Path tableDir, HLog log, FileSystem fs, Configuration conf, final HRegionInfo regionInfo, final HTableDescriptor htd, RegionServerServices rsServices) { + super(tableDir, log, fs, conf, regionInfo, htd, rsServices); + } + + private int getAcquiredLockCount() { + return acqioredLockCount; + } + + @Override + public Integer getLock(Integer lockid, HashedBytes row, boolean waitForLock) throws IOException { + acqioredLockCount++; + return super.getLock(lockid, row, waitForLock); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java index bd83a7116b4f..8eed964a4175 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -85,7 +84,16 @@ protected void tearDown() throws Exception { EnvironmentEdgeManagerTestHelper.reset(); } - private void initHRegion(byte[] tableName, String callingMethod, + /** + * Callers must afterward call {@link HRegion#closeHRegion(HRegion)} + * @param tableName + * @param callingMethod + * @param conf + * @param families + * @throws IOException + * @return created and initialized region. + */ + private HRegion initHRegion(byte[] tableName, String callingMethod, HBaseConfiguration conf, String family) throws IOException { HTableDescriptor htd = new HTableDescriptor(tableName); HColumnDescriptor familyDesc; @@ -99,8 +107,9 @@ private void initHRegion(byte[] tableName, String callingMethod, HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); Path path = new Path(DIR + callingMethod); - region = HRegion.createHRegion(info, path, conf, htd); + HRegion r = HRegion.createHRegion(info, path, conf, htd); blockCache = new CacheConfig(conf).getBlockCache(); + return r; } private void putData(String family, String row, String col, long version) @@ -212,45 +221,50 @@ public void testBlocksRead() throws Exception { String FAMILY = "cf1"; KeyValue kvs[]; HBaseConfiguration conf = getConf(); - initHRegion(TABLE, getName(), conf, FAMILY); - - putData(FAMILY, "row", "col1", 1); - putData(FAMILY, "row", "col2", 2); - putData(FAMILY, "row", "col3", 3); - putData(FAMILY, "row", "col4", 4); - putData(FAMILY, "row", "col5", 5); - putData(FAMILY, "row", "col6", 6); - putData(FAMILY, "row", "col7", 7); - region.flushcache(); - - // Expected block reads: 1 - // The top block has the KV we are - // interested. So only 1 seek is needed. - kvs = getData(FAMILY, "row", "col1", 1); - assertEquals(1, kvs.length); - verifyData(kvs[0], "row", "col1", 1); - - // Expected block reads: 2 - // The top block and next block has the KVs we are - // interested. So only 2 seek is needed. - kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2"), 2); - assertEquals(2, kvs.length); - verifyData(kvs[0], "row", "col1", 1); - verifyData(kvs[1], "row", "col2", 2); - - // Expected block reads: 3 - // The first 2 seeks is to find out col2. [HBASE-4443] - // One additional seek for col3 - // So 3 seeks are needed. - kvs = getData(FAMILY, "row", Arrays.asList("col2", "col3"), 3); - assertEquals(2, kvs.length); - verifyData(kvs[0], "row", "col2", 2); - verifyData(kvs[1], "row", "col3", 3); - - // Expected block reads: 2. [HBASE-4443] - kvs = getData(FAMILY, "row", Arrays.asList("col5"), 2); - assertEquals(1, kvs.length); - verifyData(kvs[0], "row", "col5", 5); + this.region = initHRegion(TABLE, getName(), conf, FAMILY); + + try { + putData(FAMILY, "row", "col1", 1); + putData(FAMILY, "row", "col2", 2); + putData(FAMILY, "row", "col3", 3); + putData(FAMILY, "row", "col4", 4); + putData(FAMILY, "row", "col5", 5); + putData(FAMILY, "row", "col6", 6); + putData(FAMILY, "row", "col7", 7); + region.flushcache(); + + // Expected block reads: 1 + // The top block has the KV we are + // interested. So only 1 seek is needed. + kvs = getData(FAMILY, "row", "col1", 1); + assertEquals(1, kvs.length); + verifyData(kvs[0], "row", "col1", 1); + + // Expected block reads: 2 + // The top block and next block has the KVs we are + // interested. So only 2 seek is needed. + kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2"), 2); + assertEquals(2, kvs.length); + verifyData(kvs[0], "row", "col1", 1); + verifyData(kvs[1], "row", "col2", 2); + + // Expected block reads: 3 + // The first 2 seeks is to find out col2. [HBASE-4443] + // One additional seek for col3 + // So 3 seeks are needed. + kvs = getData(FAMILY, "row", Arrays.asList("col2", "col3"), 3); + assertEquals(2, kvs.length); + verifyData(kvs[0], "row", "col2", 2); + verifyData(kvs[1], "row", "col3", 3); + + // Expected block reads: 2. [HBASE-4443] + kvs = getData(FAMILY, "row", Arrays.asList("col5"), 2); + assertEquals(1, kvs.length); + verifyData(kvs[0], "row", "col5", 5); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } /** @@ -264,97 +278,102 @@ public void testLazySeekBlocksRead() throws Exception { String FAMILY = "cf1"; KeyValue kvs[]; HBaseConfiguration conf = getConf(); - initHRegion(TABLE, getName(), conf, FAMILY); - - // File 1 - putData(FAMILY, "row", "col1", 1); - putData(FAMILY, "row", "col2", 2); - region.flushcache(); - - // File 2 - putData(FAMILY, "row", "col1", 3); - putData(FAMILY, "row", "col2", 4); - region.flushcache(); - - // Expected blocks read: 1. - // File 2's top block is also the KV we are - // interested. So only 1 seek is needed. - kvs = getData(FAMILY, "row", Arrays.asList("col1"), 1); - assertEquals(1, kvs.length); - verifyData(kvs[0], "row", "col1", 3); - - // Expected blocks read: 2 - // File 2's top block has the "col1" KV we are - // interested. We also need "col2" which is in a block - // of its own. So, we need that block as well. - kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2"), 2); - assertEquals(2, kvs.length); - verifyData(kvs[0], "row", "col1", 3); - verifyData(kvs[1], "row", "col2", 4); - - // File 3: Add another column - putData(FAMILY, "row", "col3", 5); - region.flushcache(); - - // Expected blocks read: 1 - // File 3's top block has the "col3" KV we are - // interested. So only 1 seek is needed. - kvs = getData(FAMILY, "row", "col3", 1); - assertEquals(1, kvs.length); - verifyData(kvs[0], "row", "col3", 5); - - // Get a column from older file. - // For ROWCOL Bloom filter: Expected blocks read: 1. - // For ROW Bloom filter: Expected blocks read: 2. - // For NONE Bloom filter: Expected blocks read: 2. - kvs = getData(FAMILY, "row", Arrays.asList("col1"), 1, 2, 2); - assertEquals(1, kvs.length); - verifyData(kvs[0], "row", "col1", 3); - - // File 4: Delete the entire row. - deleteFamily(FAMILY, "row", 6); - region.flushcache(); - - // For ROWCOL Bloom filter: Expected blocks read: 2. - // For ROW Bloom filter: Expected blocks read: 3. - // For NONE Bloom filter: Expected blocks read: 3. - kvs = getData(FAMILY, "row", "col1", 2, 3, 3); - assertEquals(0, kvs.length); - kvs = getData(FAMILY, "row", "col2", 3, 4, 4); - assertEquals(0, kvs.length); - kvs = getData(FAMILY, "row", "col3", 2); - assertEquals(0, kvs.length); - kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2", "col3"), 4); - assertEquals(0, kvs.length); - - // File 5: Delete - deleteFamily(FAMILY, "row", 10); - region.flushcache(); - - // File 6: some more puts, but with timestamps older than the - // previous delete. - putData(FAMILY, "row", "col1", 7); - putData(FAMILY, "row", "col2", 8); - putData(FAMILY, "row", "col3", 9); - region.flushcache(); - - // Baseline expected blocks read: 8. [HBASE-4532] - kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2", "col3"), 5); - assertEquals(0, kvs.length); - - // File 7: Put back new data - putData(FAMILY, "row", "col1", 11); - putData(FAMILY, "row", "col2", 12); - putData(FAMILY, "row", "col3", 13); - region.flushcache(); - - - // Expected blocks read: 5. [HBASE-4585] - kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2", "col3"), 5); - assertEquals(3, kvs.length); - verifyData(kvs[0], "row", "col1", 11); - verifyData(kvs[1], "row", "col2", 12); - verifyData(kvs[2], "row", "col3", 13); + this.region = initHRegion(TABLE, getName(), conf, FAMILY); + + try { + // File 1 + putData(FAMILY, "row", "col1", 1); + putData(FAMILY, "row", "col2", 2); + region.flushcache(); + + // File 2 + putData(FAMILY, "row", "col1", 3); + putData(FAMILY, "row", "col2", 4); + region.flushcache(); + + // Expected blocks read: 1. + // File 2's top block is also the KV we are + // interested. So only 1 seek is needed. + kvs = getData(FAMILY, "row", Arrays.asList("col1"), 1); + assertEquals(1, kvs.length); + verifyData(kvs[0], "row", "col1", 3); + + // Expected blocks read: 2 + // File 2's top block has the "col1" KV we are + // interested. We also need "col2" which is in a block + // of its own. So, we need that block as well. + kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2"), 2); + assertEquals(2, kvs.length); + verifyData(kvs[0], "row", "col1", 3); + verifyData(kvs[1], "row", "col2", 4); + + // File 3: Add another column + putData(FAMILY, "row", "col3", 5); + region.flushcache(); + + // Expected blocks read: 1 + // File 3's top block has the "col3" KV we are + // interested. So only 1 seek is needed. + kvs = getData(FAMILY, "row", "col3", 1); + assertEquals(1, kvs.length); + verifyData(kvs[0], "row", "col3", 5); + + // Get a column from older file. + // For ROWCOL Bloom filter: Expected blocks read: 1. + // For ROW Bloom filter: Expected blocks read: 2. + // For NONE Bloom filter: Expected blocks read: 2. + kvs = getData(FAMILY, "row", Arrays.asList("col1"), 1, 2, 2); + assertEquals(1, kvs.length); + verifyData(kvs[0], "row", "col1", 3); + + // File 4: Delete the entire row. + deleteFamily(FAMILY, "row", 6); + region.flushcache(); + + // For ROWCOL Bloom filter: Expected blocks read: 2. + // For ROW Bloom filter: Expected blocks read: 3. + // For NONE Bloom filter: Expected blocks read: 3. + kvs = getData(FAMILY, "row", "col1", 2, 3, 3); + assertEquals(0, kvs.length); + kvs = getData(FAMILY, "row", "col2", 3, 4, 4); + assertEquals(0, kvs.length); + kvs = getData(FAMILY, "row", "col3", 2); + assertEquals(0, kvs.length); + kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2", "col3"), 4); + assertEquals(0, kvs.length); + + // File 5: Delete + deleteFamily(FAMILY, "row", 10); + region.flushcache(); + + // File 6: some more puts, but with timestamps older than the + // previous delete. + putData(FAMILY, "row", "col1", 7); + putData(FAMILY, "row", "col2", 8); + putData(FAMILY, "row", "col3", 9); + region.flushcache(); + + // Baseline expected blocks read: 8. [HBASE-4532] + kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2", "col3"), 5); + assertEquals(0, kvs.length); + + // File 7: Put back new data + putData(FAMILY, "row", "col1", 11); + putData(FAMILY, "row", "col2", 12); + putData(FAMILY, "row", "col3", 13); + region.flushcache(); + + + // Expected blocks read: 5. [HBASE-4585] + kvs = getData(FAMILY, "row", Arrays.asList("col1", "col2", "col3"), 5); + assertEquals(3, kvs.length); + verifyData(kvs[0], "row", "col1", 11); + verifyData(kvs[1], "row", "col2", 12); + verifyData(kvs[2], "row", "col3", 13); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } /** @@ -367,62 +386,71 @@ public void testBlocksStoredWhenCachingDisabled() throws Exception { String FAMILY = "cf1"; HBaseConfiguration conf = getConf(); - initHRegion(TABLE, getName(), conf, FAMILY); - - putData(FAMILY, "row", "col1", 1); - putData(FAMILY, "row", "col2", 2); - region.flushcache(); - - // Execute a scan with caching turned off - // Expected blocks stored: 0 - long blocksStart = getBlkCount(); - Scan scan = new Scan(); - scan.setCacheBlocks(false); - RegionScanner rs = region.getScanner(scan); - List result = new ArrayList(2); - rs.next(result); - assertEquals(2 * BLOOM_TYPE.length, result.size()); - rs.close(); - long blocksEnd = getBlkCount(); - - assertEquals(blocksStart, blocksEnd); - - // Execute with caching turned on - // Expected blocks stored: 2 - blocksStart = blocksEnd; - scan.setCacheBlocks(true); - rs = region.getScanner(scan); - result = new ArrayList(2); - rs.next(result); - assertEquals(2 * BLOOM_TYPE.length, result.size()); - rs.close(); - blocksEnd = getBlkCount(); + this.region = initHRegion(TABLE, getName(), conf, FAMILY); + + try { + putData(FAMILY, "row", "col1", 1); + putData(FAMILY, "row", "col2", 2); + region.flushcache(); + + // Execute a scan with caching turned off + // Expected blocks stored: 0 + long blocksStart = getBlkCount(); + Scan scan = new Scan(); + scan.setCacheBlocks(false); + RegionScanner rs = region.getScanner(scan); + List result = new ArrayList(2); + rs.next(result); + assertEquals(2 * BLOOM_TYPE.length, result.size()); + rs.close(); + long blocksEnd = getBlkCount(); + + assertEquals(blocksStart, blocksEnd); + + // Execute with caching turned on + // Expected blocks stored: 2 + blocksStart = blocksEnd; + scan.setCacheBlocks(true); + rs = region.getScanner(scan); + result = new ArrayList(2); + rs.next(result); + assertEquals(2 * BLOOM_TYPE.length, result.size()); + rs.close(); + blocksEnd = getBlkCount(); - assertEquals(2 * BLOOM_TYPE.length, blocksEnd - blocksStart); - } + assertEquals(2 * BLOOM_TYPE.length, blocksEnd - blocksStart); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } + } - @Test + @Test public void testLazySeekBlocksReadWithDelete() throws Exception { byte[] TABLE = Bytes.toBytes("testLazySeekBlocksReadWithDelete"); String FAMILY = "cf1"; KeyValue kvs[]; HBaseConfiguration conf = getConf(); - initHRegion(TABLE, getName(), conf, FAMILY); - - deleteFamily(FAMILY, "row", 200); - for (int i = 0; i < 100; i++) { - putData(FAMILY, "row", "col" + i, i); + this.region = initHRegion(TABLE, getName(), conf, FAMILY); + try { + deleteFamily(FAMILY, "row", 200); + for (int i = 0; i < 100; i++) { + putData(FAMILY, "row", "col" + i, i); + } + putData(FAMILY, "row", "col99", 201); + region.flushcache(); + + kvs = getData(FAMILY, "row", Arrays.asList("col0"), 2); + assertEquals(0, kvs.length); + + kvs = getData(FAMILY, "row", Arrays.asList("col99"), 2); + assertEquals(1, kvs.length); + verifyData(kvs[0], "row", "col99", 201); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } - putData(FAMILY, "row", "col99", 201); - region.flushcache(); - - kvs = getData(FAMILY, "row", Arrays.asList("col0"), 2); - assertEquals(0, kvs.length); - - kvs = getData(FAMILY, "row", Arrays.asList("col99"), 2); - assertEquals(1, kvs.length); - verifyData(kvs[0], "row", "col99", 201); - } + } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java new file mode 100644 index 000000000000..932a8fbe9ff5 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java @@ -0,0 +1,137 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +import org.apache.hadoop.hbase.HBaseTestCase; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; +import org.apache.hadoop.hbase.io.hfile.Compression; +import org.apache.hadoop.hbase.io.hfile.BlockType.BlockCategory; +import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; +import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.BlockMetricType; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.Assert; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@SuppressWarnings("deprecation") +@Category(SmallTests.class) +public class TestBlocksScanned extends HBaseTestCase { + private static byte [] FAMILY = Bytes.toBytes("family"); + private static byte [] COL = Bytes.toBytes("col"); + private static byte [] START_KEY = Bytes.toBytes("aaa"); + private static byte [] END_KEY = Bytes.toBytes("zzz"); + private static int BLOCK_SIZE = 70; + + private static HBaseTestingUtility TEST_UTIL = null; + + @Override + public void setUp() throws Exception { + super.setUp(); + SchemaMetrics.setUseTableNameInTest(true); + TEST_UTIL = new HBaseTestingUtility(); + } + + @Test + public void testBlocksScanned() throws Exception { + byte [] tableName = Bytes.toBytes("TestBlocksScanned"); + HTableDescriptor table = new HTableDescriptor(tableName); + + table.addFamily( + new HColumnDescriptor(FAMILY) + .setMaxVersions(10) + .setBlockCacheEnabled(true) + .setBlocksize(BLOCK_SIZE) + .setCompressionType(Compression.Algorithm.NONE) + ); + _testBlocksScanned(table); + } + + @Test + public void testBlocksScannedWithEncoding() throws Exception { + byte [] tableName = Bytes.toBytes("TestBlocksScannedWithEncoding"); + HTableDescriptor table = new HTableDescriptor(tableName); + + table.addFamily( + new HColumnDescriptor(FAMILY) + .setMaxVersions(10) + .setBlockCacheEnabled(true) + .setDataBlockEncoding(DataBlockEncoding.FAST_DIFF) + .setBlocksize(BLOCK_SIZE) + .setCompressionType(Compression.Algorithm.NONE) + ); + _testBlocksScanned(table); + } + + private void _testBlocksScanned(HTableDescriptor table) throws Exception { + HRegion r = createNewHRegion(table, START_KEY, END_KEY, + TEST_UTIL.getConfiguration()); + addContent(r, FAMILY, COL); + r.flushcache(); + + // Get the per-cf metrics + SchemaMetrics schemaMetrics = + SchemaMetrics.getInstance(Bytes.toString(table.getName()), Bytes.toString(FAMILY)); + Map schemaMetricSnapshot = SchemaMetrics.getMetricsSnapshot(); + + // Do simple test of getting one row only first. + Scan scan = new Scan(Bytes.toBytes("aaa"), Bytes.toBytes("aaz")); + scan.addColumn(FAMILY, COL); + scan.setMaxVersions(1); + + InternalScanner s = r.getScanner(scan); + List results = new ArrayList(); + while (s.next(results)); + s.close(); + + int expectResultSize = 'z' - 'a'; + Assert.assertEquals(expectResultSize, results.size()); + + int kvPerBlock = (int) Math.ceil(BLOCK_SIZE / (double) results.get(0).getLength()); + Assert.assertEquals(2, kvPerBlock); + + long expectDataBlockRead = (long) Math.ceil(expectResultSize / (double) kvPerBlock); + long expectIndexBlockRead = expectDataBlockRead; + + verifyDataAndIndexBlockRead(schemaMetricSnapshot, schemaMetrics, + expectDataBlockRead, expectIndexBlockRead); + } + + private void verifyDataAndIndexBlockRead(Map previousMetricSnapshot, + SchemaMetrics schemaMetrics, long expectDataBlockRead, long expectedIndexBlockRead){ + Map currentMetricsSnapshot = SchemaMetrics.getMetricsSnapshot(); + Map diffs = + SchemaMetrics.diffMetrics(previousMetricSnapshot, currentMetricsSnapshot); + + long dataBlockRead = SchemaMetrics.getLong(diffs, + schemaMetrics.getBlockMetricName(BlockCategory.DATA, false, BlockMetricType.READ_COUNT)); + long indexBlockRead = SchemaMetrics.getLong(diffs, + schemaMetrics.getBlockMetricName(BlockCategory.INDEX, false, BlockMetricType.READ_COUNT)); + + Assert.assertEquals(expectDataBlockRead, dataBlockRead); + Assert.assertEquals(expectedIndexBlockRead, indexBlockRead); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java new file mode 100644 index 000000000000..2ce15f78c26a --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java @@ -0,0 +1,270 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.regionserver; + +import static org.junit.Assert.*; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.Random; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.fs.HFileSystem; +import org.apache.hadoop.hbase.io.hfile.BlockCache; +import org.apache.hadoop.hbase.io.hfile.BlockCacheKey; +import org.apache.hadoop.hbase.io.hfile.BlockType; +import org.apache.hadoop.hbase.io.hfile.CacheConfig; +import org.apache.hadoop.hbase.io.hfile.Compression; +import org.apache.hadoop.hbase.io.hfile.HFile; +import org.apache.hadoop.hbase.io.hfile.HFileBlock; +import org.apache.hadoop.hbase.io.hfile.HFileReaderV2; +import org.apache.hadoop.hbase.io.hfile.HFileScanner; +import org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2; +import org.apache.hadoop.hbase.regionserver.StoreFile.BloomType; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.util.Bytes; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.junit.runners.Parameterized.Parameters; + +/** + * Tests {@link HFile} cache-on-write functionality for data blocks, non-root + * index blocks, and Bloom filter blocks, as specified by the column family. + */ +@RunWith(Parameterized.class) +@Category(MediumTests.class) +public class TestCacheOnWriteInSchema { + + private static final Log LOG = LogFactory.getLog(TestCacheOnWriteInSchema.class); + + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final String DIR = TEST_UTIL.getDataTestDir("TestCacheOnWriteInSchema").toString(); + private static final byte [] table = Bytes.toBytes("table"); + private static byte [] family = Bytes.toBytes("family"); + private static final int NUM_KV = 25000; + private static final Random rand = new Random(12983177L); + /** The number of valid key types possible in a store file */ + private static final int NUM_VALID_KEY_TYPES = + KeyValue.Type.values().length - 2; + + private static enum CacheOnWriteType { + DATA_BLOCKS(BlockType.DATA, BlockType.ENCODED_DATA), + BLOOM_BLOCKS(BlockType.BLOOM_CHUNK), + INDEX_BLOCKS(BlockType.LEAF_INDEX, BlockType.INTERMEDIATE_INDEX); + + private final BlockType blockType1; + private final BlockType blockType2; + + private CacheOnWriteType(BlockType blockType) { + this(blockType, blockType); + } + + private CacheOnWriteType(BlockType blockType1, BlockType blockType2) { + this.blockType1 = blockType1; + this.blockType2 = blockType2; + } + + public boolean shouldBeCached(BlockType blockType) { + return blockType == blockType1 || blockType == blockType2; + } + + public void modifyFamilySchema(HColumnDescriptor family) { + switch (this) { + case DATA_BLOCKS: + family.setCacheDataOnWrite(true); + break; + case BLOOM_BLOCKS: + family.setCacheBloomsOnWrite(true); + break; + case INDEX_BLOCKS: + family.setCacheIndexesOnWrite(true); + break; + } + } + } + + private final CacheOnWriteType cowType; + private Configuration conf; + private final String testDescription; + private Store store; + private FileSystem fs; + + public TestCacheOnWriteInSchema(CacheOnWriteType cowType) { + this.cowType = cowType; + testDescription = "[cacheOnWrite=" + cowType + "]"; + System.out.println(testDescription); + } + + @Parameters + public static Collection getParameters() { + List cowTypes = new ArrayList(); + for (CacheOnWriteType cowType : CacheOnWriteType.values()) { + cowTypes.add(new Object[] { cowType }); + } + return cowTypes; + } + + @Before + public void setUp() throws IOException { + conf = TEST_UTIL.getConfiguration(); + conf.setInt(HFile.FORMAT_VERSION_KEY, HFile.MAX_FORMAT_VERSION); + conf.setBoolean(CacheConfig.CACHE_BLOCKS_ON_WRITE_KEY, false); + conf.setBoolean(CacheConfig.CACHE_INDEX_BLOCKS_ON_WRITE_KEY, false); + conf.setBoolean(CacheConfig.CACHE_BLOOM_BLOCKS_ON_WRITE_KEY, false); + + fs = HFileSystem.get(conf); + + // Create the schema + HColumnDescriptor hcd = new HColumnDescriptor(family); + hcd.setBloomFilterType(BloomType.ROWCOL); + cowType.modifyFamilySchema(hcd); + HTableDescriptor htd = new HTableDescriptor(table); + htd.addFamily(hcd); + + // Create a store based on the schema + Path basedir = new Path(DIR); + Path logdir = new Path(DIR+"/logs"); + Path oldLogDir = new Path(basedir, HConstants.HREGION_OLDLOGDIR_NAME); + fs.delete(logdir, true); + HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); + HLog hlog = new HLog(fs, logdir, oldLogDir, conf); + HRegion region = new HRegion(basedir, hlog, fs, conf, info, htd, null); + store = new Store(basedir, region, hcd, fs, conf); + } + + @After + public void tearDown() { + try { + fs.delete(new Path(DIR), true); + } catch (IOException e) { + LOG.error("Could not delete " + DIR, e); + } + } + + @Test + public void testCacheOnWriteInSchema() throws IOException { + // Write some random data into the store + StoreFile.Writer writer = store.createWriterInTmp(Integer.MAX_VALUE, + Compression.Algorithm.NONE, false, true); + writeStoreFile(writer); + writer.close(); + // Verify the block types of interest were cached on write + readStoreFile(writer.getPath()); + } + + private void readStoreFile(Path path) throws IOException { + CacheConfig cacheConf = store.getCacheConfig(); + BlockCache cache = cacheConf.getBlockCache(); + StoreFile sf = new StoreFile(fs, path, conf, cacheConf, + BloomType.ROWCOL, null); + store.passSchemaMetricsTo(sf); + HFileReaderV2 reader = (HFileReaderV2) sf.createReader().getHFileReader(); + try { + // Open a scanner with (on read) caching disabled + HFileScanner scanner = reader.getScanner(false, false); + assertTrue(testDescription, scanner.seekTo()); + // Cribbed from io.hfile.TestCacheOnWrite + long offset = 0; + HFileBlock prevBlock = null; + while (offset < reader.getTrailer().getLoadOnOpenDataOffset()) { + long onDiskSize = -1; + if (prevBlock != null) { + onDiskSize = prevBlock.getNextBlockOnDiskSizeWithHeader(); + } + // Flags: don't cache the block, use pread, this is not a compaction. + // Also, pass null for expected block type to avoid checking it. + HFileBlock block = reader.readBlock(offset, onDiskSize, false, true, + false, null); + BlockCacheKey blockCacheKey = new BlockCacheKey(reader.getName(), + offset); + boolean isCached = cache.getBlock(blockCacheKey, true, false) != null; + boolean shouldBeCached = cowType.shouldBeCached(block.getBlockType()); + if (shouldBeCached != isCached) { + throw new AssertionError( + "shouldBeCached: " + shouldBeCached+ "\n" + + "isCached: " + isCached + "\n" + + "Test description: " + testDescription + "\n" + + "block: " + block + "\n" + + "blockCacheKey: " + blockCacheKey); + } + prevBlock = block; + offset += block.getOnDiskSizeWithHeader(); + } + } finally { + reader.close(); + } + } + + private static KeyValue.Type generateKeyType(Random rand) { + if (rand.nextBoolean()) { + // Let's make half of KVs puts. + return KeyValue.Type.Put; + } else { + KeyValue.Type keyType = + KeyValue.Type.values()[1 + rand.nextInt(NUM_VALID_KEY_TYPES)]; + if (keyType == KeyValue.Type.Minimum || keyType == KeyValue.Type.Maximum) + { + throw new RuntimeException("Generated an invalid key type: " + keyType + + ". " + "Probably the layout of KeyValue.Type has changed."); + } + return keyType; + } + } + + private void writeStoreFile(StoreFile.Writer writer) throws IOException { + final int rowLen = 32; + for (int i = 0; i < NUM_KV; ++i) { + byte[] k = TestHFileWriterV2.randomOrderedKey(rand, i); + byte[] v = TestHFileWriterV2.randomValue(rand); + int cfLen = rand.nextInt(k.length - rowLen + 1); + KeyValue kv = new KeyValue( + k, 0, rowLen, + k, rowLen, cfLen, + k, rowLen + cfLen, k.length - rowLen - cfLen, + rand.nextLong(), + generateKeyType(rand), + v, 0, v.length); + writer.append(kv); + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} + diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java index 84296ab6f36b..0eec002bdcd5 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -62,95 +61,99 @@ public void testDuplicateVersions() throws IOException { HRegion region = HRegion.createHRegion(info, TEST_UTIL.getDataTestDir(), TEST_UTIL .getConfiguration(), htd); - - List rows = generateRandomWords(10, "row"); - List allColumns = generateRandomWords(10, "column"); - List values = generateRandomWords(100, "value"); - - long maxTimestamp = 2; - double selectPercent = 0.5; - int numberOfTests = 5; - double flushPercentage = 0.2; - double minorPercentage = 0.2; - double majorPercentage = 0.2; - double putPercentage = 0.2; - - HashMap allKVMap = new HashMap(); - - HashMap[] kvMaps = new HashMap[numberOfTests]; - ArrayList[] columnLists = new ArrayList[numberOfTests]; - - for (int i = 0; i < numberOfTests; i++) { - kvMaps[i] = new HashMap(); - columnLists[i] = new ArrayList(); - for (String column : allColumns) { - if (Math.random() < selectPercent) { - columnLists[i].add(column); + try { + List rows = generateRandomWords(10, "row"); + List allColumns = generateRandomWords(10, "column"); + List values = generateRandomWords(100, "value"); + + long maxTimestamp = 2; + double selectPercent = 0.5; + int numberOfTests = 5; + double flushPercentage = 0.2; + double minorPercentage = 0.2; + double majorPercentage = 0.2; + double putPercentage = 0.2; + + HashMap allKVMap = new HashMap(); + + HashMap[] kvMaps = new HashMap[numberOfTests]; + ArrayList[] columnLists = new ArrayList[numberOfTests]; + + for (int i = 0; i < numberOfTests; i++) { + kvMaps[i] = new HashMap(); + columnLists[i] = new ArrayList(); + for (String column : allColumns) { + if (Math.random() < selectPercent) { + columnLists[i].add(column); + } } } - } - for (String value : values) { - for (String row : rows) { - Put p = new Put(Bytes.toBytes(row)); - p.setWriteToWAL(false); - for (String column : allColumns) { - for (long timestamp = 1; timestamp <= maxTimestamp; timestamp++) { - KeyValue kv = - KeyValueTestUtil.create(row, family, column, timestamp, value); - if (Math.random() < putPercentage) { - p.add(kv); - allKVMap.put(kv.getKeyString(), kv); - for (int i = 0; i < numberOfTests; i++) { - if (columnLists[i].contains(column)) { - kvMaps[i].put(kv.getKeyString(), kv); + for (String value : values) { + for (String row : rows) { + Put p = new Put(Bytes.toBytes(row)); + p.setWriteToWAL(false); + for (String column : allColumns) { + for (long timestamp = 1; timestamp <= maxTimestamp; timestamp++) { + KeyValue kv = + KeyValueTestUtil.create(row, family, column, timestamp, value); + if (Math.random() < putPercentage) { + p.add(kv); + allKVMap.put(kv.getKeyString(), kv); + for (int i = 0; i < numberOfTests; i++) { + if (columnLists[i].contains(column)) { + kvMaps[i].put(kv.getKeyString(), kv); + } } } } } - } - region.put(p); - if (Math.random() < flushPercentage) { - LOG.info("Flushing... "); - region.flushcache(); - } + region.put(p); + if (Math.random() < flushPercentage) { + LOG.info("Flushing... "); + region.flushcache(); + } - if (Math.random() < minorPercentage) { - LOG.info("Minor compacting... "); - region.compactStores(false); - } + if (Math.random() < minorPercentage) { + LOG.info("Minor compacting... "); + region.compactStores(false); + } - if (Math.random() < majorPercentage) { - LOG.info("Major compacting... "); - region.compactStores(true); + if (Math.random() < majorPercentage) { + LOG.info("Major compacting... "); + region.compactStores(true); + } } } - } - for (int i = 0; i < numberOfTests + 1; i++) { - Collection kvSet; - Scan scan = new Scan(); - scan.setMaxVersions(); - if (i < numberOfTests) { - kvSet = kvMaps[i].values(); - for (String column : columnLists[i]) { - scan.addColumn(familyBytes, Bytes.toBytes(column)); - } - LOG.info("ExplicitColumns scanner"); - LOG.info("Columns: " + columnLists[i].size() + " Keys: " - + kvSet.size()); - } else { - kvSet = allKVMap.values(); - LOG.info("Wildcard scanner"); - LOG.info("Columns: " + allColumns.size() + " Keys: " + kvSet.size()); + for (int i = 0; i < numberOfTests + 1; i++) { + Collection kvSet; + Scan scan = new Scan(); + scan.setMaxVersions(); + if (i < numberOfTests) { + if (columnLists[i].size() == 0) continue; // HBASE-7700 + kvSet = kvMaps[i].values(); + for (String column : columnLists[i]) { + scan.addColumn(familyBytes, Bytes.toBytes(column)); + } + LOG.info("ExplicitColumns scanner"); + LOG.info("Columns: " + columnLists[i].size() + " Keys: " + + kvSet.size()); + } else { + kvSet = allKVMap.values(); + LOG.info("Wildcard scanner"); + LOG.info("Columns: " + allColumns.size() + " Keys: " + kvSet.size()); + } + InternalScanner scanner = region.getScanner(scan); + List results = new ArrayList(); + while (scanner.next(results)) + ; + assertEquals(kvSet.size(), results.size()); + assertTrue(results.containsAll(kvSet)); } - InternalScanner scanner = region.getScanner(scan); - List results = new ArrayList(); - while (scanner.next(results)) - ; - assertEquals(kvSet.size(), results.size()); - assertTrue(results.containsAll(kvSet)); + } finally { + HRegion.closeHRegion(region); } region.close(); @@ -241,6 +244,7 @@ public void testReseeking() throws IOException { Scan scan = new Scan(); scan.setMaxVersions(); if (i < numberOfTests) { + if (columnLists[i].size() == 0) continue; // HBASE-7700 kvSet = kvMaps[i].values(); for (String column : columnLists[i]) { scan.addColumn(familyBytes, Bytes.toBytes(column)); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java index 80605e755f21..b6dac3e76b33 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -33,15 +32,20 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.SmallTests; import org.apache.hadoop.hbase.io.hfile.CacheConfig; import org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder; import org.apache.hadoop.hbase.regionserver.compactions.CompactSelection; import org.apache.hadoop.hbase.regionserver.wal.HLog; import org.apache.hadoop.hbase.util.Bytes; +import org.junit.experimental.categories.Category; import com.google.common.collect.Lists; -import org.junit.experimental.categories.Category; @Category(SmallTests.class) public class TestCompactSelection extends TestCase { @@ -86,9 +90,10 @@ public void setUp() throws Exception { HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); HLog hlog = new HLog(fs, logdir, oldLogDir, conf); - HRegion.createHRegion(info, basedir, conf, htd); + HRegion region = HRegion.createHRegion(info, basedir, conf, htd); + HRegion.closeHRegion(region); Path tableDir = new Path(basedir, Bytes.toString(htd.getName())); - HRegion region = new HRegion(tableDir, hlog, fs, conf, info, htd, null); + region = new HRegion(tableDir, hlog, fs, conf, info, htd, null); store = new Store(basedir, region, hcd, fs, conf); TEST_FILE = StoreFile.getRandomFilename(fs, store.getHomedir()); @@ -99,6 +104,8 @@ public void setUp() throws Exception { static class MockStoreFile extends StoreFile { long length = 0; boolean isRef = false; + TimeRangeTracker timeRangeTracker; + long entryCount; MockStoreFile(long length, boolean isRef) throws IOException { super(TEST_UTIL.getTestFileSystem(), TEST_FILE, @@ -123,14 +130,35 @@ boolean isReference() { return this.isRef; } + void setTimeRangeTracker(TimeRangeTracker timeRangeTracker) { + this.timeRangeTracker = timeRangeTracker; + } + + void setEntries(long entryCount) { + this.entryCount = entryCount; + } + @Override public StoreFile.Reader getReader() { final long len = this.length; + final TimeRangeTracker timeRange = this.timeRangeTracker; + final long entries = this.entryCount; return new StoreFile.Reader() { @Override public long length() { return len; } + + @Override + public long getMaxTimestamp() { + return timeRange == null ? Long.MAX_VALUE + : timeRange.maximumTimestamp; + } + + @Override + public long getEntries() { + return entries; + } }; } } @@ -236,7 +264,7 @@ public void testCompactionRatio() throws IOException { assertEquals(maxFiles, store.compactSelection(sfCreate(true, 7,6,5,4,3,2,1)).getFilesToCompact().size()); // reference compaction - compactEquals(sfCreate(true, 7, 6, 5, 4, 3, 2, 1), 5, 4, 3, 2, 1); + compactEquals(sfCreate(true, 7, 6, 5, 4, 3, 2, 1), 7, 6, 5, 4, 3); // empty case compactEquals(new ArrayList() /* empty */); @@ -281,6 +309,23 @@ public void testOffPeakCompactionRatio() throws IOException { compactEquals(sfCreate(999,50,12,12, 1), 12, 12, 1); } + public void testCompactionEmptyHFile() throws IOException { + // Do not compact empty store file + List candidates = sfCreate(0); + for (StoreFile file : candidates) { + if (file instanceof MockStoreFile) { + MockStoreFile mockFile = (MockStoreFile) file; + mockFile.setTimeRangeTracker(new TimeRangeTracker(-1, -1)); + mockFile.setEntries(0); + } + } + // Test Default compactions + CompactSelection compactSelection = new CompactSelection(conf, candidates); + CompactSelection result = compactSelection + .selectExpiredStoreFilesToCompact(600L); + assertTrue(result == null); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java index 7d0261c9c8e9..daa2297fcb63 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -24,10 +23,12 @@ import java.io.IOException; import java.util.ArrayList; +import java.util.Collections; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Map.Entry; +import java.util.concurrent.CountDownLatch; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -51,9 +52,12 @@ import org.apache.hadoop.hbase.io.hfile.HFileDataBlockEncoder; import org.apache.hadoop.hbase.io.hfile.HFileScanner; import org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest; +import org.apache.hadoop.hbase.regionserver.metrics.RegionServerMetrics; import org.apache.hadoop.hbase.regionserver.wal.HLog; import org.apache.hadoop.hbase.util.Bytes; import org.junit.experimental.categories.Category; +import org.mockito.Mockito; import org.mockito.invocation.InvocationOnMock; import org.mockito.stubbing.Answer; @@ -76,6 +80,7 @@ public class TestCompaction extends HBaseTestCase { private int compactionThreshold; private byte[] firstRowBytes, secondRowBytes, thirdRowBytes; final private byte[] col1, col2; + private static final long MAX_FILES_TO_COMPACT = 10; /** constructor */ public TestCompaction() throws Exception { @@ -131,6 +136,7 @@ public void testMajorCompactingToNoOutput() throws IOException { r.delete(new Delete(results.get(0).getRow()), null, false); if (!result) break; } while(true); + s.close(); // Flush r.flushcache(); // Major compact. @@ -377,11 +383,10 @@ public void testMinorCompactionWithDeleteVersion2() throws Exception { deleteVersion.deleteColumn(fam2, col2, 1); /* * the table has 4 versions: 0, 1, 2, and 3. - * 0 does not count. * We delete 1. - * Should have 2 remaining. + * Should have 3 remaining. */ - testMinorCompactionWithDelete(deleteVersion, 2); + testMinorCompactionWithDelete(deleteVersion, 3); } /* @@ -564,8 +569,12 @@ private int count() throws IOException { } private void createStoreFile(final HRegion region) throws IOException { + createStoreFile(region, Bytes.toString(COLUMN_FAMILY)); + } + + private void createStoreFile(final HRegion region, String family) throws IOException { HRegionIncommon loader = new HRegionIncommon(region); - addContent(loader, Bytes.toString(COLUMN_FAMILY)); + addContent(loader, family); loader.flushcache(); } @@ -585,8 +594,10 @@ public void testCompactionWithCorruptResult() throws Exception { List storeFiles = store.getStorefiles(); long maxId = StoreFile.getMaxSequenceIdInList(storeFiles); + Compactor tool = new Compactor(this.conf); - StoreFile.Writer compactedFile = store.compactStore(storeFiles, false, maxId); + StoreFile.Writer compactedFile = tool.compactForTesting(store, this.conf, storeFiles, false, + maxId); // Now lets corrupt the compacted file. FileSystem fs = FileSystem.get(conf); @@ -612,6 +623,127 @@ public void testCompactionWithCorruptResult() throws Exception { fail("testCompactionWithCorruptResult failed since no exception was" + "thrown while completing a corrupt file"); } + + /** + * Test for HBASE-5920 - Test user requested major compactions always occurring + */ + public void testNonUserMajorCompactionRequest() throws Exception { + Store store = r.getStore(COLUMN_FAMILY); + createStoreFile(r); + for (int i = 0; i < MAX_FILES_TO_COMPACT + 1; i++) { + createStoreFile(r); + } + store.triggerMajorCompaction(); + + CompactionRequest request = store.requestCompaction(Store.NO_PRIORITY, null); + assertNotNull("Expected to receive a compaction request", request); + assertEquals( + "System-requested major compaction should not occur if there are too many store files", + false, + request.isMajor()); + } + + /** + * Test for HBASE-5920 + */ + public void testUserMajorCompactionRequest() throws IOException{ + Store store = r.getStore(COLUMN_FAMILY); + createStoreFile(r); + for (int i = 0; i < MAX_FILES_TO_COMPACT + 1; i++) { + createStoreFile(r); + } + store.triggerMajorCompaction(); + CompactionRequest request = store.requestCompaction(Store.PRIORITY_USER, null); + assertNotNull("Expected to receive a compaction request", request); + assertEquals( + "User-requested major compaction should always occur, even if there are too many store files", + true, + request.isMajor()); + } + + /** + * Create a custom compaction request and be sure that we can track it through the queue, knowing + * when the compaction is completed. + */ + public void testTrackingCompactionRequest() throws Exception { + // setup a compact/split thread on a mock server + HRegionServer mockServer = Mockito.mock(HRegionServer.class); + Mockito.when(mockServer.getConfiguration()).thenReturn(r.getConf()); + CompactSplitThread thread = new CompactSplitThread(mockServer); + Mockito.when(mockServer.getCompactSplitThread()).thenReturn(thread); + // simple stop for the metrics - we ignore any updates in the test + RegionServerMetrics mockMetrics = Mockito.mock(RegionServerMetrics.class); + Mockito.when(mockServer.getMetrics()).thenReturn(mockMetrics); + + // setup a region/store with some files + Store store = r.getStore(COLUMN_FAMILY); + createStoreFile(r); + for (int i = 0; i < MAX_FILES_TO_COMPACT + 1; i++) { + createStoreFile(r); + } + + CountDownLatch latch = new CountDownLatch(1); + TrackableCompactionRequest request = new TrackableCompactionRequest(r, store, latch); + thread.requestCompaction(r, store, "test custom comapction", Store.PRIORITY_USER, request); + // wait for the latch to complete. + latch.await(); + + thread.interruptIfNecessary(); + } + + public void testMultipleCustomCompactionRequests() throws Exception { + // setup a compact/split thread on a mock server + HRegionServer mockServer = Mockito.mock(HRegionServer.class); + Mockito.when(mockServer.getConfiguration()).thenReturn(r.getConf()); + CompactSplitThread thread = new CompactSplitThread(mockServer); + Mockito.when(mockServer.getCompactSplitThread()).thenReturn(thread); + // simple stop for the metrics - we ignore any updates in the test + RegionServerMetrics mockMetrics = Mockito.mock(RegionServerMetrics.class); + Mockito.when(mockServer.getMetrics()).thenReturn(mockMetrics); + + // setup a region/store with some files + int numStores = r.getStores().size(); + List requests = new ArrayList(numStores); + CountDownLatch latch = new CountDownLatch(numStores); + // create some store files and setup requests for each store on which we want to do a + // compaction + for (Store store : r.getStores().values()) { + createStoreFile(r, store.getColumnFamilyName()); + createStoreFile(r, store.getColumnFamilyName()); + createStoreFile(r, store.getColumnFamilyName()); + requests.add(new TrackableCompactionRequest(r, store, latch)); + } + + thread.requestCompaction(r, "test mulitple custom comapctions", Store.PRIORITY_USER, + Collections.unmodifiableList(requests)); + + // wait for the latch to complete. + latch.await(); + + thread.interruptIfNecessary(); + } + + /** + * Simple {@link CompactionRequest} on which you can wait until the requested compaction finishes. + */ + public static class TrackableCompactionRequest extends CompactionRequest { + private CountDownLatch done; + + /** + * Constructor for a custom compaction. Uses the setXXX methods to update the state of the + * compaction before being used. + */ + public TrackableCompactionRequest(HRegion region, Store store, CountDownLatch finished) { + super(region, store, Store.PRIORITY_USER); + this.done = finished; + } + + @Override + public void run() { + super.run(); + this.done.countDown(); + } + } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionState.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionState.java new file mode 100644 index 000000000000..96849b423a24 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionState.java @@ -0,0 +1,241 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.regionserver; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Random; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.CompactionState; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** Unit tests to test retrieving table/region compaction state*/ +@Category(LargeTests.class) +public class TestCompactionState { + final static Log LOG = LogFactory.getLog(TestCompactionState.class); + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private final static Random random = new Random(); + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.startMiniCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + @Test(timeout=600000) + public void testMajorCompaction() throws IOException, InterruptedException { + compaction("testMajorCompaction", 8, CompactionState.MAJOR, false); + } + + @Test(timeout=600000) + public void testMinorCompaction() throws IOException, InterruptedException { + compaction("testMinorCompaction", 15, CompactionState.MINOR, false); + } + + @Test(timeout=600000) + public void testMajorCompactionOnFamily() throws IOException, InterruptedException { + compaction("testMajorCompactionOnFamily", 8, CompactionState.MAJOR, true); + } + + @Test(timeout=600000) + public void testMinorCompactionOnFamily() throws IOException, InterruptedException { + compaction("testMinorCompactionOnFamily", 15, CompactionState.MINOR, true); + } + + @Test + public void testInvalidColumnFamily() throws IOException, InterruptedException { + byte [] table = Bytes.toBytes("testInvalidColumnFamily"); + byte [] family = Bytes.toBytes("family"); + byte [] fakecf = Bytes.toBytes("fakecf"); + boolean caughtMinorCompact = false; + boolean caughtMajorCompact = false; + HTable ht = null; + try { + ht = TEST_UTIL.createTable(table, family); + HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + try { + admin.compact(table, fakecf); + } catch (IOException ioe) { + caughtMinorCompact = true; + } + try { + admin.majorCompact(table, fakecf); + } catch (IOException ioe) { + caughtMajorCompact = true; + } + } finally { + if (ht != null) { + TEST_UTIL.deleteTable(table); + } + assertTrue(caughtMinorCompact); + assertTrue(caughtMajorCompact); + } + } + + /** + * Load data to a table, flush it to disk, trigger compaction, + * confirm the compaction state is right and wait till it is done. + * + * @param tableName + * @param flushes + * @param expectedState + * @param singleFamily otherwise, run compaction on all cfs + * @throws IOException + * @throws InterruptedException + */ + private void compaction(final String tableName, final int flushes, + final CompactionState expectedState, boolean singleFamily) + throws IOException, InterruptedException { + // Create a table with regions + byte [] table = Bytes.toBytes(tableName); + byte [] family = Bytes.toBytes("family"); + byte [][] families = + {family, Bytes.add(family, Bytes.toBytes("2")), Bytes.add(family, Bytes.toBytes("3"))}; + HTable ht = null; + try { + ht = TEST_UTIL.createTable(table, families); + loadData(ht, families, 3000, flushes); + HRegionServer rs = TEST_UTIL.getMiniHBaseCluster().getRegionServer(0); + List regions = rs.getOnlineRegions(table); + int countBefore = countStoreFilesInFamilies(regions, families); + int countBeforeSingleFamily = countStoreFilesInFamily(regions, family); + assertTrue(countBefore > 0); // there should be some data files + HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + if (expectedState == CompactionState.MINOR) { + if (singleFamily) { + admin.compact(table, family); + } else { + admin.compact(table); + } + } else { + if (singleFamily) { + admin.majorCompact(table, family); + } else { + admin.majorCompact(table); + } + } + long curt = System.currentTimeMillis(); + long waitTime = 5000; + long endt = curt + waitTime; + CompactionState state = admin.getCompactionState(table); + while (state == CompactionState.NONE && curt < endt) { + Thread.sleep(10); + state = admin.getCompactionState(table); + curt = System.currentTimeMillis(); + } + // Now, should have the right compaction state, + // otherwise, the compaction should have already been done + if (expectedState != state) { + for (HRegion region: regions) { + state = CompactionRequest.getCompactionState(region.getRegionId()); + assertEquals(CompactionState.NONE, state); + } + } else { + curt = System.currentTimeMillis(); + waitTime = 20000; + endt = curt + waitTime; + state = admin.getCompactionState(table); + while (state != CompactionState.NONE && curt < endt) { + Thread.sleep(10); + state = admin.getCompactionState(table); + curt = System.currentTimeMillis(); + } + // Now, compaction should be done. + assertEquals(CompactionState.NONE, state); + } + int countAfter = countStoreFilesInFamilies(regions, families); + int countAfterSingleFamily = countStoreFilesInFamily(regions, family); + assertTrue(countAfter < countBefore); + if (!singleFamily) { + if (expectedState == CompactionState.MAJOR) assertTrue(families.length == countAfter); + else assertTrue(families.length < countAfter); + } else { + int singleFamDiff = countBeforeSingleFamily - countAfterSingleFamily; + // assert only change was to single column family + assertTrue(singleFamDiff == (countBefore - countAfter)); + if (expectedState == CompactionState.MAJOR) { + assertTrue(1 == countAfterSingleFamily); + } else { + assertTrue(1 < countAfterSingleFamily); + } + } + } finally { + if (ht != null) { + TEST_UTIL.deleteTable(table); + } + } + } + + private static int countStoreFilesInFamily( + List regions, final byte[] family) { + return countStoreFilesInFamilies(regions, new byte[][]{family}); + } + + private static int countStoreFilesInFamilies(List regions, final byte[][] families) { + int count = 0; + for (HRegion region: regions) { + count += region.getStoreFileList(families).size(); + } + return count; + } + + private static void loadData(final HTable ht, final byte[][] families, + final int rows, final int flushes) throws IOException { + List puts = new ArrayList(rows); + byte[] qualifier = Bytes.toBytes("val"); + for (int i = 0; i < flushes; i++) { + for (int k = 0; k < rows; k++) { + byte[] row = Bytes.toBytes(random.nextLong()); + Put p = new Put(row); + for (int j = 0; j < families.length; ++j) { + p.add(families[ j ], qualifier, row); + } + puts.add(p); + } + ht.put(puts); + ht.flushCommits(); + TEST_UTIL.flush(); + puts.clear(); + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionWithCoprocessor.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionWithCoprocessor.java new file mode 100644 index 000000000000..ba30a9fdf388 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionWithCoprocessor.java @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; +import org.junit.experimental.categories.Category; + +/** + * Make sure all compaction tests still pass with the preFlush and preCompact + * overridden to implement the default behavior + */ +@Category(MediumTests.class) +public class TestCompactionWithCoprocessor extends TestCompaction { + /** constructor */ + public TestCompactionWithCoprocessor() throws Exception { + super(); + conf.setStrings(CoprocessorHost.USER_REGION_COPROCESSOR_CONF_KEY, + NoOpScanPolicyObserver.class.getName()); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java index 8859793c1bdf..9a22c3fd9393 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -298,6 +297,8 @@ private Path writeStoreFile(int t, BloomType bt, List kvs) BLOCK_SIZES[t]) .withOutputDir(TEST_UTIL.getDataTestDir()) .withBloomType(bt) + .withChecksumType(HFile.DEFAULT_CHECKSUM_TYPE) + .withBytesPerChecksum(HFile.DEFAULT_BYTES_PER_CHECKSUM) .build(); assertTrue(w.hasGeneralBloom()); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java new file mode 100644 index 000000000000..e147eb49e4f6 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java @@ -0,0 +1,234 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.util.HashMap; +import java.util.Map; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.io.ImmutableBytesWritable; +import org.apache.hadoop.hbase.regionserver.CompoundConfiguration; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.SmallTests; + +import org.junit.experimental.categories.Category; +import org.junit.Test; + +import junit.framework.TestCase; + +@Category(SmallTests.class) +public class TestCompoundConfiguration extends TestCase { + private Configuration baseConf; + private int baseConfSize; + + @Override + protected void setUp() throws Exception { + baseConf = new Configuration(); + baseConf.set("A", "1"); + baseConf.setInt("B", 2); + baseConf.set("C", "3"); + baseConfSize = baseConf.size(); + } + + @Test + public void testBasicFunctionality() throws ClassNotFoundException { + CompoundConfiguration compoundConf = new CompoundConfiguration() + .add(baseConf); + assertEquals("1", compoundConf.get("A")); + assertEquals(2, compoundConf.getInt("B", 0)); + assertEquals(3, compoundConf.getInt("C", 0)); + assertEquals(0, compoundConf.getInt("D", 0)); + + assertEquals(CompoundConfiguration.class, compoundConf + .getClassByName(CompoundConfiguration.class.getName())); + try { + compoundConf.getClassByName("bad_class_name"); + fail("Trying to load bad_class_name should throw an exception"); + } catch (ClassNotFoundException e) { + // win! + } + } + + @Test + public void testPut() { + CompoundConfiguration compoundConf = new CompoundConfiguration() + .add(baseConf); + assertEquals("1", compoundConf.get("A")); + assertEquals(2, compoundConf.getInt("B", 0)); + assertEquals(3, compoundConf.getInt("C", 0)); + assertEquals(0, compoundConf.getInt("D", 0)); + + compoundConf.set("A", "1337"); + compoundConf.set("string", "stringvalue"); + assertEquals(1337, compoundConf.getInt("A", 0)); + assertEquals("stringvalue", compoundConf.get("string")); + + // we didn't modify the base conf + assertEquals("1", baseConf.get("A")); + assertNull(baseConf.get("string")); + + // adding to the base shows up in the compound + baseConf.set("setInParent", "fromParent"); + assertEquals("fromParent", compoundConf.get("setInParent")); + } + + @Test + public void testWithConfig() { + Configuration conf = new Configuration(); + conf.set("B", "2b"); + conf.set("C", "33"); + conf.set("D", "4"); + + CompoundConfiguration compoundConf = new CompoundConfiguration() + .add(baseConf) + .add(conf); + assertEquals("1", compoundConf.get("A")); + assertEquals("2b", compoundConf.get("B")); + assertEquals(33, compoundConf.getInt("C", 0)); + assertEquals("4", compoundConf.get("D")); + assertEquals(4, compoundConf.getInt("D", 0)); + assertNull(compoundConf.get("E")); + assertEquals(6, compoundConf.getInt("F", 6)); + + int cnt = 0; + for (Map.Entry entry : compoundConf) { + cnt++; + if (entry.getKey().equals("B")) assertEquals("2b", entry.getValue()); + else if (entry.getKey().equals("G")) assertEquals(null, entry.getValue()); + } + // verify that entries from ImmutableConfigMap's are merged in the iterator's view + assertEquals(baseConfSize + 1, cnt); + } + + private ImmutableBytesWritable strToIbw(String s) { + return new ImmutableBytesWritable(Bytes.toBytes(s)); + } + + @Test + public void testWithIbwMap() { + Map map = + new HashMap(); + map.put(strToIbw("B"), strToIbw("2b")); + map.put(strToIbw("C"), strToIbw("33")); + map.put(strToIbw("D"), strToIbw("4")); + // unlike config, note that IBW Maps can accept null values + map.put(strToIbw("G"), null); + + CompoundConfiguration compoundConf = new CompoundConfiguration() + .add(baseConf) + .add(map); + assertEquals("1", compoundConf.get("A")); + assertEquals("2b", compoundConf.get("B")); + assertEquals(33, compoundConf.getInt("C", 0)); + assertEquals("4", compoundConf.get("D")); + assertEquals(4, compoundConf.getInt("D", 0)); + assertNull(compoundConf.get("E")); + assertEquals(6, compoundConf.getInt("F", 6)); + assertNull(compoundConf.get("G")); + + int cnt = 0; + for (Map.Entry entry : compoundConf) { + cnt++; + if (entry.getKey().equals("B")) assertEquals("2b", entry.getValue()); + else if (entry.getKey().equals("G")) assertEquals(null, entry.getValue()); + } + // verify that entries from ImmutableConfigMap's are merged in the iterator's view + assertEquals(baseConfSize + 2, cnt); + + // Verify that adding map after compound configuration is modified overrides properly + CompoundConfiguration conf2 = new CompoundConfiguration(); + conf2.set("X", "modification"); + conf2.set("D", "not4"); + assertEquals("modification", conf2.get("X")); + assertEquals("not4", conf2.get("D")); + conf2.add(map); + assertEquals("4", conf2.get("D")); // map overrides + } + + @Test + public void testWithStringMap() { + Map map = new HashMap(); + map.put("B", "2b"); + map.put("C", "33"); + map.put("D", "4"); + // unlike config, note that IBW Maps can accept null values + map.put("G", null); + + CompoundConfiguration compoundConf = new CompoundConfiguration().addStringMap(map); + assertEquals("2b", compoundConf.get("B")); + assertEquals(33, compoundConf.getInt("C", 0)); + assertEquals("4", compoundConf.get("D")); + assertEquals(4, compoundConf.getInt("D", 0)); + assertNull(compoundConf.get("E")); + assertEquals(6, compoundConf.getInt("F", 6)); + assertNull(compoundConf.get("G")); + + int cnt = 0; + for (Map.Entry entry : compoundConf) { + cnt++; + if (entry.getKey().equals("B")) assertEquals("2b", entry.getValue()); + else if (entry.getKey().equals("G")) assertEquals(null, entry.getValue()); + } + // verify that entries from ImmutableConfigMap's are merged in the iterator's view + assertEquals(4, cnt); + + // Verify that adding map after compound configuration is modified overrides properly + CompoundConfiguration conf2 = new CompoundConfiguration(); + conf2.set("X", "modification"); + conf2.set("D", "not4"); + assertEquals("modification", conf2.get("X")); + assertEquals("not4", conf2.get("D")); + conf2.addStringMap(map); + assertEquals("4", conf2.get("D")); // map overrides + } + + @Test + public void testLaterConfigsOverrideEarlier() { + Configuration map1 = new Configuration(false); + map1.set("A", "2"); + map1.set("D", "5"); + Configuration map2 = new Configuration(false); + String newValueForA = "3", newValueForB = "4"; + map2.set("A", newValueForA); + map2.set("B", newValueForB); + + CompoundConfiguration compoundConf = new CompoundConfiguration() + .add(map1).add(baseConf); + assertEquals("1", compoundConf.get("A")); + assertEquals("5", compoundConf.get("D")); + compoundConf.add(map2); + assertEquals(newValueForA, compoundConf.get("A")); + assertEquals(newValueForB, compoundConf.get("B")); + assertEquals("5", compoundConf.get("D")); + + int cnt = 0; + for (Map.Entry entry : compoundConf) { + cnt++; + if (entry.getKey().equals("A")) assertEquals(newValueForA, entry.getValue()); + else if (entry.getKey().equals("B")) assertEquals(newValueForB, entry.getValue()); + } + // verify that entries from ImmutableConfigMap's are merged in the iterator's view + assertEquals(baseConfSize + 1, cnt); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestCorruptedRegionStoreFile.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCorruptedRegionStoreFile.java new file mode 100644 index 000000000000..c35a056dfa36 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestCorruptedRegionStoreFile.java @@ -0,0 +1,246 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.ArrayList; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Durability; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.io.HFileLink; +import org.apache.hadoop.hbase.util.JVMClusterUtil.RegionServerThread; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.FSVisitor; + +import org.junit.After; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +@Category(LargeTests.class) +public class TestCorruptedRegionStoreFile { + private static final Log LOG = LogFactory.getLog(TestCorruptedRegionStoreFile.class); + + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + private static final String FAMILY_NAME_STR = "f"; + private static final byte[] FAMILY_NAME = Bytes.toBytes(FAMILY_NAME_STR); + + private static final int NUM_FILES = 25; + private static final int ROW_PER_FILE = 2000; + private static final int NUM_ROWS = NUM_FILES * ROW_PER_FILE; + + private final ArrayList storeFiles = new ArrayList(); + private Path tableDir; + private int rowCount; + + private static void setupConf(Configuration conf) { + conf.setLong("hbase.hstore.compaction.min", 20); + conf.setLong("hbase.hstore.compaction.max", 39); + conf.setLong("hbase.hstore.blockingStoreFiles", 40); + } + + private void setupTable(final String tableName) throws Exception { + // load the table + HTable table = UTIL.createTable(Bytes.toBytes(tableName), FAMILY_NAME); + try { + rowCount = 0; + byte[] value = new byte[1024]; + byte[] q = Bytes.toBytes("q"); + while (rowCount < NUM_ROWS) { + Put put = new Put(Bytes.toBytes(String.format("%010d", rowCount))); + put.setDurability(Durability.SKIP_WAL); + put.add(FAMILY_NAME, q, value); + table.put(put); + + if ((rowCount++ % ROW_PER_FILE) == 0) { + // flush it + UTIL.getHBaseAdmin().flush(tableName); + } + } + } finally { + UTIL.getHBaseAdmin().flush(tableName); + table.close(); + } + + assertEquals(NUM_ROWS, rowCount); + + // get the store file paths + storeFiles.clear(); + tableDir = FSUtils.getTablePath(getRootDir(), tableName); + FSVisitor.visitTableStoreFiles(getFileSystem(), tableDir, new FSVisitor.StoreFileVisitor() { + @Override + public void storeFile(final String region, final String family, final String hfile) + throws IOException { + HFileLink link = HFileLink.create(UTIL.getConfiguration(), tableName, region, family, hfile); + storeFiles.add(link.getOriginPath()); + } + }); + assertTrue("expected at least 1 store file", storeFiles.size() > 0); + LOG.info("store-files: " + storeFiles); + } + + @Before + public void setup() throws Exception { + setupConf(UTIL.getConfiguration()); + UTIL.startMiniCluster(2, 3); + } + + @After + public void tearDown() throws Exception { + try { + UTIL.shutdownMiniCluster(); + } catch (Exception e) { + LOG.warn("failure shutting down cluster", e); + } + } + + @Test(timeout=90000) + public void testLosingFileDuringScan() throws Exception { + final String tableName = "testLosingFileDuringScan"; + setupTable(tableName); + assertEquals(rowCount, fullScanAndCount(tableName)); + + final FileSystem fs = getFileSystem(); + final Path tmpStoreFilePath = new Path(UTIL.getDataTestDir(), "corruptedHFile"); + + // try to query with the missing file + int count = fullScanAndCount(tableName, new ScanInjector() { + private boolean hasFile = true; + + @Override + public void beforeScanNext(HTable table) throws Exception { + // move the path away (now the region is corrupted) + if (hasFile) { + fs.copyToLocalFile(true, storeFiles.get(0), tmpStoreFilePath); + LOG.info("Move file to local"); + evictHFileCache(storeFiles.get(0)); + hasFile = false; + } + } + }); + assertTrue("expected one file lost: rowCount=" + count, count >= (NUM_ROWS - ROW_PER_FILE)); + } + + @Test(timeout=90000) + public void testLosingFileAfterScannerInit() throws Exception { + final String tableName = "testLosingFileAfterScannerInit"; + setupTable(tableName); + assertEquals(rowCount, fullScanAndCount(tableName)); + + final FileSystem fs = getFileSystem(); + final Path tmpStoreFilePath = new Path(UTIL.getDataTestDir(), "corruptedHFile"); + + // try to query with the missing file + int count = fullScanAndCount(tableName, new ScanInjector() { + private boolean hasFile = true; + + @Override + public void beforeScan(HTable table, Scan scan) throws Exception { + // move the path away (now the region is corrupted) + if (hasFile) { + fs.copyToLocalFile(true, storeFiles.get(0), tmpStoreFilePath); + LOG.info("Move file to local"); + evictHFileCache(storeFiles.get(0)); + hasFile = false; + } + } + }); + assertTrue("expected one file lost: rowCount=" + count, count >= (NUM_ROWS - ROW_PER_FILE)); + } + + // ========================================================================== + // Helpers + // ========================================================================== + private FileSystem getFileSystem() { + return UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getFileSystem(); + } + + private Path getRootDir() { + return UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getRootDir(); + } + + private void evictHFileCache(final Path hfile) throws Exception { + for (RegionServerThread rst: UTIL.getMiniHBaseCluster().getRegionServerThreads()) { + HRegionServer rs = rst.getRegionServer(); + rs.getCacheConfig().getBlockCache().evictBlocksByHfileName(hfile.getName()); + } + Thread.sleep(6000); + } + + private int fullScanAndCount(final String tableName) throws Exception { + return fullScanAndCount(tableName, new ScanInjector()); + } + + private int fullScanAndCount(final String tableName, final ScanInjector injector) + throws Exception { + HTable table = new HTable(UTIL.getConfiguration(), tableName); + int count = 0; + try { + Scan scan = new Scan(); + scan.setCaching(1); + scan.setCacheBlocks(false); + injector.beforeScan(table, scan); + ResultScanner scanner = table.getScanner(scan); + try { + while (true) { + injector.beforeScanNext(table); + Result result = scanner.next(); + injector.afterScanNext(table, result); + if (result == null) break; + if ((count++ % 1000) == 0) { + LOG.debug("scan next " + count); + } + } + } finally { + scanner.close(); + injector.afterScan(table); + } + } finally { + table.close(); + } + return count; + } + + private class ScanInjector { + protected void beforeScan(HTable table, Scan scan) throws Exception {} + protected void beforeScanNext(HTable table) throws Exception {} + protected void afterScanNext(HTable table, Result result) throws Exception {} + protected void afterScan(HTable table) throws Exception {} + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java index 7bfe4cd95831..bf0d0a6a9f2d 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java @@ -17,29 +17,60 @@ */ package org.apache.hadoop.hbase.regionserver; +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; import java.io.IOException; +import java.util.List; +import java.util.Map; +import java.util.NavigableMap; +import java.util.Random; +import java.util.Set; +import org.apache.commons.io.IOUtils; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.Chore; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HServerAddress; import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.NotServingRegionException; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.Stoppable; +import org.apache.hadoop.hbase.catalog.MetaEditor; +import org.apache.hadoop.hbase.catalog.MetaReader; import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HConnection; import org.apache.hadoop.hbase.client.HConnectionManager; import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.MetaScanner; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; import org.apache.hadoop.hbase.util.PairOfSameType; +import org.apache.hadoop.hbase.util.StoppableImplementation; +import org.apache.hadoop.hbase.util.Threads; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; +import com.google.common.collect.Iterators; +import com.google.common.collect.Sets; + @Category(LargeTests.class) public class TestEndToEndSplitTransaction { + private static final Log LOG = LogFactory.getLog(TestEndToEndSplitTransaction.class); private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final Configuration conf = TEST_UTIL.getConfiguration(); @BeforeClass public static void beforeAllTests() throws Exception { @@ -51,7 +82,7 @@ public static void beforeAllTests() throws Exception { public static void afterAllTests() throws Exception { TEST_UTIL.shutdownMiniCluster(); } - + @Test public void testMasterOpsWhileSplitting() throws Exception { byte[] tableName = Bytes.toBytes("TestSplit"); @@ -103,7 +134,8 @@ public void testMasterOpsWhileSplitting() throws Exception { assertTrue(test(con, tableName, lastRow, server)); // 4. phase III - split.transitionZKNode(server, regions.getFirst(), regions.getSecond()); + split.transitionZKNode(server, server, regions.getFirst(), + regions.getSecond()); assertTrue(test(con, tableName, firstRow, server)); assertTrue(test(con, tableName, lastRow, server)); } @@ -127,6 +159,333 @@ private boolean test(HConnection con, byte[] tableName, byte[] row, return true; } + /** + * Tests that the client sees meta table changes as atomic during splits + */ + @Test + public void testFromClientSideWhileSplitting() throws Throwable { + LOG.info("Starting testFromClientSideWhileSplitting"); + final byte[] TABLENAME = Bytes.toBytes("testFromClientSideWhileSplitting"); + final byte[] FAMILY = Bytes.toBytes("family"); + + //SplitTransaction will update the meta table by offlining the parent region, and adding info + //for daughters. + HTable table = TEST_UTIL.createTable(TABLENAME, FAMILY); + + Stoppable stopper = new StoppableImplementation(); + RegionSplitter regionSplitter = new RegionSplitter(table); + RegionChecker regionChecker = new RegionChecker(conf, stopper, TABLENAME); + + regionChecker.start(); + regionSplitter.start(); + + //wait until the splitter is finished + regionSplitter.join(); + stopper.stop(null); + + if (regionChecker.ex != null) { + throw regionChecker.ex; + } + + if (regionSplitter.ex != null) { + throw regionSplitter.ex; + } + + //one final check + regionChecker.verify(); + } + + static class RegionSplitter extends Thread { + Throwable ex; + HTable table; + byte[] tableName, family; + HBaseAdmin admin; + HRegionServer rs; + + RegionSplitter(HTable table) throws IOException { + this.table = table; + this.tableName = table.getTableName(); + this.family = table.getTableDescriptor().getFamiliesKeys().iterator().next(); + admin = TEST_UTIL.getHBaseAdmin(); + rs = TEST_UTIL.getMiniHBaseCluster().getRegionServer(0); + } + + public void run() { + try { + Random random = new Random(); + for (int i=0; i< 5; i++) { + NavigableMap regions = MetaScanner.allTableRegions(conf, null, + tableName, false); + if (regions.size() == 0) { + continue; + } + int regionIndex = random.nextInt(regions.size()); + + //pick a random region and split it into two + HRegionInfo region = Iterators.get(regions.keySet().iterator(), regionIndex); + + //pick the mid split point + int start = 0, end = Integer.MAX_VALUE; + if (region.getStartKey().length > 0) { + start = Bytes.toInt(region.getStartKey()); + } + if (region.getEndKey().length > 0) { + end = Bytes.toInt(region.getEndKey()); + } + int mid = start + ((end - start) / 2); + byte[] splitPoint = Bytes.toBytes(mid); + + //put some rows to the regions + addData(start); + addData(mid); + + flushAndBlockUntilDone(admin, rs, region.getRegionName()); + compactAndBlockUntilDone(admin, rs, region.getRegionName()); + + log("Initiating region split for:" + region.getRegionNameAsString()); + try { + admin.split(region.getRegionName(), splitPoint); + //wait until the split is complete + blockUntilRegionSplit(conf, 50000, region.getRegionName(), true); + + } catch (NotServingRegionException ex) { + //ignore + } + } + } catch (Throwable ex) { + this.ex = ex; + } + } + + void addData(int start) throws IOException { + for (int i=start; i< start + 100; i++) { + Put put = new Put(Bytes.toBytes(i)); + + put.add(family, family, Bytes.toBytes(i)); + table.put(put); + } + table.flushCommits(); + } + } + + /** + * Checks regions using MetaScanner, MetaReader and HTable methods + */ + static class RegionChecker extends Chore { + Configuration conf; + byte[] tableName; + Throwable ex; + + RegionChecker(Configuration conf, Stoppable stopper, byte[] tableName) { + super("RegionChecker", 10, stopper); + this.conf = conf; + this.tableName = tableName; + this.setDaemon(true); + } + + /** verify region boundaries obtained from MetaScanner */ + void verifyRegionsUsingMetaScanner() throws Exception { + + //MetaScanner.allTableRegions() + NavigableMap regions = MetaScanner.allTableRegions(conf, null, + tableName, false); + verifyTableRegions(regions.keySet()); + + //MetaScanner.listAllRegions() + List regionList = MetaScanner.listAllRegions(conf, false); + verifyTableRegions(Sets.newTreeSet(regionList)); + } + + /** verify region boundaries obtained from HTable.getStartEndKeys() */ + void verifyRegionsUsingHTable() throws IOException { + HTable table = null; + try { + //HTable.getStartEndKeys() + table = new HTable(conf, tableName); + Pair keys = table.getStartEndKeys(); + verifyStartEndKeys(keys); + + //HTable.getRegionsInfo() + Map regions = table.getRegionsInfo(); + verifyTableRegions(regions.keySet()); + } finally { + IOUtils.closeQuietly(table); + } + } + + void verify() throws Exception { + verifyRegionsUsingMetaScanner(); + verifyRegionsUsingHTable(); + } + + void verifyTableRegions(Set regions) { + log("Verifying " + regions.size() + " regions"); + + byte[][] startKeys = new byte[regions.size()][]; + byte[][] endKeys = new byte[regions.size()][]; + + int i=0; + for (HRegionInfo region : regions) { + startKeys[i] = region.getStartKey(); + endKeys[i] = region.getEndKey(); + i++; + } + + Pair keys = new Pair(startKeys, endKeys); + verifyStartEndKeys(keys); + } + + void verifyStartEndKeys(Pair keys) { + byte[][] startKeys = keys.getFirst(); + byte[][] endKeys = keys.getSecond(); + assertEquals(startKeys.length, endKeys.length); + assertTrue("Found 0 regions for the table", startKeys.length > 0); + + assertArrayEquals("Start key for the first region is not byte[0]", + HConstants.EMPTY_START_ROW, startKeys[0]); + byte[] prevEndKey = HConstants.EMPTY_START_ROW; + + // ensure that we do not have any gaps + for (int i=0; i 0) { + Threads.sleep(50); + } + } + + public static void compactAndBlockUntilDone(HBaseAdmin admin, HRegionServer rs, byte[] regionName) + throws IOException, InterruptedException { + log("Compacting region: " + Bytes.toStringBinary(regionName)); + admin.majorCompact(regionName); + log("blocking until compaction is complete: " + Bytes.toStringBinary(regionName)); + Threads.sleepWithoutInterrupt(500); + while (rs.compactSplitThread.getCompactionQueueSize() > 0) { + Threads.sleep(50); + } + } + + /** Blocks until the region split is complete in META and region server opens the daughters */ + public static void blockUntilRegionSplit(Configuration conf, long timeout, + final byte[] regionName, boolean waitForDaughters) + throws IOException, InterruptedException { + long start = System.currentTimeMillis(); + log("blocking until region is split:" + Bytes.toStringBinary(regionName)); + HRegionInfo daughterA = null, daughterB = null; + HTable metaTable = new HTable(conf, HConstants.META_TABLE_NAME); + + try { + while (System.currentTimeMillis() - start < timeout) { + Result result = getRegionRow(metaTable, regionName); + if (result == null) { + break; + } + + HRegionInfo region = MetaReader.parseCatalogResult(result).getFirst(); + if(region.isSplitParent()) { + log("found parent region: " + region.toString()); + PairOfSameType pair = MetaReader.getDaughterRegions(result); + daughterA = pair.getFirst(); + daughterB = pair.getSecond(); + break; + } + Threads.sleep(100); + } + + //if we are here, this means the region split is complete or timed out + if (waitForDaughters) { + long rem = timeout - (System.currentTimeMillis() - start); + blockUntilRegionIsInMeta(metaTable, rem, daughterA); + + rem = timeout - (System.currentTimeMillis() - start); + blockUntilRegionIsInMeta(metaTable, rem, daughterB); + + rem = timeout - (System.currentTimeMillis() - start); + blockUntilRegionIsOpened(conf, rem, daughterA); + + rem = timeout - (System.currentTimeMillis() - start); + blockUntilRegionIsOpened(conf, rem, daughterB); + } + } finally { + IOUtils.closeQuietly(metaTable); + } + } + + public static Result getRegionRow(HTable metaTable, byte[] regionName) throws IOException { + Get get = new Get(regionName); + return metaTable.get(get); + } + + public static void blockUntilRegionIsInMeta(HTable metaTable, long timeout, HRegionInfo hri) + throws IOException, InterruptedException { + log("blocking until region is in META: " + hri.getRegionNameAsString()); + long start = System.currentTimeMillis(); + while (System.currentTimeMillis() - start < timeout) { + Result result = getRegionRow(metaTable, hri.getRegionName()); + if (result != null) { + HRegionInfo info = MetaReader.parseCatalogResult(result).getFirst(); + if (info != null && !info.isOffline()) { + log("found region in META: " + hri.getRegionNameAsString()); + break; + } + } + Threads.sleep(10); + } + } + + public static void blockUntilRegionIsOpened(Configuration conf, long timeout, HRegionInfo hri) + throws IOException, InterruptedException { + log("blocking until region is opened for reading:" + hri.getRegionNameAsString()); + long start = System.currentTimeMillis(); + HTable table = new HTable(conf, hri.getTableName()); + + try { + Get get = new Get(hri.getStartKey()); + while (System.currentTimeMillis() - start < timeout) { + try { + table.get(get); + break; + } catch(IOException ex) { + //wait some more + } + Threads.sleep(10); + } + } finally { + IOUtils.closeQuietly(table); + } + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java index 246b14581ba6..354f66869e4b 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -45,9 +44,9 @@ public class TestExplicitColumnTracker extends HBaseTestCase { private void runTest(int maxVersions, TreeSet trackColumns, List scannerColumns, - List expected) throws IOException { + List expected, int lookAhead) throws IOException { ColumnTracker exp = new ExplicitColumnTracker( - trackColumns, 0, maxVersions, Long.MIN_VALUE); + trackColumns, 0, maxVersions, Long.MIN_VALUE, lookAhead); //Initialize result @@ -56,7 +55,7 @@ private void runTest(int maxVersions, long timestamp = 0; //"Match" for(byte [] col : scannerColumns){ - result.add(exp.checkColumn(col, 0, col.length, ++timestamp, + result.add(ScanQueryMatcher.checkColumn(exp, col, 0, col.length, ++timestamp, KeyValue.Type.Put.getCode(), false)); } @@ -96,7 +95,7 @@ public void testGet_SingleVersion() throws IOException{ scanner.add(col4); scanner.add(col5); - runTest(maxVersions, columns, scanner, expected); + runTest(maxVersions, columns, scanner, expected, 0); } public void testGet_MultiVersion() throws IOException{ @@ -151,9 +150,63 @@ public void testGet_MultiVersion() throws IOException{ scanner.add(col5); //Initialize result - runTest(maxVersions, columns, scanner, expected); + runTest(maxVersions, columns, scanner, expected, 0); } + public void testGet_MultiVersionWithLookAhead() throws IOException{ + if(PRINT){ + System.out.println("\nMultiVersion"); + } + + //Create tracker + TreeSet columns = new TreeSet(Bytes.BYTES_COMPARATOR); + //Looking for every other + columns.add(col2); + columns.add(col4); + + List expected = new ArrayList(); + expected.add(ScanQueryMatcher.MatchCode.SKIP); + expected.add(ScanQueryMatcher.MatchCode.SKIP); + expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_COL); + + expected.add(ScanQueryMatcher.MatchCode.INCLUDE); // col2; 1st version + expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_COL); // col2; 2nd version + expected.add(ScanQueryMatcher.MatchCode.SKIP); + + expected.add(ScanQueryMatcher.MatchCode.SKIP); + expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_COL); + expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_COL); + + expected.add(ScanQueryMatcher.MatchCode.INCLUDE); // col4; 1st version + expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_ROW); // col4; 2nd version + expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_ROW); + + expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_ROW); + expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_ROW); + expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_ROW); + int maxVersions = 2; + + //Create "Scanner" + List scanner = new ArrayList(); + scanner.add(col1); + scanner.add(col1); + scanner.add(col1); + scanner.add(col2); + scanner.add(col2); + scanner.add(col2); + scanner.add(col3); + scanner.add(col3); + scanner.add(col3); + scanner.add(col4); + scanner.add(col4); + scanner.add(col4); + scanner.add(col5); + scanner.add(col5); + scanner.add(col5); + + //Initialize result + runTest(maxVersions, columns, scanner, expected, 2); + } /** * hbase-2259 @@ -166,17 +219,17 @@ public void testStackOverflow() throws IOException{ } ColumnTracker explicit = new ExplicitColumnTracker(columns, 0, maxVersions, - Long.MIN_VALUE); + Long.MIN_VALUE, 0); for (int i = 0; i < 100000; i+=2) { byte [] col = Bytes.toBytes("col"+i); - explicit.checkColumn(col, 0, col.length, 1, KeyValue.Type.Put.getCode(), + ScanQueryMatcher.checkColumn(explicit, col, 0, col.length, 1, KeyValue.Type.Put.getCode(), false); } - explicit.update(); + explicit.reset(); for (int i = 1; i < 100000; i+=2) { byte [] col = Bytes.toBytes("col"+i); - explicit.checkColumn(col, 0, col.length, 1, KeyValue.Type.Put.getCode(), + ScanQueryMatcher.checkColumn(explicit, col, 0, col.length, 1, KeyValue.Type.Put.getCode(), false); } } @@ -194,7 +247,7 @@ public void testInfiniteLoop() throws IOException { new ScanQueryMatcher.MatchCode[] { ScanQueryMatcher.MatchCode.SEEK_NEXT_COL, ScanQueryMatcher.MatchCode.SEEK_NEXT_COL }); - runTest(1, columns, scanner, expected); + runTest(1, columns, scanner, expected, 0); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java index f93487d6adb4..6f918b5fc983 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -41,6 +40,7 @@ import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.fs.HFileSystem; import org.apache.hadoop.hbase.io.hfile.CacheConfig; import org.apache.hadoop.hbase.io.hfile.HFileScanner; import org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder; @@ -69,10 +69,12 @@ public void testHFileScannerThrowsErrors() throws IOException { Path hfilePath = new Path(new Path( util.getDataTestDir("internalScannerExposesErrors"), "regionname"), "familyname"); - FaultyFileSystem fs = new FaultyFileSystem(util.getTestFileSystem()); + HFileSystem hfs = (HFileSystem)util.getTestFileSystem(); + FaultyFileSystem faultyfs = new FaultyFileSystem(hfs.getBackingFs()); + FileSystem fs = new HFileSystem(faultyfs); CacheConfig cacheConf = new CacheConfig(util.getConfiguration()); StoreFile.Writer writer = new StoreFile.WriterBuilder( - util.getConfiguration(), cacheConf, fs, 2*1024) + util.getConfiguration(), cacheConf, hfs, 2*1024) .withOutputDir(hfilePath) .build(); TestStoreFile.writeStoreFile( @@ -85,14 +87,14 @@ public void testHFileScannerThrowsErrors() throws IOException { StoreFile.Reader reader = sf.createReader(); HFileScanner scanner = reader.getScanner(false, true); - FaultyInputStream inStream = fs.inStreams.get(0).get(); + FaultyInputStream inStream = faultyfs.inStreams.get(0).get(); assertNotNull(inStream); scanner.seekTo(); // Do at least one successful read assertTrue(scanner.next()); - inStream.startFaults(); + faultyfs.startFaults(); try { int scanned=0; @@ -116,10 +118,12 @@ public void testStoreFileScannerThrowsErrors() throws IOException { Path hfilePath = new Path(new Path( util.getDataTestDir("internalScannerExposesErrors"), "regionname"), "familyname"); - FaultyFileSystem fs = new FaultyFileSystem(util.getTestFileSystem()); + HFileSystem hfs = (HFileSystem)util.getTestFileSystem(); + FaultyFileSystem faultyfs = new FaultyFileSystem(hfs.getBackingFs()); + HFileSystem fs = new HFileSystem(faultyfs); CacheConfig cacheConf = new CacheConfig(util.getConfiguration()); StoreFile.Writer writer = new StoreFile.WriterBuilder( - util.getConfiguration(), cacheConf, fs, 2 * 1024) + util.getConfiguration(), cacheConf, hfs, 2 * 1024) .withOutputDir(hfilePath) .build(); TestStoreFile.writeStoreFile( @@ -132,14 +136,13 @@ public void testStoreFileScannerThrowsErrors() throws IOException { Collections.singletonList(sf), false, true, false); KeyValueScanner scanner = scanners.get(0); - FaultyInputStream inStream = fs.inStreams.get(0).get(); + FaultyInputStream inStream = faultyfs.inStreams.get(0).get(); assertNotNull(inStream); scanner.seek(KeyValue.LOWESTKEY); // Do at least one successful read assertNotNull(scanner.next()); - - inStream.startFaults(); + faultyfs.startFaults(); try { int scanned=0; @@ -220,6 +223,15 @@ public FSDataInputStream open(Path p, int bufferSize) throws IOException { inStreams.add(new SoftReference(faulty)); return faulty; } + + /** + * Starts to simulate faults on all streams opened so far + */ + public void startFaults() { + for (SoftReference is: inStreams) { + is.get().startFaults(); + } + } } static class FaultyInputStream extends FSDataInputStream { diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java index 5f97167543a6..9478a57fead2 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestHBase7051.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHBase7051.java new file mode 100644 index 000000000000..5636dd611845 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHBase7051.java @@ -0,0 +1,205 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import static org.junit.Assert.assertEquals; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.MultithreadedTestUtil; +import org.apache.hadoop.hbase.MultithreadedTestUtil.TestContext; +import org.apache.hadoop.hbase.MultithreadedTestUtil.TestThread; +import org.apache.hadoop.hbase.client.Mutation; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.filter.BinaryComparator; +import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp; +import org.apache.hadoop.hbase.io.HeapSize; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.HashedBytes; +import org.apache.hadoop.hbase.util.Pair; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import com.google.common.collect.Lists; + +/** + * Test of HBASE-7051; that checkAndPuts and puts behave atomically with respect to each other. + * Rather than perform a bunch of trials to verify atomicity, this test recreates a race condition + * that causes the test to fail if checkAndPut doesn't wait for outstanding put transactions + * to complete. It does this by invasively overriding HRegion function to affect the timing of + * the operations. + */ +@Category(SmallTests.class) +public class TestHBase7051 { + + private static CountDownLatch latch = new CountDownLatch(1); + private enum TestStep { + INIT, // initial put of 10 to set value of the cell + PUT_STARTED, // began doing a put of 50 to cell + PUT_COMPLETED, // put complete (released RowLock, but may not have advanced MVCC). + CHECKANDPUT_STARTED, // began checkAndPut: if 10 -> 11 + CHECKANDPUT_COMPLETED // completed checkAndPut + // NOTE: at the end of these steps, the value of the cell should be 50, not 11! + } + private static volatile TestStep testStep = TestStep.INIT; + private final String family = "f1"; + + @Test + public void testPutAndCheckAndPutInParallel() throws Exception { + + final String tableName = "testPutAndCheckAndPut"; + Configuration conf = HBaseConfiguration.create(); + conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class); + final MockHRegion region = (MockHRegion) TestHRegion.initHRegion(Bytes.toBytes(tableName), + tableName, conf, Bytes.toBytes(family)); + + List> putsAndLocks = Lists.newArrayList(); + Put[] puts = new Put[1]; + Put put = new Put(Bytes.toBytes("r1")); + put.add(Bytes.toBytes(family), Bytes.toBytes("q1"), Bytes.toBytes("10")); + puts[0] = put; + Pair pair = new Pair(puts[0], null); + + putsAndLocks.add(pair); + + region.batchMutate(putsAndLocks.toArray(new Pair[0])); + MultithreadedTestUtil.TestContext ctx = + new MultithreadedTestUtil.TestContext(conf); + ctx.addThread(new PutThread(ctx, region)); + ctx.addThread(new CheckAndPutThread(ctx, region)); + ctx.startThreads(); + while (testStep != TestStep.CHECKANDPUT_COMPLETED) { + Thread.sleep(100); + } + ctx.stop(); + Scan s = new Scan(); + RegionScanner scanner = region.getScanner(s); + List results = new ArrayList(); + scanner.next(results, 2); + for (KeyValue keyValue : results) { + assertEquals("50",Bytes.toString(keyValue.getValue())); + } + + } + + private class PutThread extends TestThread { + private MockHRegion region; + PutThread(TestContext ctx, MockHRegion region) { + super(ctx); + this.region = region; + } + + public void doWork() throws Exception { + List> putsAndLocks = Lists.newArrayList(); + Put[] puts = new Put[1]; + Put put = new Put(Bytes.toBytes("r1")); + put.add(Bytes.toBytes(family), Bytes.toBytes("q1"), Bytes.toBytes("50")); + puts[0] = put; + Pair pair = new Pair(puts[0], null); + putsAndLocks.add(pair); + testStep = TestStep.PUT_STARTED; + region.batchMutate(putsAndLocks.toArray(new Pair[0])); + } + } + + private class CheckAndPutThread extends TestThread { + private MockHRegion region; + CheckAndPutThread(TestContext ctx, MockHRegion region) { + super(ctx); + this.region = region; + } + + public void doWork() throws Exception { + Put[] puts = new Put[1]; + Put put = new Put(Bytes.toBytes("r1")); + put.add(Bytes.toBytes(family), Bytes.toBytes("q1"), Bytes.toBytes("11")); + puts[0] = put; + while (testStep != TestStep.PUT_COMPLETED) { + Thread.sleep(100); + } + testStep = TestStep.CHECKANDPUT_STARTED; + region.checkAndMutate(Bytes.toBytes("r1"), Bytes.toBytes(family), Bytes.toBytes("q1"), + CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes("10")), put, null, true); + testStep = TestStep.CHECKANDPUT_COMPLETED; + } + } + + public static class MockHRegion extends HRegion { + + public MockHRegion(Path tableDir, HLog log, FileSystem fs, Configuration conf, + final HRegionInfo regionInfo, final HTableDescriptor htd, RegionServerServices rsServices) { + super(tableDir, log, fs, conf, regionInfo, htd, rsServices); + } + + @Override + public void releaseRowLock(Integer lockId) { + if (testStep == TestStep.INIT) { + super.releaseRowLock(lockId); + return; + } + + if (testStep == TestStep.PUT_STARTED) { + try { + testStep = TestStep.PUT_COMPLETED; + super.releaseRowLock(lockId); + // put has been written to the memstore and the row lock has been released, but the + // MVCC has not been advanced. Prior to fixing HBASE-7051, the following order of + // operations would cause the non-atomicity to show up: + // 1) Put releases row lock (where we are now) + // 2) CheckAndPut grabs row lock and reads the value prior to the put (10) + // because the MVCC has not advanced + // 3) Put advances MVCC + // So, in order to recreate this order, we wait for the checkAndPut to grab the rowLock + // (see below), and then wait some more to give the checkAndPut time to read the old + // value. + latch.await(); + Thread.sleep(1000); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + else if (testStep == TestStep.CHECKANDPUT_STARTED) { + super.releaseRowLock(lockId); + } + } + + @Override + public Integer getLock(Integer lockid, HashedBytes row, boolean waitForLock) throws IOException { + if (testStep == TestStep.CHECKANDPUT_STARTED) { + latch.countDown(); + } + return super.getLock(lockid, row, waitForLock); + } + + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java index 8db43a41ae08..5f0644ba876d 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,7 +19,10 @@ package org.apache.hadoop.hbase.regionserver; +import java.io.DataInput; +import java.io.DataOutput; import java.io.IOException; +import java.io.InterruptedIOException; import java.util.ArrayList; import java.util.Arrays; import java.util.HashMap; @@ -37,6 +39,7 @@ import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.DroppedSnapshotException; import org.apache.hadoop.hbase.DoNotRetryIOException; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HBaseTestCase; @@ -51,22 +54,28 @@ import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.MiniHBaseCluster; import org.apache.hadoop.hbase.MultithreadedTestUtil; +import org.apache.hadoop.hbase.MultithreadedTestUtil.RepeatingTestThread; import org.apache.hadoop.hbase.MultithreadedTestUtil.TestThread; +import org.apache.hadoop.hbase.client.Append; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Increment; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.RowMutations; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.filter.BinaryComparator; import org.apache.hadoop.hbase.filter.ColumnCountGetFilter; import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp; import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterBase; import org.apache.hadoop.hbase.filter.FilterList; import org.apache.hadoop.hbase.filter.NullComparator; import org.apache.hadoop.hbase.filter.PrefixFilter; +import org.apache.hadoop.hbase.filter.SingleColumnValueExcludeFilter; import org.apache.hadoop.hbase.filter.SingleColumnValueFilter; -import org.apache.hadoop.hbase.io.hfile.Compression; +import org.apache.hadoop.hbase.monitoring.MonitoredRPCHandler; import org.apache.hadoop.hbase.monitoring.MonitoredTask; import org.apache.hadoop.hbase.monitoring.TaskMonitor; import org.apache.hadoop.hbase.regionserver.HRegion.RegionScannerImpl; @@ -83,8 +92,10 @@ import org.apache.hadoop.hbase.util.Pair; import org.apache.hadoop.hbase.util.PairOfSameType; import org.apache.hadoop.hbase.util.Threads; +import org.junit.Assert; import org.junit.Test; import org.junit.experimental.categories.Category; +import org.mockito.Mockito; import com.google.common.collect.Lists; @@ -96,14 +107,17 @@ * HRegions or in the HBaseMaster, so only basic testing is possible. */ @Category(MediumTests.class) +@SuppressWarnings("deprecation") public class TestHRegion extends HBaseTestCase { + // Do not spin up clusters in here. If you need to spin up a cluster, do it + // over in TestHRegionOnCluster. static final Log LOG = LogFactory.getLog(TestHRegion.class); private static final String COLUMN_FAMILY = "MyCF"; HRegion region = null; - private HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); - private final String DIR = TEST_UTIL.getDataTestDir("TestHRegion").toString(); + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final String DIR = TEST_UTIL.getDataTestDir("TestHRegion").toString(); private final int MAX_VERSIONS = 2; @@ -142,45 +156,137 @@ protected void tearDown() throws Exception { // /tmp/testtable ////////////////////////////////////////////////////////////////////////////// + public void testCompactionAffectedByScanners() throws Exception { + String method = "testCompactionAffectedByScanners"; + byte[] tableName = Bytes.toBytes(method); + byte[] family = Bytes.toBytes("family"); + this.region = initHRegion(tableName, method, conf, family); + + Put put = new Put(Bytes.toBytes("r1")); + put.add(family, Bytes.toBytes("q1"), Bytes.toBytes("v1")); + region.put(put); + region.flushcache(); + + + Scan scan = new Scan(); + scan.setMaxVersions(3); + // open the first scanner + RegionScanner scanner1 = region.getScanner(scan); + + Delete delete = new Delete(Bytes.toBytes("r1")); + region.delete(delete, null, false); + region.flushcache(); + + // open the second scanner + RegionScanner scanner2 = region.getScanner(scan); + + List results = new ArrayList(); + + System.out.println("Smallest read point:" + region.getSmallestReadPoint()); + + // make a major compaction + region.compactStores(true); + + // open the third scanner + RegionScanner scanner3 = region.getScanner(scan); + + // get data from scanner 1, 2, 3 after major compaction + scanner1.next(results); + System.out.println(results); + assertEquals(1, results.size()); + + results.clear(); + scanner2.next(results); + System.out.println(results); + assertEquals(0, results.size()); + + results.clear(); + scanner3.next(results); + System.out.println(results); + assertEquals(0, results.size()); + } + + @Test + public void testToShowNPEOnRegionScannerReseek() throws Exception{ + String method = "testToShowNPEOnRegionScannerReseek"; + byte[] tableName = Bytes.toBytes(method); + byte[] family = Bytes.toBytes("family"); + this.region = initHRegion(tableName, method, conf, family); + + Put put = new Put(Bytes.toBytes("r1")); + put.add(family, Bytes.toBytes("q1"), Bytes.toBytes("v1")); + region.put(put); + put = new Put(Bytes.toBytes("r2")); + put.add(family, Bytes.toBytes("q1"), Bytes.toBytes("v1")); + region.put(put); + region.flushcache(); + + + Scan scan = new Scan(); + scan.setMaxVersions(3); + // open the first scanner + RegionScanner scanner1 = region.getScanner(scan); + + System.out.println("Smallest read point:" + region.getSmallestReadPoint()); + + region.compactStores(true); + + scanner1.reseek(Bytes.toBytes("r2")); + List results = new ArrayList(); + scanner1.next(results); + KeyValue keyValue = results.get(0); + Assert.assertTrue(Bytes.compareTo(keyValue.getRow(), Bytes.toBytes("r2")) == 0); + scanner1.close(); + } public void testSkipRecoveredEditsReplay() throws Exception { String method = "testSkipRecoveredEditsReplay"; byte[] tableName = Bytes.toBytes(method); byte[] family = Bytes.toBytes("family"); - Configuration conf = HBaseConfiguration.create(); - initHRegion(tableName, method, conf, family); - Path regiondir = region.getRegionDir(); - FileSystem fs = region.getFilesystem(); - byte[] regionName = region.getRegionInfo().getEncodedNameAsBytes(); + this.region = initHRegion(tableName, method, conf, family); + try { + Path regiondir = region.getRegionDir(); + FileSystem fs = region.getFilesystem(); + byte[] regionName = region.getRegionInfo().getEncodedNameAsBytes(); - Path recoveredEditsDir = HLog.getRegionDirRecoveredEditsDir(regiondir); + Path recoveredEditsDir = HLog.getRegionDirRecoveredEditsDir(regiondir); - long maxSeqId = 1050; - long minSeqId = 1000; + long maxSeqId = 1050; + long minSeqId = 1000; - for (long i = minSeqId; i <= maxSeqId; i += 10) { - Path recoveredEdits = new Path(recoveredEditsDir, String.format("%019d", i)); - fs.create(recoveredEdits); - HLog.Writer writer = HLog.createWriter(fs, recoveredEdits, conf); + for (long i = minSeqId; i <= maxSeqId; i += 10) { + Path recoveredEdits = new Path(recoveredEditsDir, String.format("%019d", i)); + fs.create(recoveredEdits); + HLog.Writer writer = HLog.createWriter(fs, recoveredEdits, conf); - long time = System.nanoTime(); - WALEdit edit = new WALEdit(); - edit.add(new KeyValue(row, family, Bytes.toBytes(i), - time, KeyValue.Type.Put, Bytes.toBytes(i))); - writer.append(new HLog.Entry(new HLogKey(regionName, tableName, - i, time, HConstants.DEFAULT_CLUSTER_ID), edit)); + long time = System.nanoTime(); + WALEdit edit = new WALEdit(); + edit.add(new KeyValue(row, family, Bytes.toBytes(i), + time, KeyValue.Type.Put, Bytes.toBytes(i))); + writer.append(new HLog.Entry(new HLogKey(regionName, tableName, + i, time, HConstants.DEFAULT_CLUSTER_ID), edit)); - writer.close(); - } - MonitoredTask status = TaskMonitor.get().createStatus(method); - long seqId = region.replayRecoveredEditsIfAny(regiondir, minSeqId-1, null, status); - assertEquals(maxSeqId, seqId); - Get get = new Get(row); - Result result = region.get(get, null); - for (long i = minSeqId; i <= maxSeqId; i += 10) { - List kvs = result.getColumn(family, Bytes.toBytes(i)); - assertEquals(1, kvs.size()); - assertEquals(Bytes.toBytes(i), kvs.get(0).getValue()); + writer.close(); + } + MonitoredTask status = TaskMonitor.get().createStatus(method); + Map maxSeqIdInStores = new TreeMap( + Bytes.BYTES_COMPARATOR); + for (Store store : region.getStores().values()) { + maxSeqIdInStores.put(store.getColumnFamilyName().getBytes(), + minSeqId - 1); + } + long seqId = region.replayRecoveredEditsIfAny(regiondir, maxSeqIdInStores, null, status); + assertEquals(maxSeqId, seqId); + Get get = new Get(row); + Result result = region.get(get, null); + for (long i = minSeqId; i <= maxSeqId; i += 10) { + List kvs = result.getColumn(family, Bytes.toBytes(i)); + assertEquals(1, kvs.size()); + assertEquals(Bytes.toBytes(i), kvs.get(0).getValue()); + } + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -188,44 +294,55 @@ public void testSkipRecoveredEditsReplaySomeIgnored() throws Exception { String method = "testSkipRecoveredEditsReplaySomeIgnored"; byte[] tableName = Bytes.toBytes(method); byte[] family = Bytes.toBytes("family"); - initHRegion(tableName, method, HBaseConfiguration.create(), family); - Path regiondir = region.getRegionDir(); - FileSystem fs = region.getFilesystem(); - byte[] regionName = region.getRegionInfo().getEncodedNameAsBytes(); - - Path recoveredEditsDir = HLog.getRegionDirRecoveredEditsDir(regiondir); - - long maxSeqId = 1050; - long minSeqId = 1000; - - for (long i = minSeqId; i <= maxSeqId; i += 10) { - Path recoveredEdits = new Path(recoveredEditsDir, String.format("%019d", i)); - fs.create(recoveredEdits); - HLog.Writer writer = HLog.createWriter(fs, recoveredEdits, conf); - - long time = System.nanoTime(); - WALEdit edit = new WALEdit(); - edit.add(new KeyValue(row, family, Bytes.toBytes(i), - time, KeyValue.Type.Put, Bytes.toBytes(i))); - writer.append(new HLog.Entry(new HLogKey(regionName, tableName, - i, time, HConstants.DEFAULT_CLUSTER_ID), edit)); - - writer.close(); - } - long recoverSeqId = 1030; - MonitoredTask status = TaskMonitor.get().createStatus(method); - long seqId = region.replayRecoveredEditsIfAny(regiondir, recoverSeqId-1, null, status); - assertEquals(maxSeqId, seqId); - Get get = new Get(row); - Result result = region.get(get, null); - for (long i = minSeqId; i <= maxSeqId; i += 10) { - List kvs = result.getColumn(family, Bytes.toBytes(i)); - if (i < recoverSeqId) { - assertEquals(0, kvs.size()); - } else { - assertEquals(1, kvs.size()); - assertEquals(Bytes.toBytes(i), kvs.get(0).getValue()); + this.region = initHRegion(tableName, method, conf, family); + try { + Path regiondir = region.getRegionDir(); + FileSystem fs = region.getFilesystem(); + byte[] regionName = region.getRegionInfo().getEncodedNameAsBytes(); + + Path recoveredEditsDir = HLog.getRegionDirRecoveredEditsDir(regiondir); + + long maxSeqId = 1050; + long minSeqId = 1000; + + for (long i = minSeqId; i <= maxSeqId; i += 10) { + Path recoveredEdits = new Path(recoveredEditsDir, String.format("%019d", i)); + fs.create(recoveredEdits); + HLog.Writer writer = HLog.createWriter(fs, recoveredEdits, conf); + + long time = System.nanoTime(); + WALEdit edit = new WALEdit(); + edit.add(new KeyValue(row, family, Bytes.toBytes(i), + time, KeyValue.Type.Put, Bytes.toBytes(i))); + writer.append(new HLog.Entry(new HLogKey(regionName, tableName, + i, time, HConstants.DEFAULT_CLUSTER_ID), edit)); + + writer.close(); + } + long recoverSeqId = 1030; + MonitoredTask status = TaskMonitor.get().createStatus(method); + Map maxSeqIdInStores = new TreeMap( + Bytes.BYTES_COMPARATOR); + for (Store store : region.getStores().values()) { + maxSeqIdInStores.put(store.getColumnFamilyName().getBytes(), + recoverSeqId - 1); } + long seqId = region.replayRecoveredEditsIfAny(regiondir, maxSeqIdInStores, null, status); + assertEquals(maxSeqId, seqId); + Get get = new Get(row); + Result result = region.get(get, null); + for (long i = minSeqId; i <= maxSeqId; i += 10) { + List kvs = result.getColumn(family, Bytes.toBytes(i)); + if (i < recoverSeqId) { + assertEquals(0, kvs.size()); + } else { + assertEquals(1, kvs.size()); + assertEquals(Bytes.toBytes(i), kvs.get(0).getValue()); + } + } + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -233,25 +350,37 @@ public void testSkipRecoveredEditsReplayAllIgnored() throws Exception { String method = "testSkipRecoveredEditsReplayAllIgnored"; byte[] tableName = Bytes.toBytes(method); byte[] family = Bytes.toBytes("family"); - initHRegion(tableName, method, HBaseConfiguration.create(), family); - Path regiondir = region.getRegionDir(); - FileSystem fs = region.getFilesystem(); - - Path recoveredEditsDir = HLog.getRegionDirRecoveredEditsDir(regiondir); - for (int i = 1000; i < 1050; i += 10) { + this.region = initHRegion(tableName, method, conf, family); + try { + Path regiondir = region.getRegionDir(); + FileSystem fs = region.getFilesystem(); + + Path recoveredEditsDir = HLog.getRegionDirRecoveredEditsDir(regiondir); + for (int i = 1000; i < 1050; i += 10) { + Path recoveredEdits = new Path( + recoveredEditsDir, String.format("%019d", i)); + FSDataOutputStream dos= fs.create(recoveredEdits); + dos.writeInt(i); + dos.close(); + } + long minSeqId = 2000; Path recoveredEdits = new Path( - recoveredEditsDir, String.format("%019d", i)); + recoveredEditsDir, String.format("%019d", minSeqId-1)); FSDataOutputStream dos= fs.create(recoveredEdits); - dos.writeInt(i); dos.close(); + + Map maxSeqIdInStores = new TreeMap( + Bytes.BYTES_COMPARATOR); + for (Store store : region.getStores().values()) { + maxSeqIdInStores.put(store.getColumnFamilyName().getBytes(), minSeqId); + } + long seqId = region.replayRecoveredEditsIfAny(regiondir, + maxSeqIdInStores, null, null); + assertEquals(minSeqId, seqId); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } - long minSeqId = 2000; - Path recoveredEdits = new Path( - recoveredEditsDir, String.format("%019d", minSeqId-1)); - FSDataOutputStream dos= fs.create(recoveredEdits); - dos.close(); - long seqId = region.replayRecoveredEditsIfAny(regiondir, minSeqId, null, null); - assertEquals(minSeqId, seqId); } public void testGetWhileRegionClose() throws IOException { @@ -261,52 +390,56 @@ public void testGetWhileRegionClose() throws IOException { //Setting up region String method = this.getName(); - initHRegion(tableName, method, hc, families); - - // Put data in region - final int startRow = 100; - putData(startRow, numRows, qual1, families); - putData(startRow, numRows, qual2, families); - putData(startRow, numRows, qual3, families); - // this.region.flushcache(); - final AtomicBoolean done = new AtomicBoolean(false); - final AtomicInteger gets = new AtomicInteger(0); - GetTillDoneOrException [] threads = new GetTillDoneOrException[10]; - try { - // Set ten threads running concurrently getting from the region. - for (int i = 0; i < threads.length / 2; i++) { - threads[i] = new GetTillDoneOrException(i, Bytes.toBytes("" + startRow), - done, gets); - threads[i].setDaemon(true); - threads[i].start(); - } - // Artificially make the condition by setting closing flag explicitly. - // I can't make the issue happen with a call to region.close(). - this.region.closing.set(true); - for (int i = threads.length / 2; i < threads.length; i++) { - threads[i] = new GetTillDoneOrException(i, Bytes.toBytes("" + startRow), - done, gets); - threads[i].setDaemon(true); - threads[i].start(); - } - } finally { - if (this.region != null) { - this.region.close(); - this.region.getLog().closeAndDelete(); - } - } - done.set(true); - for (GetTillDoneOrException t: threads) { + this.region = initHRegion(tableName, method, hc, families); + try { + // Put data in region + final int startRow = 100; + putData(startRow, numRows, qual1, families); + putData(startRow, numRows, qual2, families); + putData(startRow, numRows, qual3, families); + // this.region.flushcache(); + final AtomicBoolean done = new AtomicBoolean(false); + final AtomicInteger gets = new AtomicInteger(0); + GetTillDoneOrException [] threads = new GetTillDoneOrException[10]; try { - t.join(); - } catch (InterruptedException e) { - e.printStackTrace(); + // Set ten threads running concurrently getting from the region. + for (int i = 0; i < threads.length / 2; i++) { + threads[i] = new GetTillDoneOrException(i, Bytes.toBytes("" + startRow), + done, gets); + threads[i].setDaemon(true); + threads[i].start(); + } + // Artificially make the condition by setting closing flag explicitly. + // I can't make the issue happen with a call to region.close(). + this.region.closing.set(true); + for (int i = threads.length / 2; i < threads.length; i++) { + threads[i] = new GetTillDoneOrException(i, Bytes.toBytes("" + startRow), + done, gets); + threads[i].setDaemon(true); + threads[i].start(); + } + } finally { + if (this.region != null) { + this.region.close(); + this.region.getLog().closeAndDelete(); + } } - if (t.e != null) { - LOG.info("Exception=" + t.e); - assertFalse("Found a NPE in " + t.getName(), - t.e instanceof NullPointerException); + done.set(true); + for (GetTillDoneOrException t: threads) { + try { + t.join(); + } catch (InterruptedException e) { + e.printStackTrace(); + } + if (t.e != null) { + LOG.info("Exception=" + t.e); + assertFalse("Found a NPE in " + t.getName(), + t.e instanceof NullPointerException); + } } + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -350,38 +483,78 @@ public void testWeirdCacheBehaviour() throws Exception { byte[][] FAMILIES = new byte[][] { Bytes.toBytes("trans-blob"), Bytes.toBytes("trans-type"), Bytes.toBytes("trans-date"), Bytes.toBytes("trans-tags"), Bytes.toBytes("trans-group") }; - initHRegion(TABLE, getName(), FAMILIES); - String value = "this is the value"; - String value2 = "this is some other value"; - String keyPrefix1 = "prefix1"; // UUID.randomUUID().toString(); - String keyPrefix2 = "prefix2"; // UUID.randomUUID().toString(); - String keyPrefix3 = "prefix3"; // UUID.randomUUID().toString(); - putRows(this.region, 3, value, keyPrefix1); - putRows(this.region, 3, value, keyPrefix2); - putRows(this.region, 3, value, keyPrefix3); - // this.region.flushCommits(); - putRows(this.region, 3, value2, keyPrefix1); - putRows(this.region, 3, value2, keyPrefix2); - putRows(this.region, 3, value2, keyPrefix3); - System.out.println("Checking values for key: " + keyPrefix1); - assertEquals("Got back incorrect number of rows from scan", 3, - getNumberOfRows(keyPrefix1, value2, this.region)); - System.out.println("Checking values for key: " + keyPrefix2); - assertEquals("Got back incorrect number of rows from scan", 3, - getNumberOfRows(keyPrefix2, value2, this.region)); - System.out.println("Checking values for key: " + keyPrefix3); - assertEquals("Got back incorrect number of rows from scan", 3, - getNumberOfRows(keyPrefix3, value2, this.region)); - deleteColumns(this.region, value2, keyPrefix1); - deleteColumns(this.region, value2, keyPrefix2); - deleteColumns(this.region, value2, keyPrefix3); - System.out.println("Starting important checks....."); - assertEquals("Got back incorrect number of rows from scan: " + keyPrefix1, - 0, getNumberOfRows(keyPrefix1, value2, this.region)); - assertEquals("Got back incorrect number of rows from scan: " + keyPrefix2, - 0, getNumberOfRows(keyPrefix2, value2, this.region)); - assertEquals("Got back incorrect number of rows from scan: " + keyPrefix3, - 0, getNumberOfRows(keyPrefix3, value2, this.region)); + this.region = initHRegion(TABLE, getName(), conf, FAMILIES); + try { + String value = "this is the value"; + String value2 = "this is some other value"; + String keyPrefix1 = "prefix1"; // UUID.randomUUID().toString(); + String keyPrefix2 = "prefix2"; // UUID.randomUUID().toString(); + String keyPrefix3 = "prefix3"; // UUID.randomUUID().toString(); + putRows(this.region, 3, value, keyPrefix1); + putRows(this.region, 3, value, keyPrefix2); + putRows(this.region, 3, value, keyPrefix3); + // this.region.flushCommits(); + putRows(this.region, 3, value2, keyPrefix1); + putRows(this.region, 3, value2, keyPrefix2); + putRows(this.region, 3, value2, keyPrefix3); + System.out.println("Checking values for key: " + keyPrefix1); + assertEquals("Got back incorrect number of rows from scan", 3, + getNumberOfRows(keyPrefix1, value2, this.region)); + System.out.println("Checking values for key: " + keyPrefix2); + assertEquals("Got back incorrect number of rows from scan", 3, + getNumberOfRows(keyPrefix2, value2, this.region)); + System.out.println("Checking values for key: " + keyPrefix3); + assertEquals("Got back incorrect number of rows from scan", 3, + getNumberOfRows(keyPrefix3, value2, this.region)); + deleteColumns(this.region, value2, keyPrefix1); + deleteColumns(this.region, value2, keyPrefix2); + deleteColumns(this.region, value2, keyPrefix3); + System.out.println("Starting important checks....."); + assertEquals("Got back incorrect number of rows from scan: " + keyPrefix1, + 0, getNumberOfRows(keyPrefix1, value2, this.region)); + assertEquals("Got back incorrect number of rows from scan: " + keyPrefix2, + 0, getNumberOfRows(keyPrefix2, value2, this.region)); + assertEquals("Got back incorrect number of rows from scan: " + keyPrefix3, + 0, getNumberOfRows(keyPrefix3, value2, this.region)); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } + } + + public void testAppendWithReadOnlyTable() throws Exception { + byte[] TABLE = Bytes.toBytes("readOnlyTable"); + this.region = initHRegion(TABLE, getName(), conf, true, Bytes.toBytes("somefamily")); + boolean exceptionCaught = false; + Append append = new Append(Bytes.toBytes("somerow")); + append.add(Bytes.toBytes("somefamily"), Bytes.toBytes("somequalifier"), + Bytes.toBytes("somevalue")); + try { + region.append(append, false); + } catch (IOException e) { + exceptionCaught = true; + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } + assertTrue(exceptionCaught == true); + } + + public void testIncrWithReadOnlyTable() throws Exception { + byte[] TABLE = Bytes.toBytes("readOnlyTable"); + this.region = initHRegion(TABLE, getName(), conf, true, Bytes.toBytes("somefamily")); + boolean exceptionCaught = false; + Increment inc = new Increment(Bytes.toBytes("somerow")); + inc.addColumn(Bytes.toBytes("somefamily"), Bytes.toBytes("somequalifier"), 1L); + try { + region.increment(inc, false); + } catch (IOException e) { + exceptionCaught = true; + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } + assertTrue(exceptionCaught == true); } private void deleteColumns(HRegion r, String value, String keyPrefix) @@ -466,17 +639,22 @@ private void putRows(HRegion r, int numRows, String value, String key) public void testFamilyWithAndWithoutColon() throws Exception { byte [] b = Bytes.toBytes(getName()); byte [] cf = Bytes.toBytes(COLUMN_FAMILY); - initHRegion(b, getName(), cf); - Put p = new Put(b); - byte [] cfwithcolon = Bytes.toBytes(COLUMN_FAMILY + ":"); - p.add(cfwithcolon, cfwithcolon, cfwithcolon); - boolean exception = false; + this.region = initHRegion(b, getName(), conf, cf); try { - this.region.put(p); - } catch (DoNotRetryIOException e) { - exception = true; + Put p = new Put(b); + byte [] cfwithcolon = Bytes.toBytes(COLUMN_FAMILY + ":"); + p.add(cfwithcolon, cfwithcolon, cfwithcolon); + boolean exception = false; + try { + this.region.put(p); + } catch (NoSuchColumnFamilyException e) { + exception = true; + } + assertTrue(exception); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } - assertTrue(exception); } @SuppressWarnings("unchecked") @@ -485,96 +663,137 @@ public void testBatchPut() throws Exception { byte[] cf = Bytes.toBytes(COLUMN_FAMILY); byte[] qual = Bytes.toBytes("qual"); byte[] val = Bytes.toBytes("val"); - initHRegion(b, getName(), cf); + this.region = initHRegion(b, getName(), conf, cf); + try { + HLog.getSyncTime(); // clear counter from prior tests + assertEquals(0, HLog.getSyncTime().count); + + LOG.info("First a batch put with all valid puts"); + final Put[] puts = new Put[10]; + for (int i = 0; i < 10; i++) { + puts[i] = new Put(Bytes.toBytes("row_" + i)); + puts[i].add(cf, qual, val); + } - HLog.getSyncTime(); // clear counter from prior tests - assertEquals(0, HLog.getSyncTime().count); + OperationStatus[] codes = this.region.put(puts); + assertEquals(10, codes.length); + for (int i = 0; i < 10; i++) { + assertEquals(OperationStatusCode.SUCCESS, codes[i] + .getOperationStatusCode()); + } + assertEquals(1, HLog.getSyncTime().count); + + LOG.info("Next a batch put with one invalid family"); + puts[5].add(Bytes.toBytes("BAD_CF"), qual, val); + codes = this.region.put(puts); + assertEquals(10, codes.length); + for (int i = 0; i < 10; i++) { + assertEquals((i == 5) ? OperationStatusCode.BAD_FAMILY : + OperationStatusCode.SUCCESS, codes[i].getOperationStatusCode()); + } + assertEquals(1, HLog.getSyncTime().count); - LOG.info("First a batch put with all valid puts"); - final Put[] puts = new Put[10]; - for (int i = 0; i < 10; i++) { - puts[i] = new Put(Bytes.toBytes("row_" + i)); - puts[i].add(cf, qual, val); - } + LOG.info("Next a batch put that has to break into two batches to avoid a lock"); + Integer lockedRow = region.obtainRowLock(Bytes.toBytes("row_2")); - OperationStatus[] codes = this.region.put(puts); - assertEquals(10, codes.length); - for (int i = 0; i < 10; i++) { - assertEquals(OperationStatusCode.SUCCESS, codes[i] - .getOperationStatusCode()); + MultithreadedTestUtil.TestContext ctx = + new MultithreadedTestUtil.TestContext(conf); + final AtomicReference retFromThread = + new AtomicReference(); + TestThread putter = new TestThread(ctx) { + @Override + public void doWork() throws IOException { + retFromThread.set(region.put(puts)); + } + }; + LOG.info("...starting put thread while holding lock"); + ctx.addThread(putter); + ctx.startThreads(); + + LOG.info("...waiting for put thread to sync first time"); + long startWait = System.currentTimeMillis(); + while (HLog.getSyncTime().count == 0) { + Thread.sleep(100); + if (System.currentTimeMillis() - startWait > 10000) { + fail("Timed out waiting for thread to sync first minibatch"); + } + } + LOG.info("...releasing row lock, which should let put thread continue"); + region.releaseRowLock(lockedRow); + LOG.info("...joining on thread"); + ctx.stop(); + LOG.info("...checking that next batch was synced"); + assertEquals(1, HLog.getSyncTime().count); + codes = retFromThread.get(); + for (int i = 0; i < 10; i++) { + assertEquals((i == 5) ? OperationStatusCode.BAD_FAMILY : + OperationStatusCode.SUCCESS, codes[i].getOperationStatusCode()); + } + + LOG.info("Nexta, a batch put which uses an already-held lock"); + lockedRow = region.obtainRowLock(Bytes.toBytes("row_2")); + LOG.info("...obtained row lock"); + List> putsAndLocks = Lists.newArrayList(); + for (int i = 0; i < 10; i++) { + Pair pair = new Pair(puts[i], null); + if (i == 2) pair.setSecond(lockedRow); + putsAndLocks.add(pair); + } + + codes = region.put(putsAndLocks.toArray(new Pair[0])); + LOG.info("...performed put"); + for (int i = 0; i < 10; i++) { + assertEquals((i == 5) ? OperationStatusCode.BAD_FAMILY : + OperationStatusCode.SUCCESS, codes[i].getOperationStatusCode()); + } + // Make sure we didn't do an extra batch + assertEquals(1, HLog.getSyncTime().count); + + // Make sure we still hold lock + assertTrue(region.isRowLocked(lockedRow)); + LOG.info("...releasing lock"); + region.releaseRowLock(lockedRow); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } - assertEquals(1, HLog.getSyncTime().count); + } - LOG.info("Next a batch put with one invalid family"); - puts[5].add(Bytes.toBytes("BAD_CF"), qual, val); - codes = this.region.put(puts); - assertEquals(10, codes.length); - for (int i = 0; i < 10; i++) { - assertEquals((i == 5) ? OperationStatusCode.SANITY_CHECK_FAILURE : - OperationStatusCode.SUCCESS, codes[i].getOperationStatusCode()); - } - assertEquals(1, HLog.getSyncTime().count); + public void testBatchPutWithTsSlop() throws Exception { + byte[] b = Bytes.toBytes(getName()); + byte[] cf = Bytes.toBytes(COLUMN_FAMILY); + byte[] qual = Bytes.toBytes("qual"); + byte[] val = Bytes.toBytes("val"); + Configuration conf = HBaseConfiguration.create(this.conf); - LOG.info("Next a batch put that has to break into two batches to avoid a lock"); - Integer lockedRow = region.obtainRowLock(Bytes.toBytes("row_2")); + // add data with a timestamp that is too recent for range. Ensure assert + conf.setInt("hbase.hregion.keyvalue.timestamp.slop.millisecs", 1000); + this.region = initHRegion(b, getName(), conf, cf); - MultithreadedTestUtil.TestContext ctx = - new MultithreadedTestUtil.TestContext(HBaseConfiguration.create()); - final AtomicReference retFromThread = - new AtomicReference(); - TestThread putter = new TestThread(ctx) { - @Override - public void doWork() throws IOException { - retFromThread.set(region.put(puts)); + try{ + HLog.getSyncTime(); // clear counter from prior tests + assertEquals(0, HLog.getSyncTime().count); + + final Put[] puts = new Put[10]; + for (int i = 0; i < 10; i++) { + puts[i] = new Put(Bytes.toBytes("row_" + i), Long.MAX_VALUE - 100); + puts[i].add(cf, qual, val); } - }; - LOG.info("...starting put thread while holding lock"); - ctx.addThread(putter); - ctx.startThreads(); - - LOG.info("...waiting for put thread to sync first time"); - long startWait = System.currentTimeMillis(); - while (HLog.getSyncTime().count == 0) { - Thread.sleep(100); - if (System.currentTimeMillis() - startWait > 10000) { - fail("Timed out waiting for thread to sync first minibatch"); - } - } - LOG.info("...releasing row lock, which should let put thread continue"); - region.releaseRowLock(lockedRow); - LOG.info("...joining on thread"); - ctx.stop(); - LOG.info("...checking that next batch was synced"); - assertEquals(1, HLog.getSyncTime().count); - codes = retFromThread.get(); - for (int i = 0; i < 10; i++) { - assertEquals((i == 5) ? OperationStatusCode.SANITY_CHECK_FAILURE : - OperationStatusCode.SUCCESS, codes[i].getOperationStatusCode()); - } - - LOG.info("Nexta, a batch put which uses an already-held lock"); - lockedRow = region.obtainRowLock(Bytes.toBytes("row_2")); - LOG.info("...obtained row lock"); - List> putsAndLocks = Lists.newArrayList(); - for (int i = 0; i < 10; i++) { - Pair pair = new Pair(puts[i], null); - if (i == 2) pair.setSecond(lockedRow); - putsAndLocks.add(pair); - } - - codes = region.put(putsAndLocks.toArray(new Pair[0])); - LOG.info("...performed put"); - for (int i = 0; i < 10; i++) { - assertEquals((i == 5) ? OperationStatusCode.SANITY_CHECK_FAILURE : - OperationStatusCode.SUCCESS, codes[i].getOperationStatusCode()); - } - // Make sure we didn't do an extra batch - assertEquals(1, HLog.getSyncTime().count); - - // Make sure we still hold lock - assertTrue(region.isRowLocked(lockedRow)); - LOG.info("...releasing lock"); - region.releaseRowLock(lockedRow); + + OperationStatus[] codes = this.region.put(puts); + assertEquals(10, codes.length); + for (int i = 0; i < 10; i++) { + assertEquals(OperationStatusCode.SANITY_CHECK_FAILURE, codes[i] + .getOperationStatusCode()); + } + assertEquals(0, HLog.getSyncTime().count); + + + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } + } ////////////////////////////////////////////////////////////////////////////// @@ -592,65 +811,68 @@ public void testCheckAndMutate_WithEmptyRowValue() throws IOException { //Setting up region String method = this.getName(); - initHRegion(tableName, method, fam1); - - //Putting empty data in key - Put put = new Put(row1); - put.add(fam1, qf1, emptyVal); - - //checkAndPut with empty value - boolean res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(emptyVal), put, lockId, true); - assertTrue(res); - - //Putting data in key - put = new Put(row1); - put.add(fam1, qf1, val1); - - //checkAndPut with correct value - res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(emptyVal), put, lockId, true); - assertTrue(res); - - // not empty anymore - res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(emptyVal), put, lockId, true); - assertFalse(res); - - Delete delete = new Delete(row1); - delete.deleteColumn(fam1, qf1); - res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(emptyVal), delete, lockId, true); - assertFalse(res); - - put = new Put(row1); - put.add(fam1, qf1, val2); - //checkAndPut with correct value - res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(val1), put, lockId, true); - assertTrue(res); - - //checkAndDelete with correct value - delete = new Delete(row1); - delete.deleteColumn(fam1, qf1); - delete.deleteColumn(fam1, qf1); - res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(val2), delete, lockId, true); - assertTrue(res); - - delete = new Delete(row1); - res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(emptyVal), delete, lockId, true); - assertTrue(res); - - //checkAndPut looking for a null value - put = new Put(row1); - put.add(fam1, qf1, val1); - - res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new NullComparator(), put, lockId, true); - assertTrue(res); - + this.region = initHRegion(tableName, method, conf, fam1); + try { + //Putting empty data in key + Put put = new Put(row1); + put.add(fam1, qf1, emptyVal); + + //checkAndPut with empty value + boolean res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(emptyVal), put, lockId, true); + assertTrue(res); + + //Putting data in key + put = new Put(row1); + put.add(fam1, qf1, val1); + + //checkAndPut with correct value + res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(emptyVal), put, lockId, true); + assertTrue(res); + + // not empty anymore + res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(emptyVal), put, lockId, true); + assertFalse(res); + + Delete delete = new Delete(row1); + delete.deleteColumn(fam1, qf1); + res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(emptyVal), delete, lockId, true); + assertFalse(res); + + put = new Put(row1); + put.add(fam1, qf1, val2); + //checkAndPut with correct value + res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(val1), put, lockId, true); + assertTrue(res); + + //checkAndDelete with correct value + delete = new Delete(row1); + delete.deleteColumn(fam1, qf1); + delete.deleteColumn(fam1, qf1); + res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(val2), delete, lockId, true); + assertTrue(res); + + delete = new Delete(row1); + res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(emptyVal), delete, lockId, true); + assertTrue(res); + + //checkAndPut looking for a null value + put = new Put(row1); + put.add(fam1, qf1, val1); + + res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new NullComparator(), put, lockId, true); + assertTrue(res); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testCheckAndMutate_WithWrongValue() throws IOException{ @@ -664,24 +886,28 @@ public void testCheckAndMutate_WithWrongValue() throws IOException{ //Setting up region String method = this.getName(); - initHRegion(tableName, method, fam1); - - //Putting data in key - Put put = new Put(row1); - put.add(fam1, qf1, val1); - region.put(put); - - //checkAndPut with wrong value - boolean res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(val2), put, lockId, true); - assertEquals(false, res); + this.region = initHRegion(tableName, method, conf, fam1); + try { + //Putting data in key + Put put = new Put(row1); + put.add(fam1, qf1, val1); + region.put(put); - //checkAndDelete with wrong value - Delete delete = new Delete(row1); - delete.deleteFamily(fam1); - res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(val2), delete, lockId, true); - assertEquals(false, res); + //checkAndPut with wrong value + boolean res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(val2), put, lockId, true); + assertEquals(false, res); + + //checkAndDelete with wrong value + Delete delete = new Delete(row1); + delete.deleteFamily(fam1); + res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(val2), delete, lockId, true); + assertEquals(false, res); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testCheckAndMutate_WithCorrectValue() throws IOException{ @@ -694,24 +920,28 @@ public void testCheckAndMutate_WithCorrectValue() throws IOException{ //Setting up region String method = this.getName(); - initHRegion(tableName, method, fam1); - - //Putting data in key - Put put = new Put(row1); - put.add(fam1, qf1, val1); - region.put(put); - - //checkAndPut with correct value - boolean res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(val1), put, lockId, true); - assertEquals(true, res); + this.region = initHRegion(tableName, method, conf, fam1); + try { + //Putting data in key + Put put = new Put(row1); + put.add(fam1, qf1, val1); + region.put(put); - //checkAndDelete with correct value - Delete delete = new Delete(row1); - delete.deleteColumn(fam1, qf1); - res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(val1), put, lockId, true); - assertEquals(true, res); + //checkAndPut with correct value + boolean res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(val1), put, lockId, true); + assertEquals(true, res); + + //checkAndDelete with correct value + Delete delete = new Delete(row1); + delete.deleteColumn(fam1, qf1); + res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(val1), put, lockId, true); + assertEquals(true, res); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testCheckAndPut_ThatPutWasWritten() throws IOException{ @@ -728,55 +958,82 @@ public void testCheckAndPut_ThatPutWasWritten() throws IOException{ //Setting up region String method = this.getName(); - initHRegion(tableName, method, families); - - //Putting data in the key to check - Put put = new Put(row1); - put.add(fam1, qf1, val1); - region.put(put); + this.region = initHRegion(tableName, method, conf, families); + try { + //Putting data in the key to check + Put put = new Put(row1); + put.add(fam1, qf1, val1); + region.put(put); - //Creating put to add - long ts = System.currentTimeMillis(); - KeyValue kv = new KeyValue(row1, fam2, qf1, ts, KeyValue.Type.Put, val2); - put = new Put(row1); - put.add(kv); + //Creating put to add + long ts = System.currentTimeMillis(); + KeyValue kv = new KeyValue(row1, fam2, qf1, ts, KeyValue.Type.Put, val2); + put = new Put(row1); + put.add(kv); - //checkAndPut with wrong value - Store store = region.getStore(fam1); - store.memstore.kvset.size(); + //checkAndPut with wrong value + Store store = region.getStore(fam1); + store.memstore.kvset.size(); - boolean res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, - new BinaryComparator(val1), put, lockId, true); - assertEquals(true, res); - store.memstore.kvset.size(); + boolean res = region.checkAndMutate(row1, fam1, qf1, CompareOp.EQUAL, + new BinaryComparator(val1), put, lockId, true); + assertEquals(true, res); + store.memstore.kvset.size(); - Get get = new Get(row1); - get.addColumn(fam2, qf1); - KeyValue [] actual = region.get(get, null).raw(); + Get get = new Get(row1); + get.addColumn(fam2, qf1); + KeyValue [] actual = region.get(get, null).raw(); - KeyValue [] expected = {kv}; + KeyValue [] expected = {kv}; - assertEquals(expected.length, actual.length); - for(int i=0; i kvs = new ArrayList(); - kvs.add(new KeyValue(row1, fam4, null, null)); + this.region = initHRegion(tableName, method, conf, fam1, fam2, fam3); + try { + List kvs = new ArrayList(); + kvs.add(new KeyValue(row1, fam4, null, null)); - //testing existing family - byte [] family = fam2; - try { - Map> deleteMap = new HashMap>(); - deleteMap.put(family, kvs); - region.delete(deleteMap, HConstants.DEFAULT_CLUSTER_ID, true); - } catch (Exception e) { - assertTrue("Family " +new String(family)+ " does not exist", false); - } + //testing existing family + byte [] family = fam2; + try { + Map> deleteMap = new HashMap>(); + deleteMap.put(family, kvs); + region.delete(deleteMap, HConstants.DEFAULT_CLUSTER_ID, true); + } catch (Exception e) { + assertTrue("Family " +new String(family)+ " does not exist", false); + } - //testing non existing family - boolean ok = false; - family = fam4; - try { - Map> deleteMap = new HashMap>(); - deleteMap.put(family, kvs); - region.delete(deleteMap, HConstants.DEFAULT_CLUSTER_ID, true); - } catch (Exception e) { - ok = true; + //testing non existing family + boolean ok = false; + family = fam4; + try { + Map> deleteMap = new HashMap>(); + deleteMap.put(family, kvs); + region.delete(deleteMap, HConstants.DEFAULT_CLUSTER_ID, true); + } catch (Exception e) { + ok = true; + } + assertEquals("Family " +new String(family)+ " does exist", true, ok); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } - assertEquals("Family " +new String(family)+ " does exist", true, ok); } public void testDelete_mixed() throws IOException, InterruptedException { @@ -926,62 +1195,67 @@ public void testDelete_mixed() throws IOException, InterruptedException { byte [] fam = Bytes.toBytes("info"); byte [][] families = {fam}; String method = this.getName(); - initHRegion(tableName, method, families); - EnvironmentEdgeManagerTestHelper.injectEdge(new IncrementingEnvironmentEdge()); + this.region = initHRegion(tableName, method, conf, families); + try { + EnvironmentEdgeManagerTestHelper.injectEdge(new IncrementingEnvironmentEdge()); - byte [] row = Bytes.toBytes("table_name"); - // column names - byte [] serverinfo = Bytes.toBytes("serverinfo"); - byte [] splitA = Bytes.toBytes("splitA"); - byte [] splitB = Bytes.toBytes("splitB"); + byte [] row = Bytes.toBytes("table_name"); + // column names + byte [] serverinfo = Bytes.toBytes("serverinfo"); + byte [] splitA = Bytes.toBytes("splitA"); + byte [] splitB = Bytes.toBytes("splitB"); - // add some data: - Put put = new Put(row); - put.add(fam, splitA, Bytes.toBytes("reference_A")); - region.put(put); + // add some data: + Put put = new Put(row); + put.add(fam, splitA, Bytes.toBytes("reference_A")); + region.put(put); - put = new Put(row); - put.add(fam, splitB, Bytes.toBytes("reference_B")); - region.put(put); + put = new Put(row); + put.add(fam, splitB, Bytes.toBytes("reference_B")); + region.put(put); - put = new Put(row); - put.add(fam, serverinfo, Bytes.toBytes("ip_address")); - region.put(put); + put = new Put(row); + put.add(fam, serverinfo, Bytes.toBytes("ip_address")); + region.put(put); - // ok now delete a split: - Delete delete = new Delete(row); - delete.deleteColumns(fam, splitA); - region.delete(delete, null, true); + // ok now delete a split: + Delete delete = new Delete(row); + delete.deleteColumns(fam, splitA); + region.delete(delete, null, true); - // assert some things: - Get get = new Get(row).addColumn(fam, serverinfo); - Result result = region.get(get, null); - assertEquals(1, result.size()); + // assert some things: + Get get = new Get(row).addColumn(fam, serverinfo); + Result result = region.get(get, null); + assertEquals(1, result.size()); - get = new Get(row).addColumn(fam, splitA); - result = region.get(get, null); - assertEquals(0, result.size()); + get = new Get(row).addColumn(fam, splitA); + result = region.get(get, null); + assertEquals(0, result.size()); - get = new Get(row).addColumn(fam, splitB); - result = region.get(get, null); - assertEquals(1, result.size()); + get = new Get(row).addColumn(fam, splitB); + result = region.get(get, null); + assertEquals(1, result.size()); - // Assert that after a delete, I can put. - put = new Put(row); - put.add(fam, splitA, Bytes.toBytes("reference_A")); - region.put(put); - get = new Get(row); - result = region.get(get, null); - assertEquals(3, result.size()); + // Assert that after a delete, I can put. + put = new Put(row); + put.add(fam, splitA, Bytes.toBytes("reference_A")); + region.put(put); + get = new Get(row); + result = region.get(get, null); + assertEquals(3, result.size()); - // Now delete all... then test I can add stuff back - delete = new Delete(row); - region.delete(delete, null, false); - assertEquals(0, region.get(get, null).size()); + // Now delete all... then test I can add stuff back + delete = new Delete(row); + region.delete(delete, null, false); + assertEquals(0, region.get(get, null).size()); - region.put(new Put(row).add(fam, splitA, Bytes.toBytes("reference_A"))); - result = region.get(get, null); - assertEquals(1, result.size()); + region.put(new Put(row).add(fam, splitA, Bytes.toBytes("reference_A"))); + result = region.get(get, null); + assertEquals(1, result.size()); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testDeleteRowWithFutureTs() throws IOException { @@ -989,34 +1263,38 @@ public void testDeleteRowWithFutureTs() throws IOException { byte [] fam = Bytes.toBytes("info"); byte [][] families = {fam}; String method = this.getName(); - initHRegion(tableName, method, families); + this.region = initHRegion(tableName, method, conf, families); + try { + byte [] row = Bytes.toBytes("table_name"); + // column names + byte [] serverinfo = Bytes.toBytes("serverinfo"); - byte [] row = Bytes.toBytes("table_name"); - // column names - byte [] serverinfo = Bytes.toBytes("serverinfo"); + // add data in the far future + Put put = new Put(row); + put.add(fam, serverinfo, HConstants.LATEST_TIMESTAMP-5,Bytes.toBytes("value")); + region.put(put); - // add data in the far future - Put put = new Put(row); - put.add(fam, serverinfo, HConstants.LATEST_TIMESTAMP-5,Bytes.toBytes("value")); - region.put(put); + // now delete something in the present + Delete delete = new Delete(row); + region.delete(delete, null, true); - // now delete something in the present - Delete delete = new Delete(row); - region.delete(delete, null, true); + // make sure we still see our data + Get get = new Get(row).addColumn(fam, serverinfo); + Result result = region.get(get, null); + assertEquals(1, result.size()); - // make sure we still see our data - Get get = new Get(row).addColumn(fam, serverinfo); - Result result = region.get(get, null); - assertEquals(1, result.size()); + // delete the future row + delete = new Delete(row,HConstants.LATEST_TIMESTAMP-3,null); + region.delete(delete, null, true); - // delete the future row - delete = new Delete(row,HConstants.LATEST_TIMESTAMP-3,null); - region.delete(delete, null, true); - - // make sure it is gone - get = new Get(row).addColumn(fam, serverinfo); - result = region.get(get, null); - assertEquals(0, result.size()); + // make sure it is gone + get = new Get(row).addColumn(fam, serverinfo); + result = region.get(get, null); + assertEquals(0, result.size()); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } /** @@ -1028,44 +1306,49 @@ public void testPutWithLatestTS() throws IOException { byte [] fam = Bytes.toBytes("info"); byte [][] families = {fam}; String method = this.getName(); - initHRegion(tableName, method, families); - - byte [] row = Bytes.toBytes("row1"); - // column names - byte [] qual = Bytes.toBytes("qual"); - - // add data with LATEST_TIMESTAMP, put without WAL - Put put = new Put(row); - put.add(fam, qual, HConstants.LATEST_TIMESTAMP, Bytes.toBytes("value")); - region.put(put, false); - - // Make sure it shows up with an actual timestamp - Get get = new Get(row).addColumn(fam, qual); - Result result = region.get(get, null); - assertEquals(1, result.size()); - KeyValue kv = result.raw()[0]; - LOG.info("Got: " + kv); - assertTrue("LATEST_TIMESTAMP was not replaced with real timestamp", - kv.getTimestamp() != HConstants.LATEST_TIMESTAMP); - - // Check same with WAL enabled (historically these took different - // code paths, so check both) - row = Bytes.toBytes("row2"); - put = new Put(row); - put.add(fam, qual, HConstants.LATEST_TIMESTAMP, Bytes.toBytes("value")); - region.put(put, true); + this.region = initHRegion(tableName, method, conf, families); + try { + byte [] row = Bytes.toBytes("row1"); + // column names + byte [] qual = Bytes.toBytes("qual"); - // Make sure it shows up with an actual timestamp - get = new Get(row).addColumn(fam, qual); - result = region.get(get, null); - assertEquals(1, result.size()); - kv = result.raw()[0]; - LOG.info("Got: " + kv); - assertTrue("LATEST_TIMESTAMP was not replaced with real timestamp", - kv.getTimestamp() != HConstants.LATEST_TIMESTAMP); + // add data with LATEST_TIMESTAMP, put without WAL + Put put = new Put(row); + put.add(fam, qual, HConstants.LATEST_TIMESTAMP, Bytes.toBytes("value")); + region.put(put, false); + + // Make sure it shows up with an actual timestamp + Get get = new Get(row).addColumn(fam, qual); + Result result = region.get(get, null); + assertEquals(1, result.size()); + KeyValue kv = result.raw()[0]; + LOG.info("Got: " + kv); + assertTrue("LATEST_TIMESTAMP was not replaced with real timestamp", + kv.getTimestamp() != HConstants.LATEST_TIMESTAMP); + + // Check same with WAL enabled (historically these took different + // code paths, so check both) + row = Bytes.toBytes("row2"); + put = new Put(row); + put.add(fam, qual, HConstants.LATEST_TIMESTAMP, Bytes.toBytes("value")); + region.put(put, true); + + // Make sure it shows up with an actual timestamp + get = new Get(row).addColumn(fam, qual); + result = region.get(get, null); + assertEquals(1, result.size()); + kv = result.raw()[0]; + LOG.info("Got: " + kv); + assertTrue("LATEST_TIMESTAMP was not replaced with real timestamp", + kv.getTimestamp() != HConstants.LATEST_TIMESTAMP); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } + /** * Tests that there is server-side filtering for invalid timestamp upper * bound. Note that the timestamp lower bound is automatically handled for us @@ -1076,22 +1359,30 @@ public void testPutWithTsSlop() throws IOException { byte[] fam = Bytes.toBytes("info"); byte[][] families = { fam }; String method = this.getName(); - HBaseConfiguration conf = new HBaseConfiguration(); + Configuration conf = HBaseConfiguration.create(this.conf); // add data with a timestamp that is too recent for range. Ensure assert conf.setInt("hbase.hregion.keyvalue.timestamp.slop.millisecs", 1000); - initHRegion(tableName, method, conf, families); + this.region = initHRegion(tableName, method, conf, families); + boolean caughtExcep = false; try { - // no TS specified == use latest. should not error - region.put(new Put(row).add(fam, Bytes.toBytes("qual"), Bytes - .toBytes("value")), false); - // TS out of range. should error - region.put(new Put(row).add(fam, Bytes.toBytes("qual"), - System.currentTimeMillis() + 2000, - Bytes.toBytes("value")), false); - fail("Expected IOE for TS out of configured timerange"); - } catch (DoNotRetryIOException ioe) { - LOG.debug("Received expected exception", ioe); + try { + // no TS specified == use latest. should not error + region.put(new Put(row).add(fam, Bytes.toBytes("qual"), Bytes + .toBytes("value")), false); + // TS out of range. should error + region.put(new Put(row).add(fam, Bytes.toBytes("qual"), + System.currentTimeMillis() + 2000, + Bytes.toBytes("value")), false); + fail("Expected IOE for TS out of configured timerange"); + } catch (DoNotRetryIOException ioe) { + LOG.debug("Received expected exception", ioe); + caughtExcep = true; + } + assertTrue("Should catch FailedSanityCheckException", caughtExcep); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -1099,39 +1390,42 @@ public void testScanner_DeleteOneFamilyNotAnother() throws IOException { byte [] tableName = Bytes.toBytes("test_table"); byte [] fam1 = Bytes.toBytes("columnA"); byte [] fam2 = Bytes.toBytes("columnB"); - initHRegion(tableName, getName(), fam1, fam2); - - byte [] rowA = Bytes.toBytes("rowA"); - byte [] rowB = Bytes.toBytes("rowB"); - - byte [] value = Bytes.toBytes("value"); + this.region = initHRegion(tableName, getName(), conf, fam1, fam2); + try { + byte [] rowA = Bytes.toBytes("rowA"); + byte [] rowB = Bytes.toBytes("rowB"); - Delete delete = new Delete(rowA); - delete.deleteFamily(fam1); + byte [] value = Bytes.toBytes("value"); - region.delete(delete, null, true); + Delete delete = new Delete(rowA); + delete.deleteFamily(fam1); - // now create data. - Put put = new Put(rowA); - put.add(fam2, null, value); - region.put(put); + region.delete(delete, null, true); - put = new Put(rowB); - put.add(fam1, null, value); - put.add(fam2, null, value); - region.put(put); + // now create data. + Put put = new Put(rowA); + put.add(fam2, null, value); + region.put(put); - Scan scan = new Scan(); - scan.addFamily(fam1).addFamily(fam2); - InternalScanner s = region.getScanner(scan); - List results = new ArrayList(); - s.next(results); - assertTrue(Bytes.equals(rowA, results.get(0).getRow())); + put = new Put(rowB); + put.add(fam1, null, value); + put.add(fam2, null, value); + region.put(put); - results.clear(); - s.next(results); - assertTrue(Bytes.equals(rowB, results.get(0).getRow())); + Scan scan = new Scan(); + scan.addFamily(fam1).addFamily(fam2); + InternalScanner s = region.getScanner(scan); + List results = new ArrayList(); + s.next(results); + assertTrue(Bytes.equals(rowA, results.get(0).getRow())); + results.clear(); + s.next(results); + assertTrue(Bytes.equals(rowB, results.get(0).getRow())); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testDeleteColumns_PostInsert() throws IOException, @@ -1149,47 +1443,50 @@ public void testDeleteFamily_PostInsert() throws IOException, InterruptedExcepti public void doTestDelete_AndPostInsert(Delete delete) throws IOException, InterruptedException { - initHRegion(tableName, getName(), fam1); - EnvironmentEdgeManagerTestHelper.injectEdge(new IncrementingEnvironmentEdge()); - Put put = new Put(row); - put.add(fam1, qual1, value1); - region.put(put); - - // now delete the value: - region.delete(delete, null, true); - - - // ok put data: - put = new Put(row); - put.add(fam1, qual1, value2); - region.put(put); - - // ok get: - Get get = new Get(row); - get.addColumn(fam1, qual1); + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + EnvironmentEdgeManagerTestHelper.injectEdge(new IncrementingEnvironmentEdge()); + Put put = new Put(row); + put.add(fam1, qual1, value1); + region.put(put); - Result r = region.get(get, null); - assertEquals(1, r.size()); - assertByteEquals(value2, r.getValue(fam1, qual1)); + // now delete the value: + region.delete(delete, null, true); - // next: - Scan scan = new Scan(row); - scan.addColumn(fam1, qual1); - InternalScanner s = region.getScanner(scan); - List results = new ArrayList(); - assertEquals(false, s.next(results)); - assertEquals(1, results.size()); - KeyValue kv = results.get(0); + // ok put data: + put = new Put(row); + put.add(fam1, qual1, value2); + region.put(put); - assertByteEquals(value2, kv.getValue()); - assertByteEquals(fam1, kv.getFamily()); - assertByteEquals(qual1, kv.getQualifier()); - assertByteEquals(row, kv.getRow()); + // ok get: + Get get = new Get(row); + get.addColumn(fam1, qual1); + + Result r = region.get(get, null); + assertEquals(1, r.size()); + assertByteEquals(value2, r.getValue(fam1, qual1)); + + // next: + Scan scan = new Scan(row); + scan.addColumn(fam1, qual1); + InternalScanner s = region.getScanner(scan); + + List results = new ArrayList(); + assertEquals(false, s.next(results)); + assertEquals(1, results.size()); + KeyValue kv = results.get(0); + + assertByteEquals(value2, kv.getValue()); + assertByteEquals(fam1, kv.getFamily()); + assertByteEquals(qual1, kv.getQualifier()); + assertByteEquals(row, kv.getRow()); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } - - public void testDelete_CheckTimestampUpdated() throws IOException { byte [] row1 = Bytes.toBytes("row1"); @@ -1199,27 +1496,31 @@ public void testDelete_CheckTimestampUpdated() //Setting up region String method = this.getName(); - initHRegion(tableName, method, fam1); - - //Building checkerList - List kvs = new ArrayList(); - kvs.add(new KeyValue(row1, fam1, col1, null)); - kvs.add(new KeyValue(row1, fam1, col2, null)); - kvs.add(new KeyValue(row1, fam1, col3, null)); + this.region = initHRegion(tableName, method, conf, fam1); + try { + //Building checkerList + List kvs = new ArrayList(); + kvs.add(new KeyValue(row1, fam1, col1, null)); + kvs.add(new KeyValue(row1, fam1, col2, null)); + kvs.add(new KeyValue(row1, fam1, col3, null)); - Map> deleteMap = new HashMap>(); - deleteMap.put(fam1, kvs); - region.delete(deleteMap, HConstants.DEFAULT_CLUSTER_ID, true); + Map> deleteMap = new HashMap>(); + deleteMap.put(fam1, kvs); + region.delete(deleteMap, HConstants.DEFAULT_CLUSTER_ID, true); - // extract the key values out the memstore: - // This is kinda hacky, but better than nothing... - long now = System.currentTimeMillis(); - KeyValue firstKv = region.getStore(fam1).memstore.kvset.first(); - assertTrue(firstKv.getTimestamp() <= now); - now = firstKv.getTimestamp(); - for (KeyValue kv: region.getStore(fam1).memstore.kvset) { - assertTrue(kv.getTimestamp() <= now); - now = kv.getTimestamp(); + // extract the key values out the memstore: + // This is kinda hacky, but better than nothing... + long now = System.currentTimeMillis(); + KeyValue firstKv = region.getStore(fam1).memstore.kvset.first(); + assertTrue(firstKv.getTimestamp() <= now); + now = firstKv.getTimestamp(); + for (KeyValue kv: region.getStore(fam1).memstore.kvset) { + assertTrue(kv.getTimestamp() <= now); + now = kv.getTimestamp(); + } + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -1235,19 +1536,23 @@ public void testGet_FamilyChecker() throws IOException { //Setting up region String method = this.getName(); - initHRegion(tableName, method, fam1); - - Get get = new Get(row1); - get.addColumn(fam2, col1); - - //Test + this.region = initHRegion(tableName, method, conf, fam1); try { - region.get(get, null); - } catch (DoNotRetryIOException e) { - assertFalse(false); - return; + Get get = new Get(row1); + get.addColumn(fam2, col1); + + //Test + try { + region.get(get, null); + } catch (DoNotRetryIOException e) { + assertFalse(false); + return; + } + assertFalse(true); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } - assertFalse(true); } public void testGet_Basic() throws IOException { @@ -1262,44 +1567,48 @@ public void testGet_Basic() throws IOException { //Setting up region String method = this.getName(); - initHRegion(tableName, method, fam1); + this.region = initHRegion(tableName, method, conf, fam1); + try { + //Add to memstore + Put put = new Put(row1); + put.add(fam1, col1, null); + put.add(fam1, col2, null); + put.add(fam1, col3, null); + put.add(fam1, col4, null); + put.add(fam1, col5, null); + region.put(put); - //Add to memstore - Put put = new Put(row1); - put.add(fam1, col1, null); - put.add(fam1, col2, null); - put.add(fam1, col3, null); - put.add(fam1, col4, null); - put.add(fam1, col5, null); - region.put(put); + Get get = new Get(row1); + get.addColumn(fam1, col2); + get.addColumn(fam1, col4); + //Expected result + KeyValue kv1 = new KeyValue(row1, fam1, col2); + KeyValue kv2 = new KeyValue(row1, fam1, col4); + KeyValue [] expected = {kv1, kv2}; + + //Test + Result res = region.get(get, null); + assertEquals(expected.length, res.size()); + for(int i=0; i result = new ArrayList(); - s.next(result); + //test2 + res = region.get(get, null); + + assertEquals(expected.length, res.size()); + for(int i=0; i result = new ArrayList(); + s.next(result); + + assertEquals(expected.length, result.size()); + for(int i=0; ithreads = new ArrayList(threadCount); - for (int i = 0; i < threadCount; i++) { - threads.add(new Thread(Integer.toString(i)) { - @Override - public void run() { - Integer [] lockids = new Integer[lockCount]; - // Get locks. - for (int i = 0; i < lockCount; i++) { - try { - byte [] rowid = Bytes.toBytes(Integer.toString(i)); - lockids[i] = region.obtainRowLock(rowid); - assertEquals(rowid, region.getRowFromLock(lockids[i])); - LOG.debug(getName() + " locked " + Bytes.toString(rowid)); - } catch (IOException e) { - e.printStackTrace(); + this.region = initHRegion(tableName, method, hc, families); + try { + final int threadCount = 10; + final int lockCount = 10; + + Listthreads = new ArrayList(threadCount); + for (int i = 0; i < threadCount; i++) { + threads.add(new Thread(Integer.toString(i)) { + @Override + public void run() { + Integer [] lockids = new Integer[lockCount]; + // Get locks. + for (int i = 0; i < lockCount; i++) { + try { + byte [] rowid = Bytes.toBytes(Integer.toString(i)); + lockids[i] = region.obtainRowLock(rowid); + assertEquals(rowid, region.getRowFromLock(lockids[i])); + LOG.debug(getName() + " locked " + Bytes.toString(rowid)); + } catch (IOException e) { + e.printStackTrace(); + } } - } - LOG.debug(getName() + " set " + - Integer.toString(lockCount) + " locks"); + LOG.debug(getName() + " set " + + Integer.toString(lockCount) + " locks"); - // Abort outstanding locks. - for (int i = lockCount - 1; i >= 0; i--) { - region.releaseRowLock(lockids[i]); - LOG.debug(getName() + " unlocked " + i); + // Abort outstanding locks. + for (int i = lockCount - 1; i >= 0; i--) { + region.releaseRowLock(lockids[i]); + LOG.debug(getName() + " unlocked " + i); + } + LOG.debug(getName() + " released " + + Integer.toString(lockCount) + " locks"); } - LOG.debug(getName() + " released " + - Integer.toString(lockCount) + " locks"); - } - }); - } + }); + } - // Startup all our threads. - for (Thread t : threads) { - t.start(); - } + // Startup all our threads. + for (Thread t : threads) { + t.start(); + } - // Now wait around till all are done. - for (Thread t: threads) { - while (t.isAlive()) { - try { - Thread.sleep(1); - } catch (InterruptedException e) { - // Go around again. + // Now wait around till all are done. + for (Thread t: threads) { + while (t.isAlive()) { + try { + Thread.sleep(1); + } catch (InterruptedException e) { + // Go around again. + } } } + LOG.info("locks completed."); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } - LOG.info("locks completed."); } ////////////////////////////////////////////////////////////////////////////// @@ -1462,7 +1784,7 @@ public void testMerge() throws IOException { Configuration hc = initSplit(); //Setting up region String method = this.getName(); - initHRegion(tableName, method, hc, families); + this.region = initHRegion(tableName, method, hc, families); try { LOG.info("" + addContent(region, fam3)); region.flushcache(); @@ -1499,10 +1821,8 @@ public void testMerge() throws IOException { } } } finally { - if (region != null) { - region.close(); - region.getLog().closeAndDelete(); - } + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -1550,15 +1870,19 @@ public void testGetScanner_WithOkFamilies() throws IOException { //Setting up region String method = this.getName(); - initHRegion(tableName, method, families); - - Scan scan = new Scan(); - scan.addFamily(fam1); - scan.addFamily(fam2); + this.region = initHRegion(tableName, method, conf, families); try { - region.getScanner(scan); - } catch (Exception e) { - assertTrue("Families could not be found in Region", false); + Scan scan = new Scan(); + scan.addFamily(fam1); + scan.addFamily(fam2); + try { + region.getScanner(scan); + } catch (Exception e) { + assertTrue("Families could not be found in Region", false); + } + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -1571,17 +1895,21 @@ public void testGetScanner_WithNotOkFamilies() throws IOException { //Setting up region String method = this.getName(); - initHRegion(tableName, method, families); - - Scan scan = new Scan(); - scan.addFamily(fam2); - boolean ok = false; + this.region = initHRegion(tableName, method, conf, families); try { - region.getScanner(scan); - } catch (Exception e) { - ok = true; + Scan scan = new Scan(); + scan.addFamily(fam2); + boolean ok = false; + try { + region.getScanner(scan); + } catch (Exception e) { + ok = true; + } + assertTrue("Families could not be found in Region", ok); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } - assertTrue("Families could not be found in Region", ok); } public void testGetScanner_WithNoFamilies() throws IOException { @@ -1596,40 +1924,45 @@ public void testGetScanner_WithNoFamilies() throws IOException { //Setting up region String method = this.getName(); - initHRegion(tableName, method, families); - - - //Putting data in Region - Put put = new Put(row1); - put.add(fam1, null, null); - put.add(fam2, null, null); - put.add(fam3, null, null); - put.add(fam4, null, null); - region.put(put); - - Scan scan = null; - HRegion.RegionScannerImpl is = null; + this.region = initHRegion(tableName, method, conf, families); + try { - //Testing to see how many scanners that is produced by getScanner, starting - //with known number, 2 - current = 1 - scan = new Scan(); - scan.addFamily(fam2); - scan.addFamily(fam4); - is = (RegionScannerImpl) region.getScanner(scan); - MultiVersionConsistencyControl.resetThreadReadPoint(region.getMVCC()); - assertEquals(1, ((RegionScannerImpl)is).storeHeap.getHeap().size()); + //Putting data in Region + Put put = new Put(row1); + put.add(fam1, null, null); + put.add(fam2, null, null); + put.add(fam3, null, null); + put.add(fam4, null, null); + region.put(put); - scan = new Scan(); - is = (RegionScannerImpl) region.getScanner(scan); - MultiVersionConsistencyControl.resetThreadReadPoint(region.getMVCC()); - assertEquals(families.length -1, - ((RegionScannerImpl)is).storeHeap.getHeap().size()); + Scan scan = null; + HRegion.RegionScannerImpl is = null; + + //Testing to see how many scanners that is produced by getScanner, starting + //with known number, 2 - current = 1 + scan = new Scan(); + scan.addFamily(fam2); + scan.addFamily(fam4); + is = (RegionScannerImpl) region.getScanner(scan); + MultiVersionConsistencyControl.resetThreadReadPoint(region.getMVCC()); + assertEquals(1, ((RegionScannerImpl)is).storeHeap.getHeap().size()); + + scan = new Scan(); + is = (RegionScannerImpl) region.getScanner(scan); + MultiVersionConsistencyControl.resetThreadReadPoint(region.getMVCC()); + assertEquals(families.length -1, + ((RegionScannerImpl)is).storeHeap.getHeap().size()); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } /** * This method tests https://issues.apache.org/jira/browse/HBASE-2516. + * @throws IOException */ - public void testGetScanner_WithRegionClosed() { + public void testGetScanner_WithRegionClosed() throws IOException { byte[] tableName = Bytes.toBytes("testtable"); byte[] fam1 = Bytes.toBytes("fam1"); byte[] fam2 = Bytes.toBytes("fam2"); @@ -1639,20 +1972,25 @@ public void testGetScanner_WithRegionClosed() { //Setting up region String method = this.getName(); try { - initHRegion(tableName, method, families); + this.region = initHRegion(tableName, method, conf, families); } catch (IOException e) { e.printStackTrace(); fail("Got IOException during initHRegion, " + e.getMessage()); } - region.closed.set(true); try { - region.getScanner(null); - fail("Expected to get an exception during getScanner on a region that is closed"); - } catch (org.apache.hadoop.hbase.NotServingRegionException e) { - //this is the correct exception that is expected - } catch (IOException e) { - fail("Got wrong type of exception - should be a NotServingRegionException, but was an IOException: " - + e.getMessage()); + region.closed.set(true); + try { + region.getScanner(null); + fail("Expected to get an exception during getScanner on a region that is closed"); + } catch (org.apache.hadoop.hbase.NotServingRegionException e) { + //this is the correct exception that is expected + } catch (IOException e) { + fail("Got wrong type of exception - should be a NotServingRegionException, but was an IOException: " + + e.getMessage()); + } + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -1670,53 +2008,56 @@ public void testRegionScanner_Next() throws IOException { //Setting up region String method = this.getName(); - initHRegion(tableName, method, families); - - //Putting data in Region - Put put = null; - put = new Put(row1); - put.add(fam1, null, ts, null); - put.add(fam2, null, ts, null); - put.add(fam3, null, ts, null); - put.add(fam4, null, ts, null); - region.put(put); + this.region = initHRegion(tableName, method, conf, families); + try { + //Putting data in Region + Put put = null; + put = new Put(row1); + put.add(fam1, null, ts, null); + put.add(fam2, null, ts, null); + put.add(fam3, null, ts, null); + put.add(fam4, null, ts, null); + region.put(put); - put = new Put(row2); - put.add(fam1, null, ts, null); - put.add(fam2, null, ts, null); - put.add(fam3, null, ts, null); - put.add(fam4, null, ts, null); - region.put(put); + put = new Put(row2); + put.add(fam1, null, ts, null); + put.add(fam2, null, ts, null); + put.add(fam3, null, ts, null); + put.add(fam4, null, ts, null); + region.put(put); - Scan scan = new Scan(); - scan.addFamily(fam2); - scan.addFamily(fam4); - InternalScanner is = region.getScanner(scan); + Scan scan = new Scan(); + scan.addFamily(fam2); + scan.addFamily(fam4); + InternalScanner is = region.getScanner(scan); - List res = null; + List res = null; - //Result 1 - List expected1 = new ArrayList(); - expected1.add(new KeyValue(row1, fam2, null, ts, KeyValue.Type.Put, null)); - expected1.add(new KeyValue(row1, fam4, null, ts, KeyValue.Type.Put, null)); + //Result 1 + List expected1 = new ArrayList(); + expected1.add(new KeyValue(row1, fam2, null, ts, KeyValue.Type.Put, null)); + expected1.add(new KeyValue(row1, fam4, null, ts, KeyValue.Type.Put, null)); - res = new ArrayList(); - is.next(res); - for(int i=0; i(); + is.next(res); + for(int i=0; i expected2 = new ArrayList(); - expected2.add(new KeyValue(row2, fam2, null, ts, KeyValue.Type.Put, null)); - expected2.add(new KeyValue(row2, fam4, null, ts, KeyValue.Type.Put, null)); + //Result 2 + List expected2 = new ArrayList(); + expected2.add(new KeyValue(row2, fam2, null, ts, KeyValue.Type.Put, null)); + expected2.add(new KeyValue(row2, fam4, null, ts, KeyValue.Type.Put, null)); - res = new ArrayList(); - is.next(res); - for(int i=0; i(); + is.next(res); + for(int i=0; i expected = new ArrayList(); - expected.add(kv13); - expected.add(kv12); + //Expected + List expected = new ArrayList(); + expected.add(kv13); + expected.add(kv12); - Scan scan = new Scan(row1); - scan.addColumn(fam1, qf1); - scan.setMaxVersions(MAX_VERSIONS); - List actual = new ArrayList(); - InternalScanner scanner = region.getScanner(scan); + Scan scan = new Scan(row1); + scan.addColumn(fam1, qf1); + scan.setMaxVersions(MAX_VERSIONS); + List actual = new ArrayList(); + InternalScanner scanner = region.getScanner(scan); - boolean hasNext = scanner.next(actual); - assertEquals(false, hasNext); + boolean hasNext = scanner.next(actual); + assertEquals(false, hasNext); - //Verify result - for(int i=0; i expected = new ArrayList(); - expected.add(kv13); - expected.add(kv12); - expected.add(kv23); - expected.add(kv22); + this.region = initHRegion(tableName, method, conf, families); + try { + //Putting data in Region + Put put = null; + KeyValue kv13 = new KeyValue(row1, fam1, qf1, ts3, KeyValue.Type.Put, null); + KeyValue kv12 = new KeyValue(row1, fam1, qf1, ts2, KeyValue.Type.Put, null); + KeyValue kv11 = new KeyValue(row1, fam1, qf1, ts1, KeyValue.Type.Put, null); + + KeyValue kv23 = new KeyValue(row1, fam1, qf2, ts3, KeyValue.Type.Put, null); + KeyValue kv22 = new KeyValue(row1, fam1, qf2, ts2, KeyValue.Type.Put, null); + KeyValue kv21 = new KeyValue(row1, fam1, qf2, ts1, KeyValue.Type.Put, null); + + put = new Put(row1); + put.add(kv13); + put.add(kv12); + put.add(kv11); + put.add(kv23); + put.add(kv22); + put.add(kv21); + region.put(put); + region.flushcache(); - Scan scan = new Scan(row1); - scan.addColumn(fam1, qf1); - scan.addColumn(fam1, qf2); - scan.setMaxVersions(MAX_VERSIONS); - List actual = new ArrayList(); - InternalScanner scanner = region.getScanner(scan); + //Expected + List expected = new ArrayList(); + expected.add(kv13); + expected.add(kv12); + expected.add(kv23); + expected.add(kv22); + + Scan scan = new Scan(row1); + scan.addColumn(fam1, qf1); + scan.addColumn(fam1, qf2); + scan.setMaxVersions(MAX_VERSIONS); + List actual = new ArrayList(); + InternalScanner scanner = region.getScanner(scan); - boolean hasNext = scanner.next(actual); - assertEquals(false, hasNext); + boolean hasNext = scanner.next(actual); + assertEquals(false, hasNext); - //Verify result - for(int i=0; i expected = new ArrayList(); - expected.add(kv14); - expected.add(kv13); - expected.add(kv12); - expected.add(kv24); - expected.add(kv23); - expected.add(kv22); + put = new Put(row1); + put.add(kv21); + put.add(kv11); + region.put(put); - Scan scan = new Scan(row1); - scan.addColumn(fam1, qf1); - scan.addColumn(fam1, qf2); - int versions = 3; - scan.setMaxVersions(versions); - List actual = new ArrayList(); - InternalScanner scanner = region.getScanner(scan); + //Expected + List expected = new ArrayList(); + expected.add(kv14); + expected.add(kv13); + expected.add(kv12); + expected.add(kv24); + expected.add(kv23); + expected.add(kv22); + + Scan scan = new Scan(row1); + scan.addColumn(fam1, qf1); + scan.addColumn(fam1, qf2); + int versions = 3; + scan.setMaxVersions(versions); + List actual = new ArrayList(); + InternalScanner scanner = region.getScanner(scan); - boolean hasNext = scanner.next(actual); - assertEquals(false, hasNext); + boolean hasNext = scanner.next(actual); + assertEquals(false, hasNext); - //Verify result - for(int i=0; i expected = new ArrayList(); - expected.add(kv13); - expected.add(kv12); - expected.add(kv23); - expected.add(kv22); + this.region = initHRegion(tableName, method, conf, families); + try { + //Putting data in Region + Put put = null; + KeyValue kv13 = new KeyValue(row1, fam1, qf1, ts3, KeyValue.Type.Put, null); + KeyValue kv12 = new KeyValue(row1, fam1, qf1, ts2, KeyValue.Type.Put, null); + KeyValue kv11 = new KeyValue(row1, fam1, qf1, ts1, KeyValue.Type.Put, null); + + KeyValue kv23 = new KeyValue(row1, fam1, qf2, ts3, KeyValue.Type.Put, null); + KeyValue kv22 = new KeyValue(row1, fam1, qf2, ts2, KeyValue.Type.Put, null); + KeyValue kv21 = new KeyValue(row1, fam1, qf2, ts1, KeyValue.Type.Put, null); + + put = new Put(row1); + put.add(kv13); + put.add(kv12); + put.add(kv11); + put.add(kv23); + put.add(kv22); + put.add(kv21); + region.put(put); - Scan scan = new Scan(row1); - scan.addFamily(fam1); - scan.setMaxVersions(MAX_VERSIONS); - List actual = new ArrayList(); - InternalScanner scanner = region.getScanner(scan); + //Expected + List expected = new ArrayList(); + expected.add(kv13); + expected.add(kv12); + expected.add(kv23); + expected.add(kv22); + + Scan scan = new Scan(row1); + scan.addFamily(fam1); + scan.setMaxVersions(MAX_VERSIONS); + List actual = new ArrayList(); + InternalScanner scanner = region.getScanner(scan); - boolean hasNext = scanner.next(actual); - assertEquals(false, hasNext); + boolean hasNext = scanner.next(actual); + assertEquals(false, hasNext); - //Verify result - for(int i=0; i expected = new ArrayList(); - expected.add(kv13); - expected.add(kv12); - expected.add(kv23); - expected.add(kv22); + this.region = initHRegion(tableName, method, conf, fam1); + try { + //Putting data in Region + Put put = null; + KeyValue kv13 = new KeyValue(row1, fam1, qf1, ts3, KeyValue.Type.Put, null); + KeyValue kv12 = new KeyValue(row1, fam1, qf1, ts2, KeyValue.Type.Put, null); + KeyValue kv11 = new KeyValue(row1, fam1, qf1, ts1, KeyValue.Type.Put, null); + + KeyValue kv23 = new KeyValue(row1, fam1, qf2, ts3, KeyValue.Type.Put, null); + KeyValue kv22 = new KeyValue(row1, fam1, qf2, ts2, KeyValue.Type.Put, null); + KeyValue kv21 = new KeyValue(row1, fam1, qf2, ts1, KeyValue.Type.Put, null); + + put = new Put(row1); + put.add(kv13); + put.add(kv12); + put.add(kv11); + put.add(kv23); + put.add(kv22); + put.add(kv21); + region.put(put); + region.flushcache(); - Scan scan = new Scan(row1); - scan.addFamily(fam1); - scan.setMaxVersions(MAX_VERSIONS); - List actual = new ArrayList(); - InternalScanner scanner = region.getScanner(scan); + //Expected + List expected = new ArrayList(); + expected.add(kv13); + expected.add(kv12); + expected.add(kv23); + expected.add(kv22); + + Scan scan = new Scan(row1); + scan.addFamily(fam1); + scan.setMaxVersions(MAX_VERSIONS); + List actual = new ArrayList(); + InternalScanner scanner = region.getScanner(scan); - boolean hasNext = scanner.next(actual); - assertEquals(false, hasNext); + boolean hasNext = scanner.next(actual); + assertEquals(false, hasNext); - //Verify result - for(int i=0; i results = new ArrayList(); - assertEquals(false, s.next(results)); - assertEquals(0, results.size()); + List results = new ArrayList(); + assertEquals(false, s.next(results)); + assertEquals(0, results.size()); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testIncrementColumnValue_UpdatingInPlace() throws IOException { - initHRegion(tableName, getName(), fam1); - - long value = 1L; - long amount = 3L; + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + long value = 1L; + long amount = 3L; - Put put = new Put(row); - put.add(fam1, qual1, Bytes.toBytes(value)); - region.put(put); + Put put = new Put(row); + put.add(fam1, qual1, Bytes.toBytes(value)); + region.put(put); - long result = region.incrementColumnValue(row, fam1, qual1, amount, true); + long result = region.incrementColumnValue(row, fam1, qual1, amount, true); - assertEquals(value+amount, result); + assertEquals(value+amount, result); - Store store = region.getStore(fam1); - // ICV removes any extra values floating around in there. - assertEquals(1, store.memstore.kvset.size()); - assertTrue(store.memstore.snapshot.isEmpty()); + Store store = region.getStore(fam1); + // ICV removes any extra values floating around in there. + assertEquals(1, store.memstore.kvset.size()); + assertTrue(store.memstore.snapshot.isEmpty()); - assertICV(row, fam1, qual1, value+amount); + assertICV(row, fam1, qual1, value+amount); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testIncrementColumnValue_BumpSnapshot() throws IOException { ManualEnvironmentEdge mee = new ManualEnvironmentEdge(); EnvironmentEdgeManagerTestHelper.injectEdge(mee); - initHRegion(tableName, getName(), fam1); - - long value = 42L; - long incr = 44L; + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + long value = 42L; + long incr = 44L; - // first put something in kvset, then snapshot it. - Put put = new Put(row); - put.add(fam1, qual1, Bytes.toBytes(value)); - region.put(put); + // first put something in kvset, then snapshot it. + Put put = new Put(row); + put.add(fam1, qual1, Bytes.toBytes(value)); + region.put(put); - // get the store in question: - Store s = region.getStore(fam1); - s.snapshot(); //bam + // get the store in question: + Store s = region.getStore(fam1); + s.snapshot(); //bam - // now increment: - long newVal = region.incrementColumnValue(row, fam1, qual1, - incr, false); + // now increment: + long newVal = region.incrementColumnValue(row, fam1, qual1, + incr, false); - assertEquals(value+incr, newVal); + assertEquals(value+incr, newVal); - // get both versions: - Get get = new Get(row); - get.setMaxVersions(); - get.addColumn(fam1,qual1); + // get both versions: + Get get = new Get(row); + get.setMaxVersions(); + get.addColumn(fam1,qual1); - Result r = region.get(get, null); - assertEquals(2, r.size()); - KeyValue first = r.raw()[0]; - KeyValue second = r.raw()[1]; + Result r = region.get(get, null); + assertEquals(2, r.size()); + KeyValue first = r.raw()[0]; + KeyValue second = r.raw()[1]; - assertTrue("ICV failed to upgrade timestamp", - first.getTimestamp() != second.getTimestamp()); + assertTrue("ICV failed to upgrade timestamp", + first.getTimestamp() != second.getTimestamp()); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testIncrementColumnValue_ConcurrentFlush() throws IOException { - initHRegion(tableName, getName(), fam1); - - long value = 1L; - long amount = 3L; + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + long value = 1L; + long amount = 3L; - Put put = new Put(row); - put.add(fam1, qual1, Bytes.toBytes(value)); - region.put(put); + Put put = new Put(row); + put.add(fam1, qual1, Bytes.toBytes(value)); + region.put(put); - // now increment during a flush - Thread t = new Thread() { - public void run() { - try { - region.flushcache(); - } catch (IOException e) { - LOG.info("test ICV, got IOE during flushcache()"); + // now increment during a flush + Thread t = new Thread() { + public void run() { + try { + region.flushcache(); + } catch (IOException e) { + LOG.info("test ICV, got IOE during flushcache()"); + } } - } - }; - t.start(); - long r = region.incrementColumnValue(row, fam1, qual1, amount, true); - assertEquals(value+amount, r); + }; + t.start(); + long r = region.incrementColumnValue(row, fam1, qual1, amount, true); + assertEquals(value+amount, r); - // this also asserts there is only 1 KeyValue in the set. - assertICV(row, fam1, qual1, value+amount); + // this also asserts there is only 1 KeyValue in the set. + assertICV(row, fam1, qual1, value+amount); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testIncrementColumnValue_heapSize() throws IOException { EnvironmentEdgeManagerTestHelper.injectEdge(new IncrementingEnvironmentEdge()); - initHRegion(tableName, getName(), fam1); - - long byAmount = 1L; - long size; + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + long byAmount = 1L; + long size; - for( int i = 0; i < 1000 ; i++) { - region.incrementColumnValue(row, fam1, qual1, byAmount, true); + for( int i = 0; i < 1000 ; i++) { + region.incrementColumnValue(row, fam1, qual1, byAmount, true); - size = region.memstoreSize.get(); - assertTrue("memstore size: " + size, size >= 0); + size = region.memstoreSize.get(); + assertTrue("memstore size: " + size, size >= 0); + } + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } public void testIncrementColumnValue_UpdatingInPlace_Negative() throws IOException { - initHRegion(tableName, getName(), fam1); - - long value = 3L; - long amount = -1L; + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + long value = 3L; + long amount = -1L; - Put put = new Put(row); - put.add(fam1, qual1, Bytes.toBytes(value)); - region.put(put); + Put put = new Put(row); + put.add(fam1, qual1, Bytes.toBytes(value)); + region.put(put); - long result = region.incrementColumnValue(row, fam1, qual1, amount, true); - assertEquals(value+amount, result); + long result = region.incrementColumnValue(row, fam1, qual1, amount, true); + assertEquals(value+amount, result); - assertICV(row, fam1, qual1, value+amount); + assertICV(row, fam1, qual1, value+amount); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testIncrementColumnValue_AddingNew() throws IOException { - initHRegion(tableName, getName(), fam1); - - long value = 1L; - long amount = 3L; - - Put put = new Put(row); - put.add(fam1, qual1, Bytes.toBytes(value)); - put.add(fam1, qual2, Bytes.toBytes(value)); - region.put(put); + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + long value = 1L; + long amount = 3L; - long result = region.incrementColumnValue(row, fam1, qual3, amount, true); - assertEquals(amount, result); + Put put = new Put(row); + put.add(fam1, qual1, Bytes.toBytes(value)); + put.add(fam1, qual2, Bytes.toBytes(value)); + region.put(put); - Get get = new Get(row); - get.addColumn(fam1, qual3); - Result rr = region.get(get, null); - assertEquals(1, rr.size()); + long result = region.incrementColumnValue(row, fam1, qual3, amount, true); + assertEquals(amount, result); - // ensure none of the other cols were incremented. - assertICV(row, fam1, qual1, value); - assertICV(row, fam1, qual2, value); - assertICV(row, fam1, qual3, amount); + Get get = new Get(row); + get.addColumn(fam1, qual3); + Result rr = region.get(get, null); + assertEquals(1, rr.size()); + + // ensure none of the other cols were incremented. + assertICV(row, fam1, qual1, value); + assertICV(row, fam1, qual2, value); + assertICV(row, fam1, qual3, amount); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testIncrementColumnValue_UpdatingFromSF() throws IOException { - initHRegion(tableName, getName(), fam1); - - long value = 1L; - long amount = 3L; + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + long value = 1L; + long amount = 3L; - Put put = new Put(row); - put.add(fam1, qual1, Bytes.toBytes(value)); - put.add(fam1, qual2, Bytes.toBytes(value)); - region.put(put); + Put put = new Put(row); + put.add(fam1, qual1, Bytes.toBytes(value)); + put.add(fam1, qual2, Bytes.toBytes(value)); + region.put(put); - // flush to disk. - region.flushcache(); + // flush to disk. + region.flushcache(); - Store store = region.getStore(fam1); - assertEquals(0, store.memstore.kvset.size()); + Store store = region.getStore(fam1); + assertEquals(0, store.memstore.kvset.size()); - long r = region.incrementColumnValue(row, fam1, qual1, amount, true); - assertEquals(value+amount, r); + long r = region.incrementColumnValue(row, fam1, qual1, amount, true); + assertEquals(value+amount, r); - assertICV(row, fam1, qual1, value+amount); + assertICV(row, fam1, qual1, value+amount); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testIncrementColumnValue_AddingNewAfterSFCheck() throws IOException { - initHRegion(tableName, getName(), fam1); - - long value = 1L; - long amount = 3L; + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + long value = 1L; + long amount = 3L; - Put put = new Put(row); - put.add(fam1, qual1, Bytes.toBytes(value)); - put.add(fam1, qual2, Bytes.toBytes(value)); - region.put(put); - region.flushcache(); + Put put = new Put(row); + put.add(fam1, qual1, Bytes.toBytes(value)); + put.add(fam1, qual2, Bytes.toBytes(value)); + region.put(put); + region.flushcache(); - Store store = region.getStore(fam1); - assertEquals(0, store.memstore.kvset.size()); + Store store = region.getStore(fam1); + assertEquals(0, store.memstore.kvset.size()); - long r = region.incrementColumnValue(row, fam1, qual3, amount, true); - assertEquals(amount, r); + long r = region.incrementColumnValue(row, fam1, qual3, amount, true); + assertEquals(amount, r); - assertICV(row, fam1, qual3, amount); + assertICV(row, fam1, qual3, amount); - region.flushcache(); + region.flushcache(); - // ensure that this gets to disk. - assertICV(row, fam1, qual3, amount); + // ensure that this gets to disk. + assertICV(row, fam1, qual3, amount); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } /** @@ -2281,73 +2678,113 @@ public void testIncrementColumnValue_AddingNewAfterSFCheck() * @throws IOException */ public void testIncrementColumnValue_UpdatingInPlace_TimestampClobber() throws IOException { - initHRegion(tableName, getName(), fam1); - - long value = 1L; - long amount = 3L; - long now = EnvironmentEdgeManager.currentTimeMillis(); - ManualEnvironmentEdge mock = new ManualEnvironmentEdge(); - mock.setValue(now); - EnvironmentEdgeManagerTestHelper.injectEdge(mock); - - // verify we catch an ICV on a put with the same timestamp - Put put = new Put(row); - put.add(fam1, qual1, now, Bytes.toBytes(value)); - region.put(put); + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + long value = 1L; + long amount = 3L; + long now = EnvironmentEdgeManager.currentTimeMillis(); + ManualEnvironmentEdge mock = new ManualEnvironmentEdge(); + mock.setValue(now); + EnvironmentEdgeManagerTestHelper.injectEdge(mock); + + // verify we catch an ICV on a put with the same timestamp + Put put = new Put(row); + put.add(fam1, qual1, now, Bytes.toBytes(value)); + region.put(put); - long result = region.incrementColumnValue(row, fam1, qual1, amount, true); + long result = region.incrementColumnValue(row, fam1, qual1, amount, true); - assertEquals(value+amount, result); + assertEquals(value+amount, result); - Store store = region.getStore(fam1); - // ICV should update the existing Put with the same timestamp - assertEquals(1, store.memstore.kvset.size()); - assertTrue(store.memstore.snapshot.isEmpty()); + Store store = region.getStore(fam1); + // ICV should update the existing Put with the same timestamp + assertEquals(1, store.memstore.kvset.size()); + assertTrue(store.memstore.snapshot.isEmpty()); - assertICV(row, fam1, qual1, value+amount); + assertICV(row, fam1, qual1, value+amount); - // verify we catch an ICV even when the put ts > now - put = new Put(row); - put.add(fam1, qual2, now+1, Bytes.toBytes(value)); - region.put(put); + // verify we catch an ICV even when the put ts > now + put = new Put(row); + put.add(fam1, qual2, now+1, Bytes.toBytes(value)); + region.put(put); - result = region.incrementColumnValue(row, fam1, qual2, amount, true); + result = region.incrementColumnValue(row, fam1, qual2, amount, true); - assertEquals(value+amount, result); + assertEquals(value+amount, result); - store = region.getStore(fam1); - // ICV should update the existing Put with the same timestamp - assertEquals(2, store.memstore.kvset.size()); - assertTrue(store.memstore.snapshot.isEmpty()); + store = region.getStore(fam1); + // ICV should update the existing Put with the same timestamp + assertEquals(2, store.memstore.kvset.size()); + assertTrue(store.memstore.snapshot.isEmpty()); - assertICV(row, fam1, qual2, value+amount); - EnvironmentEdgeManagerTestHelper.reset(); + assertICV(row, fam1, qual2, value+amount); + EnvironmentEdgeManagerTestHelper.reset(); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testIncrementColumnValue_WrongInitialSize() throws IOException { - initHRegion(tableName, getName(), fam1); - - byte[] row1 = Bytes.add(Bytes.toBytes("1234"), Bytes.toBytes(0L)); - int row1Field1 = 0; - int row1Field2 = 1; - Put put1 = new Put(row1); - put1.add(fam1, qual1, Bytes.toBytes(row1Field1)); - put1.add(fam1, qual2, Bytes.toBytes(row1Field2)); - region.put(put1); - - long result; + this.region = initHRegion(tableName, getName(), conf, fam1); try { + byte[] row1 = Bytes.add(Bytes.toBytes("1234"), Bytes.toBytes(0L)); + int row1Field1 = 0; + int row1Field2 = 1; + Put put1 = new Put(row1); + put1.add(fam1, qual1, Bytes.toBytes(row1Field1)); + put1.add(fam1, qual2, Bytes.toBytes(row1Field2)); + region.put(put1); + + long result; + try { result = region.incrementColumnValue(row1, fam1, qual1, 1, true); fail("Expected to fail here"); - } catch (Exception exception) { + } catch (Exception exception) { // Expected. - } + } - assertICV(row1, fam1, qual1, row1Field1); - assertICV(row1, fam1, qual2, row1Field2); + assertICV(row1, fam1, qual1, row1Field1); + assertICV(row1, fam1, qual2, row1Field2); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } + public void testIncrement_WrongInitialSize() throws IOException { + this.region = initHRegion(tableName, getName(), conf, fam1); + try { + byte[] row1 = Bytes.add(Bytes.toBytes("1234"), Bytes.toBytes(0L)); + long row1Field1 = 0; + int row1Field2 = 1; + Put put1 = new Put(row1); + put1.add(fam1, qual1, Bytes.toBytes(row1Field1)); + put1.add(fam1, qual2, Bytes.toBytes(row1Field2)); + region.put(put1); + Increment increment = new Increment(row1); + increment.addColumn(fam1, qual1, 1); + + //here we should be successful as normal + region.increment(increment, null, true); + assertICV(row1, fam1, qual1, row1Field1 + 1); + + //failed to increment + increment = new Increment(row1); + increment.addColumn(fam1, qual2, 1); + try { + region.increment(increment, null, true); + fail("Expected to fail here"); + } catch (Exception exception) { + // Expected. + } + assertICV(row1, fam1, qual2, row1Field2); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } + } private void assertICV(byte [] row, byte [] familiy, byte[] qualifier, @@ -2393,64 +2830,235 @@ public void testScanner_Wildcard_FromMemStoreAndFiles_EnforceVersions() //Setting up region String method = this.getName(); - initHRegion(tableName, method, fam1); + this.region = initHRegion(tableName, method, conf, fam1); + try { + //Putting data in Region + KeyValue kv14 = new KeyValue(row1, fam1, qf1, ts4, KeyValue.Type.Put, null); + KeyValue kv13 = new KeyValue(row1, fam1, qf1, ts3, KeyValue.Type.Put, null); + KeyValue kv12 = new KeyValue(row1, fam1, qf1, ts2, KeyValue.Type.Put, null); + KeyValue kv11 = new KeyValue(row1, fam1, qf1, ts1, KeyValue.Type.Put, null); + + KeyValue kv24 = new KeyValue(row1, fam1, qf2, ts4, KeyValue.Type.Put, null); + KeyValue kv23 = new KeyValue(row1, fam1, qf2, ts3, KeyValue.Type.Put, null); + KeyValue kv22 = new KeyValue(row1, fam1, qf2, ts2, KeyValue.Type.Put, null); + KeyValue kv21 = new KeyValue(row1, fam1, qf2, ts1, KeyValue.Type.Put, null); + + Put put = null; + put = new Put(row1); + put.add(kv14); + put.add(kv24); + region.put(put); + region.flushcache(); - //Putting data in Region - KeyValue kv14 = new KeyValue(row1, fam1, qf1, ts4, KeyValue.Type.Put, null); - KeyValue kv13 = new KeyValue(row1, fam1, qf1, ts3, KeyValue.Type.Put, null); - KeyValue kv12 = new KeyValue(row1, fam1, qf1, ts2, KeyValue.Type.Put, null); - KeyValue kv11 = new KeyValue(row1, fam1, qf1, ts1, KeyValue.Type.Put, null); + put = new Put(row1); + put.add(kv23); + put.add(kv13); + region.put(put); + region.flushcache(); - KeyValue kv24 = new KeyValue(row1, fam1, qf2, ts4, KeyValue.Type.Put, null); - KeyValue kv23 = new KeyValue(row1, fam1, qf2, ts3, KeyValue.Type.Put, null); - KeyValue kv22 = new KeyValue(row1, fam1, qf2, ts2, KeyValue.Type.Put, null); - KeyValue kv21 = new KeyValue(row1, fam1, qf2, ts1, KeyValue.Type.Put, null); + put = new Put(row1); + put.add(kv22); + put.add(kv12); + region.put(put); + region.flushcache(); - Put put = null; - put = new Put(row1); - put.add(kv14); - put.add(kv24); - region.put(put); - region.flushcache(); + put = new Put(row1); + put.add(kv21); + put.add(kv11); + region.put(put); - put = new Put(row1); - put.add(kv23); - put.add(kv13); - region.put(put); - region.flushcache(); + //Expected + List expected = new ArrayList(); + expected.add(kv14); + expected.add(kv13); + expected.add(kv12); + expected.add(kv24); + expected.add(kv23); + expected.add(kv22); + + Scan scan = new Scan(row1); + int versions = 3; + scan.setMaxVersions(versions); + List actual = new ArrayList(); + InternalScanner scanner = region.getScanner(scan); - put = new Put(row1); - put.add(kv22); - put.add(kv12); - region.put(put); - region.flushcache(); + boolean hasNext = scanner.next(actual); + assertEquals(false, hasNext); - put = new Put(row1); - put.add(kv21); - put.add(kv11); - region.put(put); + //Verify result + for(int i=0; i expected = new ArrayList(); - expected.add(kv14); - expected.add(kv13); - expected.add(kv12); - expected.add(kv24); - expected.add(kv23); - expected.add(kv22); + // Check two things: + // 1. result list contains expected values + // 2. result list is sorted properly - Scan scan = new Scan(row1); - int versions = 3; - scan.setMaxVersions(versions); - List actual = new ArrayList(); - InternalScanner scanner = region.getScanner(scan); + Scan scan = new Scan(); + Filter filter = new SingleColumnValueExcludeFilter(cf_essential, col_normal, + CompareOp.NOT_EQUAL, filtered_val); + scan.setFilter(filter); + scan.setLoadColumnFamiliesOnDemand(true); + InternalScanner s = region.getScanner(scan); + + List results = new ArrayList(); + assertTrue(s.next(results)); + assertEquals(results.size(), 1); + results.clear(); + + assertTrue(s.next(results)); + assertEquals(results.size(), 3); + assertTrue("orderCheck", results.get(0).matchingFamily(cf_alpha)); + assertTrue("orderCheck", results.get(1).matchingFamily(cf_essential)); + assertTrue("orderCheck", results.get(2).matchingFamily(cf_joined)); + results.clear(); + + assertFalse(s.next(results)); + assertEquals(results.size(), 0); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } + } + + /** + * HBASE-5416 + * + * Test case when scan limits amount of KVs returned on each next() call. + */ + public void testScanner_JoinedScannersWithLimits() throws IOException { + final byte [] tableName = Bytes.toBytes("testTable"); + final byte [] cf_first = Bytes.toBytes("first"); + final byte [] cf_second = Bytes.toBytes("second"); + + this.region = initHRegion(tableName, getName(), conf, cf_first, cf_second); + try { + final byte [] col_a = Bytes.toBytes("a"); + final byte [] col_b = Bytes.toBytes("b"); + + Put put; + + for (int i = 0; i < 10; i++) { + put = new Put(Bytes.toBytes("r" + Integer.toString(i))); + put.add(cf_first, col_a, Bytes.toBytes(i)); + if (i < 5) { + put.add(cf_first, col_b, Bytes.toBytes(i)); + put.add(cf_second, col_a, Bytes.toBytes(i)); + put.add(cf_second, col_b, Bytes.toBytes(i)); + } + region.put(put); + } - boolean hasNext = scanner.next(actual); - assertEquals(false, hasNext); + Scan scan = new Scan(); + scan.setLoadColumnFamiliesOnDemand(true); + Filter bogusFilter = new FilterBase() { + @Override + public boolean isFamilyEssential(byte[] name) { + return Bytes.equals(name, cf_first); + } + @Override + public void readFields(DataInput arg0) throws IOException { + } - //Verify result - for(int i=0; i results = new ArrayList(); + int index = 0; + while (true) { + boolean more = s.next(results, 3); + if ((index >> 1) < 5) { + if (index % 2 == 0) + assertEquals(results.size(), 3); + else + assertEquals(results.size(), 1); + } + else + assertEquals(results.size(), 1); + results.clear(); + index++; + if (!more) break; + } + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -2468,7 +3076,7 @@ public void testBasicSplit() throws Exception { Configuration hc = initSplit(); //Setting up region String method = this.getName(); - initHRegion(tableName, method, hc, families); + this.region = initHRegion(tableName, method, hc, families); try { LOG.info("" + addContent(region, fam3)); @@ -2544,10 +3152,8 @@ public void testBasicSplit() throws Exception { } } } finally { - if (region != null) { - region.close(); - region.getLog().closeAndDelete(); - } + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -2560,7 +3166,7 @@ public void testSplitRegion() throws IOException { //Setting up region String method = this.getName(); - initHRegion(tableName, method, hc, families); + this.region = initHRegion(tableName, method, hc, families); //Put data in region int startRow = 100; @@ -2585,10 +3191,8 @@ public void testSplitRegion() throws IOException { verifyData(regions[1], splitRow, numRows, qualifier, families); } finally { - if (region != null) { - region.close(); - region.getLog().closeAndDelete(); - } + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -2608,52 +3212,64 @@ public void testFlushCacheWhileScanning() throws IOException, InterruptedExcepti int compactInterval = 10 * flushAndScanInterval; String method = "testFlushCacheWhileScanning"; - initHRegion(tableName,method, family); - FlushThread flushThread = new FlushThread(); - flushThread.start(); - - Scan scan = new Scan(); - scan.addFamily(family); - scan.setFilter(new SingleColumnValueFilter(family, qual1, - CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes(5L)))); - - int expectedCount = 0; - List res = new ArrayList(); - - boolean toggle=true; - for (long i = 0; i < numRows; i++) { - Put put = new Put(Bytes.toBytes(i)); - put.setWriteToWAL(false); - put.add(family, qual1, Bytes.toBytes(i % 10)); - region.put(put); - - if (i != 0 && i % compactInterval == 0) { - //System.out.println("iteration = " + i); - region.compactStores(true); - } - - if (i % 10 == 5L) { - expectedCount++; - } + this.region = initHRegion(tableName,method, conf, family); + try { + FlushThread flushThread = new FlushThread(); + flushThread.start(); + + Scan scan = new Scan(); + scan.addFamily(family); + scan.setFilter(new SingleColumnValueFilter(family, qual1, + CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes(5L)))); + + int expectedCount = 0; + List res = new ArrayList(); + + boolean toggle=true; + for (long i = 0; i < numRows; i++) { + Put put = new Put(Bytes.toBytes(i)); + put.setWriteToWAL(false); + put.add(family, qual1, Bytes.toBytes(i % 10)); + region.put(put); + + if (i != 0 && i % compactInterval == 0) { + //System.out.println("iteration = " + i); + region.compactStores(true); + } - if (i != 0 && i % flushAndScanInterval == 0) { - res.clear(); - InternalScanner scanner = region.getScanner(scan); - if (toggle) { - flushThread.flush(); + if (i % 10 == 5L) { + expectedCount++; } - while (scanner.next(res)) ; - if (!toggle) { - flushThread.flush(); + + if (i != 0 && i % flushAndScanInterval == 0) { + res.clear(); + InternalScanner scanner = region.getScanner(scan); + if (toggle) { + flushThread.flush(); + } + while (scanner.next(res)) ; + if (!toggle) { + flushThread.flush(); + } + assertEquals("i=" + i, expectedCount, res.size()); + toggle = !toggle; } - assertEquals("i=" + i, expectedCount, res.size()); - toggle = !toggle; } - } - flushThread.done(); - flushThread.join(); - flushThread.checkNoError(); + flushThread.done(); + flushThread.join(); + flushThread.checkNoError(); + } finally { + try { + HRegion.closeHRegion(this.region); + } catch (DroppedSnapshotException dse) { + // We could get this on way out because we interrupt the background flusher and it could + // fail anywhere causing a DSE over in the background flusher... only it is not properly + // dealt with so could still be memory hanging out when we get to here -- memory we can't + // flush because the accounting is 'off' since original DSE. + } + this.region = null; + } } protected class FlushThread extends Thread { @@ -2734,56 +3350,68 @@ public void testWritesWhileScanning() } String method = "testWritesWhileScanning"; - initHRegion(tableName, method, families); - PutThread putThread = new PutThread(numRows, families, qualifiers); - putThread.start(); - putThread.waitForFirstPut(); + this.region = initHRegion(tableName, method, conf, families); + try { + PutThread putThread = new PutThread(numRows, families, qualifiers); + putThread.start(); + putThread.waitForFirstPut(); - FlushThread flushThread = new FlushThread(); - flushThread.start(); + FlushThread flushThread = new FlushThread(); + flushThread.start(); - Scan scan = new Scan(Bytes.toBytes("row0"), Bytes.toBytes("row1")); -// scan.setFilter(new RowFilter(CompareFilter.CompareOp.EQUAL, -// new BinaryComparator(Bytes.toBytes("row0")))); + Scan scan = new Scan(Bytes.toBytes("row0"), Bytes.toBytes("row1")); + // scan.setFilter(new RowFilter(CompareFilter.CompareOp.EQUAL, + // new BinaryComparator(Bytes.toBytes("row0")))); - int expectedCount = numFamilies * numQualifiers; - List res = new ArrayList(); + int expectedCount = numFamilies * numQualifiers; + List res = new ArrayList(); - long prevTimestamp = 0L; - for (int i = 0; i < testCount; i++) { + long prevTimestamp = 0L; + for (int i = 0; i < testCount; i++) { - if (i != 0 && i % compactInterval == 0) { - region.compactStores(true); - } + if (i != 0 && i % compactInterval == 0) { + region.compactStores(true); + } - if (i != 0 && i % flushInterval == 0) { - //System.out.println("flush scan iteration = " + i); - flushThread.flush(); - } + if (i != 0 && i % flushInterval == 0) { + //System.out.println("flush scan iteration = " + i); + flushThread.flush(); + } - boolean previousEmpty = res.isEmpty(); - res.clear(); - InternalScanner scanner = region.getScanner(scan); - while (scanner.next(res)) ; - if (!res.isEmpty() || !previousEmpty || i > compactInterval) { - assertEquals("i=" + i, expectedCount, res.size()); - long timestamp = res.get(0).getTimestamp(); - assertTrue("Timestamps were broke: " + timestamp + " prev: " + prevTimestamp, - timestamp >= prevTimestamp); - prevTimestamp = timestamp; + boolean previousEmpty = res.isEmpty(); + res.clear(); + InternalScanner scanner = region.getScanner(scan); + while (scanner.next(res)) ; + if (!res.isEmpty() || !previousEmpty || i > compactInterval) { + assertEquals("i=" + i, expectedCount, res.size()); + long timestamp = res.get(0).getTimestamp(); + assertTrue("Timestamps were broke: " + timestamp + " prev: " + prevTimestamp, + timestamp >= prevTimestamp); + prevTimestamp = timestamp; + } } - } - putThread.done(); + putThread.done(); - region.flushcache(); + region.flushcache(); - putThread.join(); - putThread.checkNoError(); + putThread.join(); + putThread.checkNoError(); - flushThread.done(); - flushThread.join(); - flushThread.checkNoError(); + flushThread.done(); + flushThread.join(); + flushThread.checkNoError(); + } finally { + try { + HRegion.closeHRegion(this.region); + } catch (DroppedSnapshotException dse) { + // We could get this on way out because we interrupt the background flusher and it could + // fail anywhere causing a DSE over in the background flusher... only it is not properly + // dealt with so could still be memory hanging out when we get to here -- memory we can't + // flush because the accounting is 'off' since original DSE. + } + this.region = null; + } } protected class PutThread extends Thread { @@ -2851,6 +3479,8 @@ public void run() { } numPutsFinished++; } + } catch (InterruptedIOException e) { + // This is fine. It means we are done, or didn't get the lock on time } catch (IOException e) { LOG.error("error while putting records", e); error = e; @@ -2865,21 +3495,19 @@ public void run() { /** * Writes very wide records and gets the latest row every time.. - * Flushes and compacts the region every now and then to keep things - * realistic. + * Flushes and compacts the region aggressivly to catch issues. * * @throws IOException by flush / scan / compaction * @throws InterruptedException when joining threads */ public void testWritesWhileGetting() - throws IOException, InterruptedException { - byte[] tableName = Bytes.toBytes("testWritesWhileScanning"); + throws Exception { + byte[] tableName = Bytes.toBytes("testWritesWhileGetting"); int testCount = 100; int numRows = 1; int numFamilies = 10; int numQualifiers = 100; - int flushInterval = 10; - int compactInterval = 10 * flushInterval; + int compactInterval = 100; byte[][] families = new byte[numFamilies][]; for (int i = 0; i < numFamilies; i++) { families[i] = Bytes.toBytes("family" + i); @@ -2889,96 +3517,159 @@ public void testWritesWhileGetting() qualifiers[i] = Bytes.toBytes("qual" + i); } - String method = "testWritesWhileGetting"; - initHRegion(tableName, method, families); - PutThread putThread = new PutThread(numRows, families, qualifiers); - putThread.start(); - putThread.waitForFirstPut(); - - FlushThread flushThread = new FlushThread(); - flushThread.start(); - - Get get = new Get(Bytes.toBytes("row0")); - Result result = null; - - int expectedCount = numFamilies * numQualifiers; - - long prevTimestamp = 0L; - for (int i = 0; i < testCount; i++) { - - if (i != 0 && i % compactInterval == 0) { - region.compactStores(true); - } - - if (i != 0 && i % flushInterval == 0) { - //System.out.println("iteration = " + i); - flushThread.flush(); - } + Configuration conf = HBaseConfiguration.create(this.conf); - boolean previousEmpty = result == null || result.isEmpty(); - result = region.get(get, null); - if (!result.isEmpty() || !previousEmpty || i > compactInterval) { - assertEquals("i=" + i, expectedCount, result.size()); - // TODO this was removed, now what dangit?! - // search looking for the qualifier in question? - long timestamp = 0; - for (KeyValue kv : result.raw()) { - if (Bytes.equals(kv.getFamily(), families[0]) - && Bytes.equals(kv.getQualifier(), qualifiers[0])) { - timestamp = kv.getTimestamp(); + String method = "testWritesWhileGetting"; + // This test flushes constantly and can cause many files to be created, possibly + // extending over the ulimit. Make sure compactions are aggressive in reducing + // the number of HFiles created. + conf.setInt("hbase.hstore.compaction.min", 1); + conf.setInt("hbase.hstore.compaction.max", 1000); + this.region = initHRegion(tableName, method, conf, families); + PutThread putThread = null; + MultithreadedTestUtil.TestContext ctx = + new MultithreadedTestUtil.TestContext(conf); + try { + putThread = new PutThread(numRows, families, qualifiers); + putThread.start(); + putThread.waitForFirstPut(); + + // Add a thread that flushes as fast as possible + ctx.addThread(new RepeatingTestThread(ctx) { + private int flushesSinceCompact = 0; + private final int maxFlushesSinceCompact = 20; + public void doAnAction() throws Exception { + if (region.flushcache().isCompactionNeeded()) { + ++flushesSinceCompact; + } + // Compact regularly to avoid creating too many files and exceeding the ulimit. + if (flushesSinceCompact == maxFlushesSinceCompact) { + region.compactStores(false); + flushesSinceCompact = 0; } } - assertTrue(timestamp >= prevTimestamp); - prevTimestamp = timestamp; - KeyValue previousKV = null; - - for (KeyValue kv : result.raw()) { - byte[] thisValue = kv.getValue(); - if (previousKV != null) { - if (Bytes.compareTo(previousKV.getValue(), thisValue) != 0) { - LOG.warn("These two KV should have the same value." + - " Previous KV:" + - previousKV + "(memStoreTS:" + previousKV.getMemstoreTS() + ")" + - ", New KV: " + - kv + "(memStoreTS:" + kv.getMemstoreTS() + ")" - ); - assertEquals(previousKV.getValue(), thisValue); + }); + ctx.startThreads(); + + Get get = new Get(Bytes.toBytes("row0")); + Result result = null; + + int expectedCount = numFamilies * numQualifiers; + + long prevTimestamp = 0L; + for (int i = 0; i < testCount; i++) { + + boolean previousEmpty = result == null || result.isEmpty(); + result = region.get(get, null); + if (!result.isEmpty() || !previousEmpty || i > compactInterval) { + assertEquals("i=" + i, expectedCount, result.size()); + // TODO this was removed, now what dangit?! + // search looking for the qualifier in question? + long timestamp = 0; + for (KeyValue kv : result.raw()) { + if (Bytes.equals(kv.getFamily(), families[0]) + && Bytes.equals(kv.getQualifier(), qualifiers[0])) { + timestamp = kv.getTimestamp(); + } + } + assertTrue(timestamp >= prevTimestamp); + prevTimestamp = timestamp; + KeyValue previousKV = null; + + for (KeyValue kv : result.raw()) { + byte[] thisValue = kv.getValue(); + if (previousKV != null) { + if (Bytes.compareTo(previousKV.getValue(), thisValue) != 0) { + LOG.warn("These two KV should have the same value." + + " Previous KV:" + + previousKV + "(memStoreTS:" + previousKV.getMemstoreTS() + ")" + + ", New KV: " + + kv + "(memStoreTS:" + kv.getMemstoreTS() + ")" + ); + assertEquals(0, Bytes.compareTo(previousKV.getValue(), thisValue)); + } } + previousKV = kv; } - previousKV = kv; } } - } - - putThread.done(); + } finally { + if (putThread != null) putThread.done(); - region.flushcache(); + region.flushcache(); - putThread.join(); - putThread.checkNoError(); + if (putThread != null) { + putThread.join(); + putThread.checkNoError(); + } - flushThread.done(); - flushThread.join(); - flushThread.checkNoError(); + ctx.stop(); + HRegion.closeHRegion(this.region); + this.region = null; + } } public void testHolesInMeta() throws Exception { String method = "testHolesInMeta"; byte[] tableName = Bytes.toBytes(method); byte[] family = Bytes.toBytes("family"); - initHRegion(tableName, Bytes.toBytes("x"), Bytes.toBytes("z"), method, - HBaseConfiguration.create(), family); - byte[] rowNotServed = Bytes.toBytes("a"); - Get g = new Get(rowNotServed); + this.region = initHRegion(tableName, Bytes.toBytes("x"), Bytes.toBytes("z"), method, + conf, false, family); try { + byte[] rowNotServed = Bytes.toBytes("a"); + Get g = new Get(rowNotServed); + try { + region.get(g, null); + fail(); + } catch (WrongRegionException x) { + // OK + } + byte[] row = Bytes.toBytes("y"); + g = new Get(row); region.get(g, null); - fail(); - } catch (WrongRegionException x) { - // OK + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } + } + + /** + * Testcase to check state of region initialization task set to ABORTED or not if any exceptions + * during initialization + * + * @throws Exception + */ + @Test + public void testStatusSettingToAbortIfAnyExceptionDuringRegionInitilization() throws Exception { + HRegionInfo info = null; + try { + FileSystem fs = Mockito.mock(FileSystem.class); + Mockito.when(fs.exists((Path) Mockito.anyObject())).thenThrow(new IOException()); + HTableDescriptor htd = new HTableDescriptor(tableName); + htd.addFamily(new HColumnDescriptor("cf")); + info = new HRegionInfo(htd.getName(), HConstants.EMPTY_BYTE_ARRAY, + HConstants.EMPTY_BYTE_ARRAY, false); + Path path = new Path(DIR + "testStatusSettingToAbortIfAnyExceptionDuringRegionInitilization"); + // no where we are instantiating HStore in this test case so useTableNameGlobally is null. To + // avoid NullPointerException we are setting useTableNameGlobally to false. + SchemaMetrics.setUseTableNameInTest(false); + region = HRegion.newHRegion(path, null, fs, conf, info, htd, null); + // region initialization throws IOException and set task state to ABORTED. + region.initialize(); + fail("Region initialization should fail due to IOException"); + } catch (IOException io) { + List tasks = TaskMonitor.get().getTasks(); + for (MonitoredTask monitoredTask : tasks) { + if (!(monitoredTask instanceof MonitoredRPCHandler) + && monitoredTask.getDescription().contains(region.toString())) { + assertTrue("Region state should be ABORTED.", + monitoredTask.getState().equals(MonitoredTask.State.ABORTED)); + break; + } + } + } finally { + HRegion.closeHRegion(region); } - byte[] row = Bytes.toBytes("y"); - g = new Get(row); - region.get(g, null); } public void testIndexesScanWithOneDeletedRow() throws IOException { @@ -2987,40 +3678,43 @@ public void testIndexesScanWithOneDeletedRow() throws IOException { //Setting up region String method = "testIndexesScanWithOneDeletedRow"; - initHRegion(tableName, method, HBaseConfiguration.create(), family); - - Put put = new Put(Bytes.toBytes(1L)); - put.add(family, qual1, 1L, Bytes.toBytes(1L)); - region.put(put); - - region.flushcache(); - - Delete delete = new Delete(Bytes.toBytes(1L), 1L, null); - //delete.deleteColumn(family, qual1); - region.delete(delete, null, true); + this.region = initHRegion(tableName, method, conf, family); + try { + Put put = new Put(Bytes.toBytes(1L)); + put.add(family, qual1, 1L, Bytes.toBytes(1L)); + region.put(put); - put = new Put(Bytes.toBytes(2L)); - put.add(family, qual1, 2L, Bytes.toBytes(2L)); - region.put(put); + region.flushcache(); - Scan idxScan = new Scan(); - idxScan.addFamily(family); - idxScan.setFilter(new FilterList(FilterList.Operator.MUST_PASS_ALL, - Arrays.asList(new SingleColumnValueFilter(family, qual1, - CompareOp.GREATER_OR_EQUAL, - new BinaryComparator(Bytes.toBytes(0L))), - new SingleColumnValueFilter(family, qual1, CompareOp.LESS_OR_EQUAL, - new BinaryComparator(Bytes.toBytes(3L))) - ))); - InternalScanner scanner = region.getScanner(idxScan); - List res = new ArrayList(); + Delete delete = new Delete(Bytes.toBytes(1L), 1L, null); + //delete.deleteColumn(family, qual1); + region.delete(delete, null, true); - //long start = System.nanoTime(); - while (scanner.next(res)) ; - //long end = System.nanoTime(); - //System.out.println("memStoreEmpty=" + memStoreEmpty + ", time=" + (end - start)/1000000D); - assertEquals(1L, res.size()); + put = new Put(Bytes.toBytes(2L)); + put.add(family, qual1, 2L, Bytes.toBytes(2L)); + region.put(put); + Scan idxScan = new Scan(); + idxScan.addFamily(family); + idxScan.setFilter(new FilterList(FilterList.Operator.MUST_PASS_ALL, + Arrays.asList(new SingleColumnValueFilter(family, qual1, + CompareOp.GREATER_OR_EQUAL, + new BinaryComparator(Bytes.toBytes(0L))), + new SingleColumnValueFilter(family, qual1, CompareOp.LESS_OR_EQUAL, + new BinaryComparator(Bytes.toBytes(3L))) + ))); + InternalScanner scanner = region.getScanner(idxScan); + List res = new ArrayList(); + + //long start = System.nanoTime(); + while (scanner.next(res)) ; + //long end = System.nanoTime(); + //System.out.println("memStoreEmpty=" + memStoreEmpty + ", time=" + (end - start)/1000000D); + assertEquals(1L, res.size()); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } ////////////////////////////////////////////////////////////////////////////// @@ -3041,46 +3735,50 @@ public void testBloomFilterSize() throws IOException { htd.addFamily(hcd); HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); Path path = new Path(DIR + "testBloomFilterSize"); - region = HRegion.createHRegion(info, path, conf, htd); - - int num_unique_rows = 10; - int duplicate_multiplier =2; - int num_storefiles = 4; - - int version = 0; - for (int f =0 ; f < num_storefiles; f++) { - for (int i = 0; i < duplicate_multiplier; i ++) { - for (int j = 0; j < num_unique_rows; j++) { - Put put = new Put(Bytes.toBytes("row" + j)); - put.setWriteToWAL(false); - put.add(fam1, qf1, version++, val1); - region.put(put); + this.region = HRegion.createHRegion(info, path, conf, htd); + try { + int num_unique_rows = 10; + int duplicate_multiplier =2; + int num_storefiles = 4; + + int version = 0; + for (int f =0 ; f < num_storefiles; f++) { + for (int i = 0; i < duplicate_multiplier; i ++) { + for (int j = 0; j < num_unique_rows; j++) { + Put put = new Put(Bytes.toBytes("row" + j)); + put.setWriteToWAL(false); + put.add(fam1, qf1, version++, val1); + region.put(put); + } } + region.flushcache(); + } + //before compaction + Store store = region.getStore(fam1); + List storeFiles = store.getStorefiles(); + for (StoreFile storefile : storeFiles) { + StoreFile.Reader reader = storefile.getReader(); + reader.loadFileInfo(); + reader.loadBloomfilter(); + assertEquals(num_unique_rows*duplicate_multiplier, reader.getEntries()); + assertEquals(num_unique_rows, reader.getFilterEntries()); } - region.flushcache(); - } - //before compaction - Store store = region.getStore(fam1); - List storeFiles = store.getStorefiles(); - for (StoreFile storefile : storeFiles) { - StoreFile.Reader reader = storefile.getReader(); - reader.loadFileInfo(); - reader.loadBloomfilter(); - assertEquals(num_unique_rows*duplicate_multiplier, reader.getEntries()); - assertEquals(num_unique_rows, reader.getFilterEntries()); - } - - region.compactStores(true); - //after compaction - storeFiles = store.getStorefiles(); - for (StoreFile storefile : storeFiles) { - StoreFile.Reader reader = storefile.getReader(); - reader.loadFileInfo(); - reader.loadBloomfilter(); - assertEquals(num_unique_rows*duplicate_multiplier*num_storefiles, - reader.getEntries()); - assertEquals(num_unique_rows, reader.getFilterEntries()); + region.compactStores(true); + + //after compaction + storeFiles = store.getStorefiles(); + for (StoreFile storefile : storeFiles) { + StoreFile.Reader reader = storefile.getReader(); + reader.loadFileInfo(); + reader.loadBloomfilter(); + assertEquals(num_unique_rows*duplicate_multiplier*num_storefiles, + reader.getEntries()); + assertEquals(num_unique_rows, reader.getFilterEntries()); + } + } finally { + HRegion.closeHRegion(this.region); + this.region = null; } } @@ -3096,32 +3794,36 @@ public void testAllColumnsWithBloomFilter() throws IOException { htd.addFamily(hcd); HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); Path path = new Path(DIR + "testAllColumnsWithBloomFilter"); - region = HRegion.createHRegion(info, path, conf, htd); - - // For row:0, col:0: insert versions 1 through 5. - byte row[] = Bytes.toBytes("row:" + 0); - byte column[] = Bytes.toBytes("column:" + 0); - Put put = new Put(row); - put.setWriteToWAL(false); - for (long idx = 1; idx <= 4; idx++) { - put.add(FAMILY, column, idx, Bytes.toBytes("value-version-" + idx)); - } - region.put(put); - - //Flush - region.flushcache(); + this.region = HRegion.createHRegion(info, path, conf, htd); + try { + // For row:0, col:0: insert versions 1 through 5. + byte row[] = Bytes.toBytes("row:" + 0); + byte column[] = Bytes.toBytes("column:" + 0); + Put put = new Put(row); + put.setWriteToWAL(false); + for (long idx = 1; idx <= 4; idx++) { + put.add(FAMILY, column, idx, Bytes.toBytes("value-version-" + idx)); + } + region.put(put); - //Get rows - Get get = new Get(row); - get.setMaxVersions(); - KeyValue[] kvs = region.get(get, null).raw(); + //Flush + region.flushcache(); - //Check if rows are correct - assertEquals(4, kvs.length); - checkOneCell(kvs[0], FAMILY, 0, 0, 4); - checkOneCell(kvs[1], FAMILY, 0, 0, 3); - checkOneCell(kvs[2], FAMILY, 0, 0, 2); - checkOneCell(kvs[3], FAMILY, 0, 0, 1); + //Get rows + Get get = new Get(row); + get.setMaxVersions(); + KeyValue[] kvs = region.get(get, null).raw(); + + //Check if rows are correct + assertEquals(4, kvs.length); + checkOneCell(kvs[0], FAMILY, 0, 0, 4); + checkOneCell(kvs[1], FAMILY, 0, 0, 3); + checkOneCell(kvs[2], FAMILY, 0, 0, 2); + checkOneCell(kvs[3], FAMILY, 0, 0, 1); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } /** @@ -3142,27 +3844,31 @@ public void testDeleteRowWithBloomFilter() throws IOException { htd.addFamily(hcd); HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); Path path = new Path(DIR + "TestDeleteRowWithBloomFilter"); - region = HRegion.createHRegion(info, path, conf, htd); - - // Insert some data - byte row[] = Bytes.toBytes("row1"); - byte col[] = Bytes.toBytes("col1"); + this.region = HRegion.createHRegion(info, path, conf, htd); + try { + // Insert some data + byte row[] = Bytes.toBytes("row1"); + byte col[] = Bytes.toBytes("col1"); - Put put = new Put(row); - put.add(familyName, col, 1, Bytes.toBytes("SomeRandomValue")); - region.put(put); - region.flushcache(); + Put put = new Put(row); + put.add(familyName, col, 1, Bytes.toBytes("SomeRandomValue")); + region.put(put); + region.flushcache(); - Delete del = new Delete(row); - region.delete(del, null, true); - region.flushcache(); + Delete del = new Delete(row); + region.delete(del, null, true); + region.flushcache(); - // Get remaining rows (should have none) - Get get = new Get(row); - get.addColumn(familyName, col); + // Get remaining rows (should have none) + Get get = new Get(row); + get.addColumn(familyName, col); - KeyValue[] keyValues = region.get(get, null).raw(); - assertTrue(keyValues.length == 0); + KeyValue[] keyValues = region.get(get, null).raw(); + assertTrue(keyValues.length == 0); + } finally { + HRegion.closeHRegion(this.region); + this.region = null; + } } @Test public void testgetHDFSBlocksDistribution() throws Exception { @@ -3173,7 +3879,7 @@ public void testDeleteRowWithBloomFilter() throws IOException { // set up a cluster with 3 nodes - MiniHBaseCluster cluster; + MiniHBaseCluster cluster = null; String dataNodeHosts[] = new String[] { "host1", "host2", "host3" }; int regionServersCount = 3; @@ -3192,17 +3898,17 @@ public void testDeleteRowWithBloomFilter() throws IOException { ht.put(put); HRegion firstRegion = htu.getHBaseCluster(). - getRegions(Bytes.toBytes(this.getName())).get(0); + getRegions(Bytes.toBytes(this.getName())).get(0); firstRegion.flushcache(); HDFSBlocksDistribution blocksDistribution1 = - firstRegion.getHDFSBlocksDistribution(); + firstRegion.getHDFSBlocksDistribution(); // given the default replication factor is 2 and we have 2 HFiles, // we will have total of 4 replica of blocks on 3 datanodes; thus there // must be at least one host that have replica for 2 HFiles. That host's // weight will be equal to the unique block weight. long uniqueBlocksWeight1 = - blocksDistribution1.getUniqueBlocksTotalWeight(); + blocksDistribution1.getUniqueBlocksTotalWeight(); String topHost = blocksDistribution1.getTopHosts().get(0); long topHostWeight = blocksDistribution1.getWeight(topHost); @@ -3221,10 +3927,261 @@ public void testDeleteRowWithBloomFilter() throws IOException { ht.close(); } finally { - htu.shutdownMiniCluster(); + if (cluster != null) { + htu.shutdownMiniCluster(); + } } } + /** + * Test case to check put function with memstore flushing for same row, same ts + * @throws Exception + */ + public void testPutWithMemStoreFlush() throws Exception { + Configuration conf = HBaseConfiguration.create(); + String method = "testPutWithMemStoreFlush"; + byte[] tableName = Bytes.toBytes(method); + byte[] family = Bytes.toBytes("family");; + byte[] qualifier = Bytes.toBytes("qualifier"); + byte[] row = Bytes.toBytes("putRow"); + byte[] value = null; + this.region = initHRegion(tableName, method, conf, family); + Put put = null; + Get get = null; + List kvs = null; + Result res = null; + + put = new Put(row); + value = Bytes.toBytes("value0"); + put.add(family, qualifier, 1234567l, value); + region.put(put); + get = new Get(row); + get.addColumn(family, qualifier); + get.setMaxVersions(); + res = this.region.get(get, null); + kvs = res.getColumn(family, qualifier); + assertEquals(1, kvs.size()); + assertEquals(Bytes.toBytes("value0"), kvs.get(0).getValue()); + + region.flushcache(); + get = new Get(row); + get.addColumn(family, qualifier); + get.setMaxVersions(); + res = this.region.get(get, null); + kvs = res.getColumn(family, qualifier); + assertEquals(1, kvs.size()); + assertEquals(Bytes.toBytes("value0"), kvs.get(0).getValue()); + + put = new Put(row); + value = Bytes.toBytes("value1"); + put.add(family, qualifier, 1234567l, value); + region.put(put); + get = new Get(row); + get.addColumn(family, qualifier); + get.setMaxVersions(); + res = this.region.get(get, null); + kvs = res.getColumn(family, qualifier); + assertEquals(1, kvs.size()); + assertEquals(Bytes.toBytes("value1"), kvs.get(0).getValue()); + + region.flushcache(); + get = new Get(row); + get.addColumn(family, qualifier); + get.setMaxVersions(); + res = this.region.get(get, null); + kvs = res.getColumn(family, qualifier); + assertEquals(1, kvs.size()); + assertEquals(Bytes.toBytes("value1"), kvs.get(0).getValue()); + } + + /** + * TestCase for increment + * + */ + private static class Incrementer implements Runnable { + private HRegion region; + private final static byte[] incRow = Bytes.toBytes("incRow"); + private final static byte[] family = Bytes.toBytes("family"); + private final static byte[] qualifier = Bytes.toBytes("qualifier"); + private final static long ONE = 1l; + private int incCounter; + + public Incrementer(HRegion region, int incCounter) { + this.region = region; + this.incCounter = incCounter; + } + + @Override + public void run() { + int count = 0; + while (count < incCounter) { + Increment inc = new Increment(incRow); + inc.addColumn(family, qualifier, ONE); + count++; + try { + region.increment(inc, null, true); + } catch (IOException e) { + e.printStackTrace(); + break; + } + } + } + } + + /** + * TestCase for append + * + */ + private static class Appender implements Runnable { + private HRegion region; + private final static byte[] appendRow = Bytes.toBytes("appendRow"); + private final static byte[] family = Bytes.toBytes("family"); + private final static byte[] qualifier = Bytes.toBytes("qualifier"); + private final static byte[] CHAR = Bytes.toBytes("a"); + private int appendCounter; + + public Appender(HRegion region, int appendCounter) { + this.region = region; + this.appendCounter = appendCounter; + } + + @Override + public void run() { + int count = 0; + while (count < appendCounter) { + Append app = new Append(appendRow); + app.add(family, qualifier, CHAR); + count++; + try { + region.append(app, null, true); + } catch (IOException e) { + e.printStackTrace(); + break; + } + } + } + } + + /** + * Test case to check append function with memstore flushing + * + * @throws Exception + */ + @Test + public void testParallelAppendWithMemStoreFlush() throws Exception { + Configuration conf = HBaseConfiguration.create(); + String method = "testParallelAppendWithMemStoreFlush"; + byte[] tableName = Bytes.toBytes(method); + byte[] family = Appender.family; + this.region = initHRegion(tableName, method, conf, family); + final HRegion region = this.region; + final AtomicBoolean appendDone = new AtomicBoolean(false); + Runnable flusher = new Runnable() { + @Override + public void run() { + while (!appendDone.get()) { + try { + region.flushcache(); + } catch (Exception e) { + e.printStackTrace(); + } + } + } + }; + + // after all append finished, the value will append to threadNum * appendCounter Appender.CHAR + int threadNum = 20; + int appendCounter = 100; + byte[] expected = new byte[threadNum * appendCounter]; + for (int i = 0; i < threadNum * appendCounter; i++) { + System.arraycopy(Appender.CHAR, 0, expected, i, 1); + } + Thread[] appenders = new Thread[threadNum]; + Thread flushThread = new Thread(flusher); + for (int i = 0; i < threadNum; i++) { + appenders[i] = new Thread(new Appender(this.region, appendCounter)); + appenders[i].start(); + } + flushThread.start(); + for (int i = 0; i < threadNum; i++) { + appenders[i].join(); + } + + appendDone.set(true); + flushThread.join(); + + Get get = new Get(Appender.appendRow); + get.addColumn(Appender.family, Appender.qualifier); + get.setMaxVersions(1); + Result res = this.region.get(get, null); + List kvs = res.getColumn(Appender.family, Appender.qualifier); + + // we just got the latest version + assertEquals(kvs.size(), 1); + KeyValue kv = kvs.get(0); + byte[] appendResult = new byte[kv.getValueLength()]; + System.arraycopy(kv.getBuffer(), kv.getValueOffset(), appendResult, 0, kv.getValueLength()); + assertEquals(expected, appendResult); + this.region = null; + } + + /** + * Test case to check increment function with memstore flushing + * @throws Exception + */ + @Test + public void testParallelIncrementWithMemStoreFlush() throws Exception { + String method = "testParallelIncrementWithMemStoreFlush"; + byte[] tableName = Bytes.toBytes(method); + byte[] family = Incrementer.family; + this.region = initHRegion(tableName, method, conf, family); + final HRegion region = this.region; + final AtomicBoolean incrementDone = new AtomicBoolean(false); + Runnable reader = new Runnable() { + @Override + public void run() { + while (!incrementDone.get()) { + try { + region.flushcache(); + } catch (Exception e) { + e.printStackTrace(); + } + } + } + }; + + //after all increment finished, the row will increment to 20*100 = 2000 + int threadNum = 20; + int incCounter = 100; + long expected = threadNum * incCounter; + Thread[] incrementers = new Thread[threadNum]; + Thread flushThread = new Thread(reader); + for (int i = 0; i < threadNum; i++) { + incrementers[i] = new Thread(new Incrementer(this.region, incCounter)); + incrementers[i].start(); + } + flushThread.start(); + for (int i = 0; i < threadNum; i++) { + incrementers[i].join(); + } + + incrementDone.set(true); + flushThread.join(); + + Get get = new Get(Incrementer.incRow); + get.addColumn(Incrementer.family, Incrementer.qualifier); + get.setMaxVersions(1); + Result res = this.region.get(get, null); + List kvs = res.getColumn(Incrementer.family, + Incrementer.qualifier); + + //we just got the latest version + assertEquals(kvs.size(), 1); + KeyValue kv = kvs.get(0); + assertEquals(expected, Bytes.toLong(kv.getBuffer(), kv.getValueOffset())); + this.region = null; + } + private void putData(int startRow, int numRows, byte [] qf, byte [] ...families) throws IOException { @@ -3305,8 +4262,45 @@ private void assertScan(final HRegion r, final byte [] fs, } } + /** + * Test that we get the expected flush results back + * @throws IOException + */ + @Test + public void testFlushResult() throws IOException { + String method = "testFlushResult"; + byte[] tableName = Bytes.toBytes(method); + byte[] family = Bytes.toBytes("family"); + + this.region = initHRegion(tableName, method, conf, family); + + // empty memstore, flush doesn't run + HRegion.FlushResult fr = region.flushcache(); + assertFalse(fr.isFlushSucceeded()); + assertFalse(fr.isCompactionNeeded()); + + // Flush enough files to get up to the threshold, doesn't need compactions + for (int i = 0; i < 3; i++) { + Put put = new Put(tableName).add(family, family, tableName); + region.put(put); + fr = region.flushcache(); + assertTrue(fr.isFlushSucceeded()); + assertFalse(fr.isCompactionNeeded()); + } + + // Two flushes after the threshold, compactions are needed + for (int i = 0; i < 2; i++) { + Put put = new Put(tableName).add(family, family, tableName); + region.put(put); + fr = region.flushcache(); + assertTrue(fr.isFlushSucceeded()); + assertTrue(fr.isCompactionNeeded()); + } + } + private Configuration initSplit() { - Configuration conf = HBaseConfiguration.create(); + Configuration conf = HBaseConfiguration.create(this.conf); + // Always compact if there is more than one store file. conf.setInt("hbase.hstore.compactionThreshold", 2); @@ -3324,33 +4318,63 @@ private Configuration initSplit() { return conf; } - private void initHRegion (byte [] tableName, String callingMethod, - byte[] ... families) - throws IOException { - initHRegion(tableName, callingMethod, HBaseConfiguration.create(), families); + /** + * @param tableName + * @param callingMethod + * @param conf + * @param families + * @throws IOException + * @return A region on which you must call {@link HRegion#closeHRegion(HRegion)} when done. + */ + public static HRegion initHRegion (byte [] tableName, String callingMethod, + Configuration conf, byte [] ... families) + throws IOException{ + return initHRegion(tableName, null, null, callingMethod, conf, false, families); } - private void initHRegion (byte [] tableName, String callingMethod, - Configuration conf, byte [] ... families) + /** + * @param tableName + * @param callingMethod + * @param conf + * @param isReadOnly + * @param families + * @throws IOException + * @return A region on which you must call {@link HRegion#closeHRegion(HRegion)} when done. + */ + public static HRegion initHRegion (byte [] tableName, String callingMethod, + Configuration conf, boolean isReadOnly, byte [] ... families) throws IOException{ - initHRegion(tableName, null, null, callingMethod, conf, families); + return initHRegion(tableName, null, null, callingMethod, conf, isReadOnly, families); } - private void initHRegion(byte[] tableName, byte[] startKey, byte[] stopKey, - String callingMethod, Configuration conf, byte[]... families) + /** + * @param tableName + * @param startKey + * @param stopKey + * @param callingMethod + * @param conf + * @param isReadOnly + * @param families + * @throws IOException + * @return A region on which you must call {@link HRegion#closeHRegion(HRegion)} when done. + */ + private static HRegion initHRegion(byte[] tableName, byte[] startKey, byte[] stopKey, + String callingMethod, Configuration conf, boolean isReadOnly, byte[]... families) throws IOException { HTableDescriptor htd = new HTableDescriptor(tableName); + htd.setReadOnly(isReadOnly); for(byte [] family : families) { htd.addFamily(new HColumnDescriptor(family)); } HRegionInfo info = new HRegionInfo(htd.getName(), startKey, stopKey, false); Path path = new Path(DIR + callingMethod); + FileSystem fs = FileSystem.get(conf); if (fs.exists(path)) { if (!fs.delete(path, true)) { throw new IOException("Failed delete of " + path); } } - region = HRegion.createHRegion(info, path, conf, htd); + return HRegion.createHRegion(info, path, conf, htd); } /** diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java new file mode 100644 index 000000000000..10a93702e903 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java @@ -0,0 +1,90 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.RegionTooBusyException; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * TestHRegion with hbase.busy.wait.duration set to 1000 (1 second). + * We can't use parameterized test since TestHRegion is old fashion. + */ +@Category(MediumTests.class) +@SuppressWarnings("deprecation") +public class TestHRegionBusyWait extends TestHRegion { + public TestHRegionBusyWait() { + conf.set("hbase.busy.wait.duration", "1000"); + } + + /** + * Test RegionTooBusyException thrown when region is busy + */ + @Test (timeout=2000) + public void testRegionTooBusy() throws IOException { + String method = "testRegionTooBusy"; + byte[] tableName = Bytes.toBytes(method); + byte[] family = Bytes.toBytes("family"); + region = initHRegion(tableName, method, conf, family); + final AtomicBoolean stopped = new AtomicBoolean(true); + Thread t = new Thread(new Runnable() { + @Override + public void run() { + try { + region.lock.writeLock().lock(); + stopped.set(false); + while (!stopped.get()) { + Thread.sleep(100); + } + } catch (InterruptedException ie) { + } finally { + region.lock.writeLock().unlock(); + } + } + }); + t.start(); + Get get = new Get(row); + try { + while (stopped.get()) { + Thread.sleep(100); + } + region.get(get, null); + fail("Should throw RegionTooBusyException"); + } catch (InterruptedException ie) { + fail("test interrupted"); + } catch (RegionTooBusyException e) { + // Good, expected + } finally { + stopped.set(true); + try { + t.join(); + } catch (Throwable e) { + } + + HRegion.closeHRegion(region); + region = null; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java index 6e1211bd7d40..0098fc94aa13 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -21,16 +20,13 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; -import java.io.IOException; - -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.SmallTests; import org.apache.hadoop.hbase.util.Bytes; -import org.apache.hadoop.hbase.util.FSTableDescriptors; -import org.apache.hadoop.hbase.util.FSUtils; import org.apache.hadoop.hbase.util.MD5Hash; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -60,32 +56,7 @@ public void testCreateHRegionInfoName() throws Exception { + id + "." + md5HashInHex + ".", nameStr); } - - @Test - public void testGetSetOfHTD() throws IOException { - HBaseTestingUtility HTU = new HBaseTestingUtility(); - final String tablename = "testGetSetOfHTD"; - // Delete the temporary table directory that might still be there from the - // previous test run. - FSTableDescriptors.deleteTableDescriptorIfExists(tablename, - HTU.getConfiguration()); - - HTableDescriptor htd = new HTableDescriptor(tablename); - FSTableDescriptors.createTableDescriptor(htd, HTU.getConfiguration()); - HRegionInfo hri = new HRegionInfo(Bytes.toBytes("testGetSetOfHTD"), - HConstants.EMPTY_START_ROW, HConstants.EMPTY_END_ROW); - HTableDescriptor htd2 = hri.getTableDesc(); - assertTrue(htd.equals(htd2)); - final String key = "SOME_KEY"; - assertNull(htd.getValue(key)); - final String value = "VALUE"; - htd.setValue(key, value); - hri.setTableDesc(htd); - HTableDescriptor htd3 = hri.getTableDesc(); - assertTrue(htd.equals(htd3)); - } - @Test public void testContainsRange() { HTableDescriptor tableDesc = new HTableDescriptor("testtable"); @@ -105,7 +76,7 @@ public void testContainsRange() { assertFalse(hri.containsRange(Bytes.toBytes("g"), Bytes.toBytes("g"))); // Single row range entirely outside assertFalse(hri.containsRange(Bytes.toBytes("z"), Bytes.toBytes("z"))); - + // Degenerate range try { hri.containsRange(Bytes.toBytes("z"), Bytes.toBytes("a")); @@ -130,6 +101,18 @@ public void testMetaTables() { assertTrue(HRegionInfo.FIRST_META_REGIONINFO.isMetaTable()); } + @Test + public void testComparator() { + byte[] tablename = Bytes.toBytes("comparatorTablename"); + byte[] empty = new byte[0]; + HRegionInfo older = new HRegionInfo(tablename, empty, empty, false, 0L); + HRegionInfo newer = new HRegionInfo(tablename, empty, empty, false, 1L); + assertTrue(older.compareTo(newer) < 0); + assertTrue(newer.compareTo(older) > 0); + assertTrue(older.compareTo(older) == 0); + assertTrue(newer.compareTo(newer) == 0); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionOnCluster.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionOnCluster.java new file mode 100644 index 000000000000..e0cf4155865d --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionOnCluster.java @@ -0,0 +1,155 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import static org.junit.Assert.*; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.MediumTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Tests that need to spin up a cluster testing an {@link HRegion}. Use + * {@link TestHRegion} if you don't need a cluster, if you can test w/ a + * standalone {@link HRegion}. + */ +@Category(MediumTests.class) +public class TestHRegionOnCluster { + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + @Test (timeout=180000) + public void testDataCorrectnessReplayingRecoveredEdits() throws Exception { + final int NUM_MASTERS = 1; + final int NUM_RS = 3; + TEST_UTIL.startMiniCluster(NUM_MASTERS, NUM_RS); + + try { + final byte[] TABLENAME = Bytes + .toBytes("testDataCorrectnessReplayingRecoveredEdits"); + final byte[] FAMILY = Bytes.toBytes("family"); + MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + HMaster master = cluster.getMaster(); + + // Create table + HTableDescriptor desc = new HTableDescriptor(TABLENAME); + desc.addFamily(new HColumnDescriptor(FAMILY)); + HBaseAdmin hbaseAdmin = TEST_UTIL.getHBaseAdmin(); + hbaseAdmin.createTable(desc); + + assertTrue(hbaseAdmin.isTableAvailable(TABLENAME)); + + // Put data: r1->v1 + HTable table = new HTable(TEST_UTIL.getConfiguration(), TABLENAME); + putDataAndVerify(table, "r1", FAMILY, "v1", 1); + + // Move region to target server + HRegionInfo regionInfo = table.getRegionLocation("r1").getRegionInfo(); + int originServerNum = cluster.getServerWith(regionInfo.getRegionName()); + HRegionServer originServer = cluster.getRegionServer(originServerNum); + int targetServerNum = (originServerNum + 1) % NUM_RS; + HRegionServer targetServer = cluster.getRegionServer(targetServerNum); + assertFalse(originServer.equals(targetServer)); + + do { + Thread.sleep(10); + } while (!originServer.getServerName().equals( + cluster.getMaster().getAssignmentManager().getRegionServerOfRegion(regionInfo))); + + hbaseAdmin.move(regionInfo.getEncodedNameAsBytes(), + Bytes.toBytes(targetServer.getServerName().getServerName())); + + do { + Thread.sleep(10); + } while (cluster.getServerWith(regionInfo.getRegionName()) == originServerNum || + !targetServer.getServerName().equals( + cluster.getMaster().getAssignmentManager().getRegionServerOfRegion(regionInfo))); + + // Put data: r2->v2 + putDataAndVerify(table, "r2", FAMILY, "v2", 2); + + // Move region to origin server + hbaseAdmin.move(regionInfo.getEncodedNameAsBytes(), + Bytes.toBytes(originServer.getServerName().getServerName())); + do { + Thread.sleep(1); + } while (cluster.getServerWith(regionInfo.getRegionName()) == targetServerNum); + + // Put data: r3->v3 + putDataAndVerify(table, "r3", FAMILY, "v3", 3); + + // Kill target server + targetServer.kill(); + cluster.getRegionServerThreads().get(targetServerNum).join(); + // Wait until finish processing of shutdown + while (master.getServerManager().areDeadServersInProgress()) { + Thread.sleep(5); + } + // Kill origin server + originServer.kill(); + cluster.getRegionServerThreads().get(originServerNum).join(); + + // Put data: r4->v4 + putDataAndVerify(table, "r4", FAMILY, "v4", 4); + + } finally { + TEST_UTIL.shutdownMiniCluster(); + } + } + + private void putDataAndVerify(HTable table, String row, byte[] family, + String value, int verifyNum) throws IOException { + System.out.println("=========Putting data :" + row); + Put put = new Put(Bytes.toBytes(row)); + put.add(family, Bytes.toBytes("q1"), Bytes.toBytes(value)); + table.put(put); + ResultScanner resultScanner = table.getScanner(new Scan()); + List results = new ArrayList(); + while (true) { + Result r = resultScanner.next(); + if (r == null) + break; + results.add(r); + } + resultScanner.close(); + if (results.size() != verifyNum) { + System.out.println(results); + } + assertEquals(verifyNum, results.size()); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java index a1bf73b6b6e4..6a5d7482c4c5 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java @@ -145,7 +145,7 @@ public Void call() throws Exception { LOG.debug("Going to connect to server " + location + " for row " + Bytes.toStringBinary(row)); byte[] regionName = location.getRegionInfo().getRegionName(); - server.bulkLoadHFiles(famPaths, regionName); + server.bulkLoadHFiles(famPaths, regionName, true); return null; } }.withRetries(); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java index 72d9c405a18b..85b35691a1a7 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java @@ -276,16 +276,16 @@ public void testRawScan() throws Exception { p.add(c0, c0, T3); region.put(p); - Delete d = new Delete(T1, ts+1, null); - region.delete(d, null, true); + Delete d = new Delete(T1, ts+1); + region.delete(d, true); - d = new Delete(T1, ts+2, null); + d = new Delete(T1, ts+2); d.deleteColumn(c0, c0, ts+2); - region.delete(d, null, true); + region.delete(d, true); - d = new Delete(T1, ts+3, null); + d = new Delete(T1, ts+3); d.deleteColumns(c0, c0, ts+3); - region.delete(d, null, true); + region.delete(d, true); Scan s = new Scan(); s.setRaw(true); @@ -293,12 +293,54 @@ public void testRawScan() throws Exception { InternalScanner scan = region.getScanner(s); List kvs = new ArrayList(); scan.next(kvs); + assertEquals(8, kvs.size()); assertTrue(kvs.get(0).isDeleteFamily()); assertEquals(kvs.get(1).getValue(), T3); assertTrue(kvs.get(2).isDelete()); assertTrue(kvs.get(3).isDeleteType()); assertEquals(kvs.get(4).getValue(), T2); assertEquals(kvs.get(5).getValue(), T1); + // we have 3 CFs, so there are two more delete markers + assertTrue(kvs.get(6).isDeleteFamily()); + assertTrue(kvs.get(7).isDeleteFamily()); + + // verify that raw scans honor the passed timerange + s = new Scan(); + s.setRaw(true); + s.setMaxVersions(); + s.setTimeRange(0, 1); + scan = region.getScanner(s); + kvs = new ArrayList(); + scan.next(kvs); + // nothing in this interval, not even delete markers + assertTrue(kvs.isEmpty()); + + // filter new delete markers + s = new Scan(); + s.setRaw(true); + s.setMaxVersions(); + s.setTimeRange(0, ts+2); + scan = region.getScanner(s); + kvs = new ArrayList(); + scan.next(kvs); + assertEquals(4, kvs.size()); + assertTrue(kvs.get(0).isDeleteFamily()); + assertEquals(kvs.get(1).getValue(), T1); + // we have 3 CFs + assertTrue(kvs.get(2).isDeleteFamily()); + assertTrue(kvs.get(3).isDeleteFamily()); + + // filter old delete markers + s = new Scan(); + s.setRaw(true); + s.setMaxVersions(); + s.setTimeRange(ts+3, ts+5); + scan = region.getScanner(s); + kvs = new ArrayList(); + scan.next(kvs); + assertEquals(2, kvs.size()); + assertEquals(kvs.get(0).getValue(), T3); + assertTrue(kvs.get(1).isDelete()); region.close(); region.getLog().closeAndDelete(); @@ -447,16 +489,16 @@ public void testRanges() throws Exception { p.add(c1, c1, T2); region.put(p); - Delete d = new Delete(T1, ts+1, null); - d.deleteColumns(c0, c0, ts+1); + Delete d = new Delete(T1, ts+2, null); + d.deleteColumns(c0, c0, ts+2); region.delete(d, null, true); - d = new Delete(T1, ts+1, null); - d.deleteFamily(c1, ts+1); + d = new Delete(T1, ts+2, null); + d.deleteFamily(c1, ts+2); region.delete(d, null, true); - d = new Delete(T2, ts+1, null); - d.deleteFamily(c0, ts+1); + d = new Delete(T2, ts+2, null); + d.deleteFamily(c0, ts+2); region.delete(d, null, true); // add an older delete, to make sure it is filtered @@ -464,7 +506,7 @@ public void testRanges() throws Exception { d.deleteFamily(c1, ts-10); region.delete(d, null, true); - // ts + 2 does NOT include the delete at ts+1 + // ts + 2 does NOT include the delete at ts+2 checkGet(region, T1, c0, c0, ts+2, T2, T1); checkGet(region, T1, c0, c1, ts+2, T2, T1); checkGet(region, T1, c1, c0, ts+2, T2, T1); @@ -610,10 +652,10 @@ public void testWithMixedCFs() throws Exception { region.put(p); // family markers are each family - Delete d = new Delete(T1, ts, null); + Delete d = new Delete(T1, ts+1, null); region.delete(d, null, true); - d = new Delete(T2, ts+1, null); + d = new Delete(T2, ts+2, null); region.delete(d, null, true); Scan s = new Scan(T1); @@ -728,7 +770,8 @@ private void checkGet(HRegion region, byte[] row, byte[] fam, byte[] col, private int countDeleteMarkers(HRegion region) throws IOException { Scan s = new Scan(); s.setRaw(true); - s.setMaxVersions(); + // use max versions from the store(s) + s.setMaxVersions(region.getStores().values().iterator().next().getScanInfo().getMaxVersions()); InternalScanner scan = region.getScanner(s); List kvs = new ArrayList(); int res = 0; diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java index 69f57ee98361..71d8a9e3d773 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueScanFixture.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueScanFixture.java index 1cc893a519bb..edc187b4a4a5 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueScanFixture.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueScanFixture.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueSkipListSet.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueSkipListSet.java index d9158bebf9c9..5f0feb50f415 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueSkipListSet.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueSkipListSet.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMXBean.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMXBean.java index 83e02c7a214e..cfa79057e720 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMXBean.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMXBean.java @@ -21,10 +21,13 @@ import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.MediumTests; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; +import org.junit.experimental.categories.Category; +@Category(MediumTests.class) public class TestMXBean { private static final HBaseTestingUtility TEST_UTIL = diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java index e91d83caef7f..4453f0ea1b73 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java index b1214f6c8540..fa4a64c202d9 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -25,7 +24,9 @@ import java.rmi.UnexpectedException; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.List; +import java.util.Map; import java.util.concurrent.atomic.AtomicReference; import junit.framework.TestCase; @@ -36,9 +37,10 @@ import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.regionserver.Store.ScanInfo; -import org.apache.hadoop.hbase.regionserver.StoreScanner.ScanType; import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.EnvironmentEdge; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; import com.google.common.base.Joiner; import com.google.common.collect.Iterables; @@ -851,7 +853,119 @@ public void testUpsertMSLAB() throws Exception { } /** - * Adds {@link #ROW_COUNT} rows and {@link #QUALIFIER_COUNT} + * Add keyvalues with a fixed memstoreTs, and checks that memstore size is decreased + * as older keyvalues are deleted from the memstore. + * @throws Exception + */ + public void testUpsertMemstoreSize() throws Exception { + Configuration conf = HBaseConfiguration.create(); + memstore = new MemStore(conf, KeyValue.COMPARATOR); + long oldSize = memstore.size.get(); + + KeyValue kv1 = KeyValueTestUtil.create("r", "f", "q", 100, "v"); + this.memstore.upsert(Collections.singletonList(kv1)); + long newSize = this.memstore.size.get(); + assert(newSize > oldSize); + + KeyValue kv2 = KeyValueTestUtil.create("r", "f", "q", 101, "v"); + this.memstore.upsert(Collections.singletonList(kv2)); + assertEquals(newSize, this.memstore.size.get()); + } + + //////////////////////////////////// + // Test for periodic memstore flushes + // based on time of oldest edit + //////////////////////////////////// + + /** + * Tests that the timeOfOldestEdit is updated correctly for the + * various edit operations in memstore. + * @throws Exception + */ + public void testUpdateToTimeOfOldestEdit() throws Exception { + try { + EnvironmentEdgeForMemstoreTest edge = new EnvironmentEdgeForMemstoreTest(); + EnvironmentEdgeManager.injectEdge(edge); + MemStore memstore = new MemStore(); + long t = memstore.timeOfOldestEdit(); + assertEquals(t, Long.MAX_VALUE); + + // test the case that the timeOfOldestEdit is updated after a KV add + memstore.add(KeyValueTestUtil.create("r", "f", "q", 100, "v")); + t = memstore.timeOfOldestEdit(); + assertTrue(t == 1234); + // snapshot() will reset timeOfOldestEdit. The method will also assert the + // value is reset to Long.MAX_VALUE + t = runSnapshot(memstore); + + // test the case that the timeOfOldestEdit is updated after a KV delete + memstore.delete(KeyValueTestUtil.create("r", "f", "q", 100, "v")); + t = memstore.timeOfOldestEdit(); + assertTrue(t == 1234); + t = runSnapshot(memstore); + + // test the case that the timeOfOldestEdit is updated after a KV upsert + List l = new ArrayList(); + KeyValue kv1 = KeyValueTestUtil.create("r", "f", "q", 100, "v"); + l.add(kv1); + memstore.upsert(l); + t = memstore.timeOfOldestEdit(); + assertTrue(t == 1234); + } finally { + EnvironmentEdgeManager.reset(); + } + } + + /** + * Tests the HRegion.shouldFlush method - adds an edit in the memstore + * and checks that shouldFlush returns true, and another where it disables + * the periodic flush functionality and tests whether shouldFlush returns + * false. + * @throws Exception + */ + public void testShouldFlush() throws Exception { + Configuration conf = new Configuration(); + conf.setInt(HRegion.MEMSTORE_PERIODIC_FLUSH_INTERVAL, 1000); + checkShouldFlush(conf, true); + // test disable flush + conf.setInt(HRegion.MEMSTORE_PERIODIC_FLUSH_INTERVAL, 0); + checkShouldFlush(conf, false); + } + + private void checkShouldFlush(Configuration conf, boolean expected) throws Exception { + try { + EnvironmentEdgeForMemstoreTest edge = new EnvironmentEdgeForMemstoreTest(); + EnvironmentEdgeManager.injectEdge(edge); + HBaseTestingUtility hbaseUtility = new HBaseTestingUtility(conf); + HRegion region = hbaseUtility.createTestRegion("foobar", new HColumnDescriptor("foo")); + + Map stores = region.getStores(); + assertTrue(stores.size() == 1); + + Store s = stores.entrySet().iterator().next().getValue(); + edge.setCurrentTimeMillis(1234); + s.add(KeyValueTestUtil.create("r", "f", "q", 100, "v")); + edge.setCurrentTimeMillis(1234 + 100); + assertTrue(region.shouldFlush() == false); + edge.setCurrentTimeMillis(1234 + 10000); + assertTrue(region.shouldFlush() == expected); + } finally { + EnvironmentEdgeManager.reset(); + } + } + + private class EnvironmentEdgeForMemstoreTest implements EnvironmentEdge { + long t = 1234; + @Override + public long currentTimeMillis() { + return t; + } + public void setCurrentTimeMillis(long t) { + this.t = t; + } + } + + /** * Adds {@link #ROW_COUNT} rows and {@link #QUALIFIER_COUNT} * @param hmc Instance to add rows to. * @return How many rows we added. * @throws IOException @@ -879,14 +993,17 @@ private int addRows(final MemStore hmc, final long ts) { return ROW_COUNT; } - private void runSnapshot(final MemStore hmc) throws UnexpectedException { + private long runSnapshot(final MemStore hmc) throws UnexpectedException { // Save off old state. int oldHistorySize = hmc.getSnapshot().size(); hmc.snapshot(); KeyValueSkipListSet ss = hmc.getSnapshot(); // Make some assertions about what just happened. assertTrue("History size has not increased", oldHistorySize < ss.size()); + long t = memstore.timeOfOldestEdit(); + assertTrue("Time of oldest edit is not Long.MAX_VALUE", t == Long.MAX_VALUE); hmc.clearSnapshot(ss); + return t; } private void isExpectedRowWithoutTimestamps(final int rowIndex, diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java index d7b01ca72eae..8d0153a9353a 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java index 33c78ab7bc9b..4e8270e27411 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -52,40 +51,45 @@ public class TestMinVersions extends HBaseTestCase { public void testGetClosestBefore() throws Exception { HTableDescriptor htd = createTableDescriptor(getName(), 1, 1000, 1, false); HRegion region = createNewHRegion(htd, null, null); + try { - // 2s in the past - long ts = EnvironmentEdgeManager.currentTimeMillis() - 2000; + // 2s in the past + long ts = EnvironmentEdgeManager.currentTimeMillis() - 2000; - Put p = new Put(T1, ts); - p.add(c0, c0, T1); - region.put(p); + Put p = new Put(T1, ts); + p.add(c0, c0, T1); + region.put(p); - p = new Put(T1, ts+1); - p.add(c0, c0, T4); - region.put(p); + p = new Put(T1, ts+1); + p.add(c0, c0, T4); + region.put(p); - p = new Put(T3, ts); - p.add(c0, c0, T3); - region.put(p); + p = new Put(T3, ts); + p.add(c0, c0, T3); + region.put(p); - // now make sure that getClosestBefore(...) get can - // rows that would be expired without minVersion. - // also make sure it gets the latest version - Result r = region.getClosestRowBefore(T1, c0); - checkResult(r, c0, T4); + // now make sure that getClosestBefore(...) get can + // rows that would be expired without minVersion. + // also make sure it gets the latest version + Result r = region.getClosestRowBefore(T1, c0); + checkResult(r, c0, T4); - r = region.getClosestRowBefore(T2, c0); - checkResult(r, c0, T4); + r = region.getClosestRowBefore(T2, c0); + checkResult(r, c0, T4); - // now flush/compact - region.flushcache(); - region.compactStores(true); + // now flush/compact + region.flushcache(); + region.compactStores(true); - r = region.getClosestRowBefore(T1, c0); - checkResult(r, c0, T4); + r = region.getClosestRowBefore(T1, c0); + checkResult(r, c0, T4); - r = region.getClosestRowBefore(T2, c0); - checkResult(r, c0, T4); + r = region.getClosestRowBefore(T2, c0); + checkResult(r, c0, T4); + } finally { + region.close(); + region.getLog().closeAndDelete(); + } } /** @@ -96,48 +100,52 @@ public void testStoreMemStore() throws Exception { // keep 3 versions minimum HTableDescriptor htd = createTableDescriptor(getName(), 3, 1000, 1, false); HRegion region = createNewHRegion(htd, null, null); - // 2s in the past long ts = EnvironmentEdgeManager.currentTimeMillis() - 2000; - Put p = new Put(T1, ts-1); - p.add(c0, c0, T2); - region.put(p); - - p = new Put(T1, ts-3); - p.add(c0, c0, T0); - region.put(p); - - // now flush/compact - region.flushcache(); - region.compactStores(true); - - p = new Put(T1, ts); - p.add(c0, c0, T3); - region.put(p); - - p = new Put(T1, ts-2); - p.add(c0, c0, T1); - region.put(p); - - p = new Put(T1, ts-3); - p.add(c0, c0, T0); - region.put(p); - - // newest version in the memstore - // the 2nd oldest in the store file - // and the 3rd, 4th oldest also in the memstore - - Get g = new Get(T1); - g.setMaxVersions(); - Result r = region.get(g, null); // this'll use ScanWildcardColumnTracker - checkResult(r, c0, T3,T2,T1); - - g = new Get(T1); - g.setMaxVersions(); - g.addColumn(c0, c0); - r = region.get(g, null); // this'll use ExplicitColumnTracker - checkResult(r, c0, T3,T2,T1); + try { + Put p = new Put(T1, ts-1); + p.add(c0, c0, T2); + region.put(p); + + p = new Put(T1, ts-3); + p.add(c0, c0, T0); + region.put(p); + + // now flush/compact + region.flushcache(); + region.compactStores(true); + + p = new Put(T1, ts); + p.add(c0, c0, T3); + region.put(p); + + p = new Put(T1, ts-2); + p.add(c0, c0, T1); + region.put(p); + + p = new Put(T1, ts-3); + p.add(c0, c0, T0); + region.put(p); + + // newest version in the memstore + // the 2nd oldest in the store file + // and the 3rd, 4th oldest also in the memstore + + Get g = new Get(T1); + g.setMaxVersions(); + Result r = region.get(g, null); // this'll use ScanWildcardColumnTracker + checkResult(r, c0, T3,T2,T1); + + g = new Get(T1); + g.setMaxVersions(); + g.addColumn(c0, c0); + r = region.get(g, null); // this'll use ExplicitColumnTracker + checkResult(r, c0, T3,T2,T1); + } finally { + region.close(); + region.getLog().closeAndDelete(); + } } /** @@ -150,47 +158,52 @@ public void testDelete() throws Exception { // 2s in the past long ts = EnvironmentEdgeManager.currentTimeMillis() - 2000; - Put p = new Put(T1, ts-2); - p.add(c0, c0, T1); - region.put(p); - - p = new Put(T1, ts-1); - p.add(c0, c0, T2); - region.put(p); - - p = new Put(T1, ts); - p.add(c0, c0, T3); - region.put(p); - - Delete d = new Delete(T1, ts-1, null); - region.delete(d, null, true); - - Get g = new Get(T1); - g.setMaxVersions(); - Result r = region.get(g, null); // this'll use ScanWildcardColumnTracker - checkResult(r, c0, T3); - - g = new Get(T1); - g.setMaxVersions(); - g.addColumn(c0, c0); - r = region.get(g, null); // this'll use ExplicitColumnTracker - checkResult(r, c0, T3); - - // now flush/compact - region.flushcache(); - region.compactStores(true); - - // try again - g = new Get(T1); - g.setMaxVersions(); - r = region.get(g, null); // this'll use ScanWildcardColumnTracker - checkResult(r, c0, T3); - - g = new Get(T1); - g.setMaxVersions(); - g.addColumn(c0, c0); - r = region.get(g, null); // this'll use ExplicitColumnTracker - checkResult(r, c0, T3); + try { + Put p = new Put(T1, ts-2); + p.add(c0, c0, T1); + region.put(p); + + p = new Put(T1, ts-1); + p.add(c0, c0, T2); + region.put(p); + + p = new Put(T1, ts); + p.add(c0, c0, T3); + region.put(p); + + Delete d = new Delete(T1, ts-1, null); + region.delete(d, null, true); + + Get g = new Get(T1); + g.setMaxVersions(); + Result r = region.get(g, null); // this'll use ScanWildcardColumnTracker + checkResult(r, c0, T3); + + g = new Get(T1); + g.setMaxVersions(); + g.addColumn(c0, c0); + r = region.get(g, null); // this'll use ExplicitColumnTracker + checkResult(r, c0, T3); + + // now flush/compact + region.flushcache(); + region.compactStores(true); + + // try again + g = new Get(T1); + g.setMaxVersions(); + r = region.get(g, null); // this'll use ScanWildcardColumnTracker + checkResult(r, c0, T3); + + g = new Get(T1); + g.setMaxVersions(); + g.addColumn(c0, c0); + r = region.get(g, null); // this'll use ExplicitColumnTracker + checkResult(r, c0, T3); + } finally { + region.close(); + region.getLog().closeAndDelete(); + } } /** @@ -203,63 +216,68 @@ public void testMemStore() throws Exception { // 2s in the past long ts = EnvironmentEdgeManager.currentTimeMillis() - 2000; - // 2nd version - Put p = new Put(T1, ts-2); - p.add(c0, c0, T2); - region.put(p); - - // 3rd version - p = new Put(T1, ts-1); - p.add(c0, c0, T3); - region.put(p); - - // 4th version - p = new Put(T1, ts); - p.add(c0, c0, T4); - region.put(p); - - // now flush/compact - region.flushcache(); - region.compactStores(true); - - // now put the first version (backdated) - p = new Put(T1, ts-3); - p.add(c0, c0, T1); - region.put(p); - - // now the latest change is in the memstore, - // but it is not the latest version - - Result r = region.get(new Get(T1), null); - checkResult(r, c0, T4); - - Get g = new Get(T1); - g.setMaxVersions(); - r = region.get(g, null); // this'll use ScanWildcardColumnTracker - checkResult(r, c0, T4,T3); - - g = new Get(T1); - g.setMaxVersions(); - g.addColumn(c0, c0); - r = region.get(g, null); // this'll use ExplicitColumnTracker - checkResult(r, c0, T4,T3); - - p = new Put(T1, ts+1); - p.add(c0, c0, T5); - region.put(p); - - // now the latest version is in the memstore - - g = new Get(T1); - g.setMaxVersions(); - r = region.get(g, null); // this'll use ScanWildcardColumnTracker - checkResult(r, c0, T5,T4); - - g = new Get(T1); - g.setMaxVersions(); - g.addColumn(c0, c0); - r = region.get(g, null); // this'll use ExplicitColumnTracker - checkResult(r, c0, T5,T4); + try { + // 2nd version + Put p = new Put(T1, ts-2); + p.add(c0, c0, T2); + region.put(p); + + // 3rd version + p = new Put(T1, ts-1); + p.add(c0, c0, T3); + region.put(p); + + // 4th version + p = new Put(T1, ts); + p.add(c0, c0, T4); + region.put(p); + + // now flush/compact + region.flushcache(); + region.compactStores(true); + + // now put the first version (backdated) + p = new Put(T1, ts-3); + p.add(c0, c0, T1); + region.put(p); + + // now the latest change is in the memstore, + // but it is not the latest version + + Result r = region.get(new Get(T1), null); + checkResult(r, c0, T4); + + Get g = new Get(T1); + g.setMaxVersions(); + r = region.get(g, null); // this'll use ScanWildcardColumnTracker + checkResult(r, c0, T4,T3); + + g = new Get(T1); + g.setMaxVersions(); + g.addColumn(c0, c0); + r = region.get(g, null); // this'll use ExplicitColumnTracker + checkResult(r, c0, T4,T3); + + p = new Put(T1, ts+1); + p.add(c0, c0, T5); + region.put(p); + + // now the latest version is in the memstore + + g = new Get(T1); + g.setMaxVersions(); + r = region.get(g, null); // this'll use ScanWildcardColumnTracker + checkResult(r, c0, T5,T4); + + g = new Get(T1); + g.setMaxVersions(); + g.addColumn(c0, c0); + r = region.get(g, null); // this'll use ExplicitColumnTracker + checkResult(r, c0, T5,T4); + } finally { + region.close(); + region.getLog().closeAndDelete(); + } } /** @@ -269,83 +287,88 @@ public void testBaseCase() throws Exception { // 1 version minimum, 1000 versions maximum, ttl = 1s HTableDescriptor htd = createTableDescriptor(getName(), 2, 1000, 1, false); HRegion region = createNewHRegion(htd, null, null); - - // 2s in the past - long ts = EnvironmentEdgeManager.currentTimeMillis() - 2000; - - // 1st version - Put p = new Put(T1, ts-3); - p.add(c0, c0, T1); - region.put(p); - - // 2nd version - p = new Put(T1, ts-2); - p.add(c0, c0, T2); - region.put(p); - - // 3rd version - p = new Put(T1, ts-1); - p.add(c0, c0, T3); - region.put(p); - - // 4th version - p = new Put(T1, ts); - p.add(c0, c0, T4); - region.put(p); - - Result r = region.get(new Get(T1), null); - checkResult(r, c0, T4); - - Get g = new Get(T1); - g.setTimeRange(0L, ts+1); - r = region.get(g, null); - checkResult(r, c0, T4); - - // oldest version still exists - g.setTimeRange(0L, ts-2); - r = region.get(g, null); - checkResult(r, c0, T1); - - // gets see only available versions - // even before compactions - g = new Get(T1); - g.setMaxVersions(); - r = region.get(g, null); // this'll use ScanWildcardColumnTracker - checkResult(r, c0, T4,T3); - - g = new Get(T1); - g.setMaxVersions(); - g.addColumn(c0, c0); - r = region.get(g, null); // this'll use ExplicitColumnTracker - checkResult(r, c0, T4,T3); - - // now flush - region.flushcache(); - - // with HBASE-4241 a flush will eliminate the expired rows - g = new Get(T1); - g.setTimeRange(0L, ts-2); - r = region.get(g, null); - assertTrue(r.isEmpty()); - - // major compaction - region.compactStores(true); - - // after compaction the 4th version is still available - g = new Get(T1); - g.setTimeRange(0L, ts+1); - r = region.get(g, null); - checkResult(r, c0, T4); - - // so is the 3rd - g.setTimeRange(0L, ts); - r = region.get(g, null); - checkResult(r, c0, T3); - - // but the 2nd and earlier versions are gone - g.setTimeRange(0L, ts-1); - r = region.get(g, null); - assertTrue(r.isEmpty()); + try { + + // 2s in the past + long ts = EnvironmentEdgeManager.currentTimeMillis() - 2000; + + // 1st version + Put p = new Put(T1, ts-3); + p.add(c0, c0, T1); + region.put(p); + + // 2nd version + p = new Put(T1, ts-2); + p.add(c0, c0, T2); + region.put(p); + + // 3rd version + p = new Put(T1, ts-1); + p.add(c0, c0, T3); + region.put(p); + + // 4th version + p = new Put(T1, ts); + p.add(c0, c0, T4); + region.put(p); + + Result r = region.get(new Get(T1), null); + checkResult(r, c0, T4); + + Get g = new Get(T1); + g.setTimeRange(0L, ts+1); + r = region.get(g, null); + checkResult(r, c0, T4); + + // oldest version still exists + g.setTimeRange(0L, ts-2); + r = region.get(g, null); + checkResult(r, c0, T1); + + // gets see only available versions + // even before compactions + g = new Get(T1); + g.setMaxVersions(); + r = region.get(g, null); // this'll use ScanWildcardColumnTracker + checkResult(r, c0, T4,T3); + + g = new Get(T1); + g.setMaxVersions(); + g.addColumn(c0, c0); + r = region.get(g, null); // this'll use ExplicitColumnTracker + checkResult(r, c0, T4,T3); + + // now flush + region.flushcache(); + + // with HBASE-4241 a flush will eliminate the expired rows + g = new Get(T1); + g.setTimeRange(0L, ts-2); + r = region.get(g, null); + assertTrue(r.isEmpty()); + + // major compaction + region.compactStores(true); + + // after compaction the 4th version is still available + g = new Get(T1); + g.setTimeRange(0L, ts+1); + r = region.get(g, null); + checkResult(r, c0, T4); + + // so is the 3rd + g.setTimeRange(0L, ts); + r = region.get(g, null); + checkResult(r, c0, T3); + + // but the 2nd and earlier versions are gone + g.setTimeRange(0L, ts-1); + r = region.get(g, null); + assertTrue(r.isEmpty()); + } finally { + region.close(); + region.getLog().closeAndDelete(); + } } /** @@ -359,62 +382,67 @@ public void testFilters() throws Exception { // 2s in the past long ts = EnvironmentEdgeManager.currentTimeMillis() - 2000; - - Put p = new Put(T1, ts-3); - p.add(c0, c0, T0); - p.add(c1, c1, T0); - region.put(p); - - p = new Put(T1, ts-2); - p.add(c0, c0, T1); - p.add(c1, c1, T1); - region.put(p); - - p = new Put(T1, ts-1); - p.add(c0, c0, T2); - p.add(c1, c1, T2); - region.put(p); - - p = new Put(T1, ts); - p.add(c0, c0, T3); - p.add(c1, c1, T3); - region.put(p); - - List tss = new ArrayList(); - tss.add(ts-1); - tss.add(ts-2); - - Get g = new Get(T1); - g.addColumn(c1,c1); - g.setFilter(new TimestampsFilter(tss)); - g.setMaxVersions(); - Result r = region.get(g, null); - checkResult(r, c1, T2,T1); - - g = new Get(T1); - g.addColumn(c0,c0); - g.setFilter(new TimestampsFilter(tss)); - g.setMaxVersions(); - r = region.get(g, null); - checkResult(r, c0, T2,T1); - - // now flush/compact - region.flushcache(); - region.compactStores(true); - - g = new Get(T1); - g.addColumn(c1,c1); - g.setFilter(new TimestampsFilter(tss)); - g.setMaxVersions(); - r = region.get(g, null); - checkResult(r, c1, T2); - - g = new Get(T1); - g.addColumn(c0,c0); - g.setFilter(new TimestampsFilter(tss)); - g.setMaxVersions(); - r = region.get(g, null); - checkResult(r, c0, T2); + try { + + Put p = new Put(T1, ts-3); + p.add(c0, c0, T0); + p.add(c1, c1, T0); + region.put(p); + + p = new Put(T1, ts-2); + p.add(c0, c0, T1); + p.add(c1, c1, T1); + region.put(p); + + p = new Put(T1, ts-1); + p.add(c0, c0, T2); + p.add(c1, c1, T2); + region.put(p); + + p = new Put(T1, ts); + p.add(c0, c0, T3); + p.add(c1, c1, T3); + region.put(p); + + List tss = new ArrayList(); + tss.add(ts-1); + tss.add(ts-2); + + Get g = new Get(T1); + g.addColumn(c1,c1); + g.setFilter(new TimestampsFilter(tss)); + g.setMaxVersions(); + Result r = region.get(g, null); + checkResult(r, c1, T2,T1); + + g = new Get(T1); + g.addColumn(c0,c0); + g.setFilter(new TimestampsFilter(tss)); + g.setMaxVersions(); + r = region.get(g, null); + checkResult(r, c0, T2,T1); + + // now flush/compact + region.flushcache(); + region.compactStores(true); + + g = new Get(T1); + g.addColumn(c1,c1); + g.setFilter(new TimestampsFilter(tss)); + g.setMaxVersions(); + r = region.get(g, null); + checkResult(r, c1, T2); + + g = new Get(T1); + g.addColumn(c0,c0); + g.setFilter(new TimestampsFilter(tss)); + g.setMaxVersions(); + r = region.get(g, null); + checkResult(r, c0, T2); + } finally { + region.close(); + region.getLog().closeAndDelete(); + } } private void checkResult(Result r, byte[] col, byte[] ... vals) { diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMiniBatchOperationInProgress.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMiniBatchOperationInProgress.java new file mode 100644 index 000000000000..5ad95a0b21f9 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMiniBatchOperationInProgress.java @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.client.Mutation; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.regionserver.wal.WALEdit; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(SmallTests.class) +public class TestMiniBatchOperationInProgress { + + @Test + public void testMiniBatchOperationInProgressMethods() { + Pair[] operations = new Pair[10]; + OperationStatus[] retCodeDetails = new OperationStatus[10]; + WALEdit[] walEditsFromCoprocessors = new WALEdit[10]; + for (int i = 0; i < 10; i++) { + operations[i] = new Pair(new Put(Bytes.toBytes(i)), null); + } + MiniBatchOperationInProgress> miniBatch = + new MiniBatchOperationInProgress>(operations, retCodeDetails, + walEditsFromCoprocessors, 0, 5); + + assertEquals(5, miniBatch.size()); + assertTrue(Bytes.equals(Bytes.toBytes(0), miniBatch.getOperation(0).getFirst().getRow())); + assertTrue(Bytes.equals(Bytes.toBytes(2), miniBatch.getOperation(2).getFirst().getRow())); + assertTrue(Bytes.equals(Bytes.toBytes(4), miniBatch.getOperation(4).getFirst().getRow())); + try { + miniBatch.getOperation(5); + fail("Should throw Exception while accessing out of range"); + } catch (ArrayIndexOutOfBoundsException e) { + } + miniBatch.setOperationStatus(1, OperationStatus.FAILURE); + assertEquals(OperationStatus.FAILURE, retCodeDetails[1]); + try { + miniBatch.setOperationStatus(6, OperationStatus.FAILURE); + fail("Should throw Exception while accessing out of range"); + } catch (ArrayIndexOutOfBoundsException e) { + } + try { + miniBatch.setWalEdit(5, new WALEdit()); + fail("Should throw Exception while accessing out of range"); + } catch (ArrayIndexOutOfBoundsException e) { + } + + miniBatch = new MiniBatchOperationInProgress>(operations, + retCodeDetails, walEditsFromCoprocessors, 7, 10); + try { + miniBatch.setWalEdit(-1, new WALEdit()); + fail("Should throw Exception while accessing out of range"); + } catch (ArrayIndexOutOfBoundsException e) { + } + try { + miniBatch.getOperation(-1); + fail("Should throw Exception while accessing out of range"); + } catch (ArrayIndexOutOfBoundsException e) { + } + try { + miniBatch.getOperation(3); + fail("Should throw Exception while accessing out of range"); + } catch (ArrayIndexOutOfBoundsException e) { + } + try { + miniBatch.getOperationStatus(9); + fail("Should throw Exception while accessing out of range"); + } catch (ArrayIndexOutOfBoundsException e) { + } + try { + miniBatch.setOperationStatus(3, OperationStatus.FAILURE); + fail("Should throw Exception while accessing out of range"); + } catch (ArrayIndexOutOfBoundsException e) { + } + assertTrue(Bytes.equals(Bytes.toBytes(7), miniBatch.getOperation(0).getFirst().getRow())); + assertTrue(Bytes.equals(Bytes.toBytes(9), miniBatch.getOperation(2).getFirst().getRow())); + miniBatch.setOperationStatus(1, OperationStatus.SUCCESS); + assertEquals(OperationStatus.SUCCESS, retCodeDetails[8]); + WALEdit wal = new WALEdit(); + miniBatch.setWalEdit(0, wal); + assertEquals(wal, walEditsFromCoprocessors[7]); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java index b716c53c0eee..2feb20a3a322 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestReadWriteConsistencyControl.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConsistencyControl.java similarity index 85% rename from src/test/java/org/apache/hadoop/hbase/regionserver/TestReadWriteConsistencyControl.java rename to src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConsistencyControl.java index fd4583360daf..7908b3001e13 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestReadWriteConsistencyControl.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConsistencyControl.java @@ -1,6 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation - * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information @@ -27,8 +25,12 @@ import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; +/** + * This is a hammer test that verifies MultiVersionConsistencyControl in a + * multiple writer single reader scenario. + */ @Category(SmallTests.class) -public class TestReadWriteConsistencyControl extends TestCase { +public class TestMultiVersionConsistencyControl extends TestCase { static class Writer implements Runnable { final AtomicBoolean finished; final MultiVersionConsistencyControl mvcc; @@ -39,20 +41,20 @@ static class Writer implements Runnable { this.mvcc = mvcc; this.status = status; } + private Random rnd = new Random(); public boolean failed = false; public void run() { while (!finished.get()) { MultiVersionConsistencyControl.WriteEntry e = mvcc.beginMemstoreInsert(); -// System.out.println("Begin write: " + e.getWriteNumber()); + // System.out.println("Begin write: " + e.getWriteNumber()); // 10 usec - 500usec (including 0) int sleepTime = rnd.nextInt(500); // 500 * 1000 = 500,000ns = 500 usec // 1 * 100 = 100ns = 1usec try { - if (sleepTime > 0) - Thread.sleep(0, sleepTime * 1000); + if (sleepTime > 0) Thread.sleep(0, sleepTime * 1000); } catch (InterruptedException e1) { } try { @@ -84,8 +86,7 @@ public void run() { long newPrev = mvcc.memstoreReadPoint(); if (newPrev < prev) { // serious problem. - System.out.println("Reader got out of order, prev: " + - prev + " next was: " + newPrev); + System.out.println("Reader got out of order, prev: " + prev + " next was: " + newPrev); readerFailed.set(true); // might as well give up failedAt.set(newPrev); @@ -97,11 +98,11 @@ public void run() { // writer thread parallelism. int n = 20; - Thread [] writers = new Thread[n]; - AtomicBoolean [] statuses = new AtomicBoolean[n]; + Thread[] writers = new Thread[n]; + AtomicBoolean[] statuses = new AtomicBoolean[n]; Thread readThread = new Thread(reader); - for (int i = 0 ; i < n ; ++i ) { + for (int i = 0; i < n; ++i) { statuses[i] = new AtomicBoolean(true); writers[i] = new Thread(new Writer(finished, mvcc, statuses[i])); writers[i].start(); @@ -126,11 +127,8 @@ public void run() { assertTrue(statuses[i].get()); } - } @org.junit.Rule - public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = - new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); } - diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java index 79db715467e4..206ca46a275c 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -83,22 +82,11 @@ public void setUp() throws Exception { } - public void testMatch_ExplicitColumns() - throws IOException { + private void _testMatch_ExplicitColumns(Scan scan, List expected) throws IOException { //Moving up from the Tracker by using Gets and List instead //of just byte [] - //Expected result - List expected = new ArrayList(); - expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_COL); - expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_COL); - expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_COL); - expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_COL); - expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_ROW); - expected.add(ScanQueryMatcher.MatchCode.DONE); - - // 2,4,5 - + // 2,4,5 ScanQueryMatcher qm = new ScanQueryMatcher(scan, new Store.ScanInfo(fam2, 0, 1, ttl, false, 0, rowComparator), get.getFamilyMap().get(fam2), EnvironmentEdgeManager.currentTimeMillis() - ttl); @@ -113,7 +101,8 @@ public void testMatch_ExplicitColumns() memstore.add(new KeyValue(row2, fam1, col1, data)); List actual = new ArrayList(); - qm.setRow(memstore.get(0).getRow()); + KeyValue k = memstore.get(0); + qm.setRow(k.getBuffer(), k.getRowOffset(), k.getRowLength()); for (KeyValue kv : memstore){ actual.add(qm.match(kv)); @@ -129,6 +118,42 @@ public void testMatch_ExplicitColumns() } } + public void testMatch_ExplicitColumns() + throws IOException { + //Moving up from the Tracker by using Gets and List instead + //of just byte [] + + //Expected result + List expected = new ArrayList(); + expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_COL); + expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_COL); + expected.add(ScanQueryMatcher.MatchCode.SEEK_NEXT_COL); + expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_COL); + expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_ROW); + expected.add(ScanQueryMatcher.MatchCode.DONE); + + _testMatch_ExplicitColumns(scan, expected); + } + + public void testMatch_ExplicitColumnsWithLookAhead() + throws IOException { + //Moving up from the Tracker by using Gets and List instead + //of just byte [] + + //Expected result + List expected = new ArrayList(); + expected.add(ScanQueryMatcher.MatchCode.SKIP); + expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_COL); + expected.add(ScanQueryMatcher.MatchCode.SKIP); + expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_COL); + expected.add(ScanQueryMatcher.MatchCode.INCLUDE_AND_SEEK_NEXT_ROW); + expected.add(ScanQueryMatcher.MatchCode.DONE); + + Scan s = new Scan(scan); + s.setAttribute(Scan.HINT_LOOKAHEAD, Bytes.toBytes(2)); + _testMatch_ExplicitColumns(s, expected); + } + public void testMatch_Wildcard() throws IOException { @@ -158,7 +183,8 @@ public void testMatch_Wildcard() List actual = new ArrayList(); - qm.setRow(memstore.get(0).getRow()); + KeyValue k = memstore.get(0); + qm.setRow(k.getBuffer(), k.getRowOffset(), k.getRowLength()); for(KeyValue kv : memstore) { actual.add(qm.match(kv)); @@ -210,7 +236,8 @@ public void testMatch_ExpiredExplicit() new KeyValue(row2, fam1, col1, now-10, data) }; - qm.setRow(kvs[0].getRow()); + KeyValue k = kvs[0]; + qm.setRow(k.getBuffer(), k.getRowOffset(), k.getRowLength()); List actual = new ArrayList(kvs.length); for (KeyValue kv : kvs) { @@ -262,7 +289,8 @@ public void testMatch_ExpiredWildcard() new KeyValue(row1, fam2, col5, now-10000, data), new KeyValue(row2, fam1, col1, now-10, data) }; - qm.setRow(kvs[0].getRow()); + KeyValue k = kvs[0]; + qm.setRow(k.getBuffer(), k.getRowOffset(), k.getRowLength()); List actual = new ArrayList(kvs.length); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java new file mode 100644 index 000000000000..1689238a056d --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenMasterInitializing.java @@ -0,0 +1,315 @@ +/* + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.master.AssignmentManager; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.master.ServerManager; +import org.apache.hadoop.hbase.master.TestMasterFailover; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread; +import org.apache.hadoop.hbase.zookeeper.ZKAssign; +import org.apache.hadoop.hbase.zookeeper.ZKTable; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.apache.zookeeper.KeeperException; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(LargeTests.class) +public class TestRSKilledWhenMasterInitializing { + private static final Log LOG = LogFactory.getLog(TestMasterFailover.class); + + private static final HBaseTestingUtility TESTUTIL = new HBaseTestingUtility(); + private static final int NUM_MASTERS = 1; + private static final int NUM_RS = 5; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + // Set it so that this test runs with my custom master + Configuration conf = TESTUTIL.getConfiguration(); + conf.setClass(HConstants.MASTER_IMPL, TestingMaster.class, HMaster.class); + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, 3); + conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MAXTOSTART, 5); + + // Start up the cluster. + TESTUTIL.startMiniCluster(NUM_MASTERS, NUM_RS); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + if (!TESTUTIL.getHBaseCluster().getMaster().isInitialized()) { + // master is not initialized and is waiting something forever. + for (MasterThread mt : TESTUTIL.getHBaseCluster().getLiveMasterThreads()) { + mt.interrupt(); + } + } + TESTUTIL.shutdownMiniCluster(); + } + + /** + * An HMaster instance used in this test. If 'TestingMaster.sleep' is set in + * the Configuration, then we'll sleep after log is split and we'll also + * return a custom RegionServerTracker. + */ + public static class TestingMaster extends HMaster { + private boolean logSplit = false; + + public TestingMaster(Configuration conf) throws IOException, + KeeperException, InterruptedException { + super(conf); + } + + public boolean isLogSplitAfterStartup() { + return logSplit; + } + } + + @Test(timeout = 180000) + public void testCorrectnessWhenMasterFailOver() throws Exception { + final byte[] TABLENAME = Bytes.toBytes("testCorrectnessWhenMasterFailOver"); + final byte[] FAMILY = Bytes.toBytes("family"); + final byte[][] SPLITKEYS = { Bytes.toBytes("b"), Bytes.toBytes("i") }; + + MiniHBaseCluster cluster = TESTUTIL.getHBaseCluster(); + + HTableDescriptor desc = new HTableDescriptor(TABLENAME); + desc.addFamily(new HColumnDescriptor(FAMILY)); + HBaseAdmin hbaseAdmin = TESTUTIL.getHBaseAdmin(); + hbaseAdmin.createTable(desc, SPLITKEYS); + + assertTrue(hbaseAdmin.isTableAvailable(TABLENAME)); + + HTable table = new HTable(TESTUTIL.getConfiguration(), TABLENAME); + List puts = new ArrayList(); + Put put1 = new Put(Bytes.toBytes("a")); + put1.add(FAMILY, Bytes.toBytes("q1"), Bytes.toBytes("value")); + Put put2 = new Put(Bytes.toBytes("h")); + put2.add(FAMILY, Bytes.toBytes("q1"), Bytes.toBytes("value")); + Put put3 = new Put(Bytes.toBytes("o")); + put3.add(FAMILY, Bytes.toBytes("q1"), Bytes.toBytes("value")); + puts.add(put1); + puts.add(put2); + puts.add(put3); + table.put(puts); + ResultScanner resultScanner = table.getScanner(new Scan()); + int count = 0; + while (resultScanner.next() != null) { + count++; + } + resultScanner.close(); + table.close(); + assertEquals(3, count); + + /* Starting test */ + cluster.getConfiguration().setBoolean("TestingMaster.sleep", true); + cluster.getConfiguration().setInt("TestingMaster.sleep.duration", 10000); + + /* NO.1 .META. region correctness */ + // First abort master + abortMaster(cluster); + TestingMaster master = startMasterAndWaitTillMetaRegionAssignment(cluster); + + // Second kill meta server + int metaServerNum = cluster.getServerWithMeta(); + int rootServerNum = cluster.getServerWith(HRegionInfo.ROOT_REGIONINFO + .getRegionName()); + HRegionServer metaRS = cluster.getRegionServer(metaServerNum); + LOG.debug("Killing metaRS and carryingRoot = " + + (metaServerNum == rootServerNum)); + metaRS.kill(); + metaRS.join(); + + /* + * Sleep double time of TestingMaster.sleep.duration, so we can ensure that + * master has already assigned ROOTandMETA or is blocking on assigning + * ROOTandMETA + */ + Thread.sleep(10000 * 2); + + waitUntilMasterIsInitialized(master); + + // Third check whether data is correct in meta region + assertTrue(hbaseAdmin.isTableAvailable(TABLENAME)); + + /* + * NO.2 -ROOT- region correctness . If the .META. server killed in the NO.1 + * is also carrying -ROOT- region, it is not needed + */ + if (rootServerNum != metaServerNum) { + // First abort master + abortMaster(cluster); + master = startMasterAndWaitTillMetaRegionAssignment(cluster); + + // Second kill meta server + HRegionServer rootRS = cluster.getRegionServer(rootServerNum); + LOG.debug("Killing rootRS"); + rootRS.kill(); + rootRS.join(); + + /* + * Sleep double time of TestingMaster.sleep.duration, so we can ensure + * that master has already assigned ROOTandMETA or is blocking on + * assigning ROOTandMETA + */ + Thread.sleep(10000 * 2); + waitUntilMasterIsInitialized(master); + + // Third check whether data is correct in meta region + assertTrue(hbaseAdmin.isTableAvailable(TABLENAME)); + } + + + /* NO.3 data region correctness */ + ServerManager serverManager = cluster.getMaster().getServerManager(); + while (serverManager.areDeadServersInProgress()) { + Thread.sleep(100); + } + // Create a ZKW to use in the test + ZooKeeperWatcher zkw = HBaseTestingUtility.getZooKeeperWatcher(TESTUTIL); + ZKAssign.blockUntilNoRIT(zkw); + + table = new HTable(TESTUTIL.getConfiguration(), TABLENAME); + resultScanner = table.getScanner(new Scan()); + count = 0; + while (resultScanner.next() != null) { + count++; + } + resultScanner.close(); + table.close(); + assertEquals(3, count); + } + + @Test (timeout=180000) + public void testMasterFailoverWhenDisablingTableRegionsInRITOnDeadRS() throws Exception { + MiniHBaseCluster cluster = TESTUTIL.getHBaseCluster(); + HMaster master = cluster.getMaster(); + // disable load balancing on this master + master.balanceSwitch(false); + + final String table = "testMasterFailoverWhenDisablingTableRegionsInRITOnDeadRS"; + byte [] FAMILY = Bytes.toBytes("family"); + byte[][] SPLIT_KEYS = + new byte[][] {Bytes.toBytes("a"), Bytes.toBytes("b"), Bytes.toBytes("c"), + Bytes.toBytes("d") }; + HTableDescriptor htd = new HTableDescriptor(table); + HColumnDescriptor hcd = new HColumnDescriptor(FAMILY); + htd.addFamily(hcd); + TESTUTIL.getHBaseAdmin().createTable(htd, SPLIT_KEYS); + AssignmentManager am = cluster.getMaster().getAssignmentManager(); + List regionsOfTable = null; + while ((regionsOfTable = am.getRegionsOfTable(table.getBytes())).size() + != (SPLIT_KEYS.length + 1)) { + Thread.sleep(10); + } + HRegionInfo closingRegion = regionsOfTable.get(0); + ServerName serverName = am.getRegionServerOfRegion(closingRegion); + HRegionServer deadRS = null; + for (int i = 0; i < cluster.getRegionServerThreads().size(); i++) { + deadRS = cluster.getRegionServer(i); + if (deadRS.getServerName().equals(serverName)) { + break; + } + } + + // Disable the table in ZK + ZKTable zkTable = am.getZKTable(); + zkTable.setDisablingTable(table); + ZKAssign.createNodeClosing(master.getZooKeeper(), closingRegion, serverName); + + // Stop the master + abortMaster(cluster); + master = startMasterAndWaitTillMetaRegionAssignment(cluster); + deadRS.kill(); + deadRS.join(); + waitUntilMasterIsInitialized(master); + am = cluster.getMaster().getAssignmentManager(); + zkTable = am.getZKTable(); + // wait for no more RIT + ZKAssign.blockUntilNoRIT(master.getZooKeeper()); + while (!master.getAssignmentManager().getZKTable().isDisabledTable(table)) { + Thread.sleep(10); + } + assertTrue("Table should be disabled state.", zkTable.isDisabledTable(table)); + HBaseAdmin admin = new HBaseAdmin(master.getConfiguration()); + admin.deleteTable(table); + } + + private void abortMaster(MiniHBaseCluster cluster) + throws InterruptedException { + for (MasterThread mt : cluster.getLiveMasterThreads()) { + if (mt.getMaster().isActiveMaster()) { + mt.getMaster().abort("Aborting for tests", new Exception("Trace info")); + mt.join(); + break; + } + } + LOG.debug("Master is aborted"); + } + + private TestingMaster startMasterAndWaitTillMetaRegionAssignment(MiniHBaseCluster cluster) + throws IOException, InterruptedException { + TestingMaster master = (TestingMaster) cluster.startMaster().getMaster(); + while (!master.isInitializationStartsMetaRegoinAssignment()) { + Thread.sleep(100); + } + return master; + } + + private void waitUntilMasterIsInitialized(HMaster master) + throws InterruptedException { + while (!master.isInitialized()) { + Thread.sleep(100); + } + while (master.getServerManager().areDeadServersInProgress()) { + Thread.sleep(100); + } + LOG.debug("master isInitialized"); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); + +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java index 84782601f202..0d1dbf8a022f 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java index 6560672abb0d..9b82cf655ceb 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java @@ -23,20 +23,34 @@ import java.io.IOException; import java.util.Arrays; import java.util.Map; +import java.util.Set; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.Append; +import org.apache.hadoop.hbase.client.Delete; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Increment; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.regionserver.metrics.RegionMetricsStorage; import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics. StoreMetricType; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; import org.junit.After; import org.junit.Before; import org.junit.Test; import org.junit.experimental.categories.Category; + /** * Test metrics incremented on region server operations. */ @@ -57,7 +71,7 @@ public class TestRegionServerMetrics { private static final SchemaMetrics ALL_METRICS = SchemaMetrics.ALL_SCHEMA_METRICS; - private static final HBaseTestingUtility TEST_UTIL = + private final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); private Map startingMetrics; @@ -77,6 +91,29 @@ public void tearDown() throws Exception { SchemaMetrics.validateMetricChanges(startingMetrics); } + private void assertTimeVaryingMetricCount(int expectedCount, String table, String cf, + String regionName, String metricPrefix) { + + Integer expectedCountInteger = new Integer(expectedCount); + + if (cf != null) { + String cfKey = + SchemaMetrics.TABLE_PREFIX + table + "." + SchemaMetrics.CF_PREFIX + cf + "." + + metricPrefix; + Pair cfPair = RegionMetricsStorage.getTimeVaryingMetric(cfKey); + assertEquals(expectedCountInteger, cfPair.getSecond()); + } + + if (regionName != null) { + String rKey = + SchemaMetrics.TABLE_PREFIX + table + "." + SchemaMetrics.REGION_PREFIX + regionName + "." + + metricPrefix; + + Pair regionPair = RegionMetricsStorage.getTimeVaryingMetric(rKey); + assertEquals(expectedCountInteger, regionPair.getSecond()); + } + } + private void assertStoreMetricEquals(long expected, SchemaMetrics schemaMetrics, StoreMetricType storeMetricType) { final String storeMetricName = @@ -84,10 +121,117 @@ private void assertStoreMetricEquals(long expected, Long startValue = startingMetrics.get(storeMetricName); assertEquals("Invalid value for store metric " + storeMetricName + " (type " + storeMetricType + ")", expected, - HRegion.getNumericMetric(storeMetricName) + RegionMetricsStorage.getNumericMetric(storeMetricName) - (startValue != null ? startValue : 0)); } + + @Test + public void testOperationMetrics() throws IOException { + String cf = "OPCF"; + String otherCf = "otherCF"; + String rk = "testRK"; + String icvCol = "icvCol"; + String appendCol = "appendCol"; + String regionName = null; + HTable hTable = + TEST_UTIL.createTable(TABLE_NAME.getBytes(), + new byte[][] { cf.getBytes(), otherCf.getBytes() }); + Set regionInfos = hTable.getRegionLocations().keySet(); + + regionName = regionInfos.toArray(new HRegionInfo[regionInfos.size()])[0].getEncodedName(); + + //Do a multi put that has one cf. Since they are in different rk's + //The lock will still be obtained and everything will be applied in one multiput. + Put pOne = new Put(rk.getBytes()); + pOne.add(cf.getBytes(), icvCol.getBytes(), Bytes.toBytes(0L)); + Put pTwo = new Put("ignored1RK".getBytes()); + pTwo.add(cf.getBytes(), "ignored".getBytes(), Bytes.toBytes(0L)); + + hTable.put(Arrays.asList(new Put[] {pOne, pTwo})); + + // Do a multiput where the cf doesn't stay consistent. + Put pThree = new Put("ignored2RK".getBytes()); + pThree.add(cf.getBytes(), "ignored".getBytes(), Bytes.toBytes("TEST1")); + Put pFour = new Put("ignored3RK".getBytes()); + pFour.add(otherCf.getBytes(), "ignored".getBytes(), Bytes.toBytes(0L)); + + hTable.put(Arrays.asList(new Put[] { pThree, pFour })); + + hTable.incrementColumnValue(rk.getBytes(), cf.getBytes(), icvCol.getBytes(), 1L); + + Increment i = new Increment(rk.getBytes()); + i.addColumn(cf.getBytes(), icvCol.getBytes(), 1L); + hTable.increment(i); + + Get g = new Get(rk.getBytes()); + g.addColumn(cf.getBytes(), appendCol.getBytes()); + hTable.get(g); + + Append a = new Append(rk.getBytes()); + a.add(cf.getBytes(), appendCol.getBytes(), Bytes.toBytes("-APPEND")); + hTable.append(a); + + Delete dOne = new Delete(rk.getBytes()); + dOne.deleteFamily(cf.getBytes()); + hTable.delete(dOne); + + Delete dTwo = new Delete(rk.getBytes()); + hTable.delete(dTwo); + + // There should be one multi put where the cf is consistent + assertTimeVaryingMetricCount(1, TABLE_NAME, cf, null, "multiput_"); + + // There were two multiputs to the cf. + assertTimeVaryingMetricCount(2, TABLE_NAME, null, regionName, "multiput_"); + + // There was one multiput where the cf was not consistent. + assertTimeVaryingMetricCount(1, TABLE_NAME, "__unknown", null, "multiput_"); + + // One increment and one append + assertTimeVaryingMetricCount(1, TABLE_NAME, cf, regionName, "incrementColumnValue_"); + assertTimeVaryingMetricCount(1, TABLE_NAME, cf, regionName, "increment_"); + assertTimeVaryingMetricCount(1, TABLE_NAME, cf, regionName, "append_"); + + // One delete where the cf is known + assertTimeVaryingMetricCount(1, TABLE_NAME, cf, null, "delete_"); + + // two deletes in the region. + assertTimeVaryingMetricCount(2, TABLE_NAME, null, regionName, "delete_"); + + // Three gets. one for gets. One for append. One for increment. + assertTimeVaryingMetricCount(4, TABLE_NAME, cf, regionName, "get_"); + + hTable.close(); + } + + @Test + public void testRemoveRegionMetrics() throws IOException, InterruptedException { + String cf = "REMOVECF"; + HTable hTable = TEST_UTIL.createTable(TABLE_NAME.getBytes(), cf.getBytes()); + HRegionInfo[] regionInfos = + hTable.getRegionLocations().keySet() + .toArray(new HRegionInfo[hTable.getRegionLocations().keySet().size()]); + + String regionName = regionInfos[0].getEncodedName(); + + // Do some operations so there are metrics. + Put pOne = new Put("TEST".getBytes()); + pOne.add(cf.getBytes(), "test".getBytes(), "test".getBytes()); + hTable.put(pOne); + + Get g = new Get("TEST".getBytes()); + g.addFamily(cf.getBytes()); + hTable.get(g); + assertTimeVaryingMetricCount(1, TABLE_NAME, cf, regionName, "get_"); + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + admin.disableTable(TABLE_NAME.getBytes()); + admin.deleteTable(TABLE_NAME.getBytes()); + + assertTimeVaryingMetricCount(0, TABLE_NAME, cf, regionName, "get_"); + hTable.close(); + } + @Test public void testMultipleRegions() throws IOException, InterruptedException { @@ -124,12 +268,89 @@ public void testMultipleRegions() throws IOException, InterruptedException { final String storeMetricName = ALL_METRICS .getStoreMetricNameMax(StoreMetricType.STORE_FILE_COUNT); assertEquals("Invalid value for store metric " + storeMetricName, - NUM_FLUSHES, HRegion.getNumericMetric(storeMetricName)); + NUM_FLUSHES, RegionMetricsStorage.getNumericMetric(storeMetricName)); } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); + + private void assertSizeMetric(String table, String[] cfs, int[] metrics) { + // we have getsize & nextsize for each column family + assertEquals(cfs.length * 2, metrics.length); + + for (int i =0; i < cfs.length; ++i) { + String prefix = SchemaMetrics.generateSchemaMetricsPrefix(table, cfs[i]); + String getMetric = prefix + SchemaMetrics.METRIC_GETSIZE; + String nextMetric = prefix + SchemaMetrics.METRIC_NEXTSIZE; + + // verify getsize and nextsize matches + int getSize = RegionMetricsStorage.getNumericMetrics().containsKey(getMetric) ? + RegionMetricsStorage.getNumericMetrics().get(getMetric).intValue() : 0; + int nextSize = RegionMetricsStorage.getNumericMetrics().containsKey(nextMetric) ? + RegionMetricsStorage.getNumericMetrics().get(nextMetric).intValue() : 0; + + assertEquals(metrics[i], getSize); + assertEquals(metrics[cfs.length + i], nextSize); + } + } + + @Test + public void testGetNextSize() throws IOException, InterruptedException { + String rowName = "row1"; + byte[] ROW = Bytes.toBytes(rowName); + String tableName = "SizeMetricTest"; + byte[] TABLE = Bytes.toBytes(tableName); + String cf1Name = "cf1"; + String cf2Name = "cf2"; + String[] cfs = new String[] {cf1Name, cf2Name}; + byte[] CF1 = Bytes.toBytes(cf1Name); + byte[] CF2 = Bytes.toBytes(cf2Name); + + long ts = 1234; + HTable hTable = TEST_UTIL.createTable(TABLE, new byte[][]{CF1, CF2}); + HBaseAdmin admin = new HBaseAdmin(TEST_UTIL.getConfiguration()); + + Put p = new Put(ROW); + p.add(CF1, CF1, ts, CF1); + p.add(CF2, CF2, ts, CF2); + hTable.put(p); + + KeyValue kv1 = new KeyValue(ROW, CF1, CF1, ts, CF1); + KeyValue kv2 = new KeyValue(ROW, CF2, CF2, ts, CF2); + int kvLength = kv1.getLength(); + assertEquals(kvLength, kv2.getLength()); + + // only cf1.getsize is set on Get + hTable.get(new Get(ROW).addFamily(CF1)); + assertSizeMetric(tableName, cfs, new int[] {kvLength, 0, 0, 0}); + + // only cf2.getsize is set on Get + hTable.get(new Get(ROW).addFamily(CF2)); + assertSizeMetric(tableName, cfs, new int[] {kvLength, kvLength, 0, 0}); + + // only cf2.nextsize is set + for (Result res : hTable.getScanner(CF2)) { + } + assertSizeMetric(tableName, cfs, + new int[] {kvLength, kvLength, 0, kvLength}); + + // only cf2.nextsize is set + for (Result res : hTable.getScanner(CF1)) { + } + assertSizeMetric(tableName, cfs, + new int[] {kvLength, kvLength, kvLength, kvLength}); + + // getsize/nextsize should not be set on flush or compaction + for (HRegion hr : TEST_UTIL.getMiniHBaseCluster().getRegions(TABLE)) { + hr.flushcache(); + hr.compactStores(); + } + assertSizeMetric(tableName, cfs, + new int[] {kvLength, kvLength, kvLength, kvLength}); + + hTable.close(); + } } diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java index 300155b20a26..4f170347436c 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java @@ -75,9 +75,9 @@ public void testIncreasingToUpperBoundRegionSplitPolicy() throws IOException { // Set max size for this 'table'. long maxSplitSize = 1024L; htd.setMaxFileSize(maxSplitSize); - // Set flush size to 1/4. IncreasingToUpperBoundRegionSplitPolicy - // grows by the square of the number of regions times flushsize each time. - long flushSize = maxSplitSize/4; + // Set flush size to 1/8. IncreasingToUpperBoundRegionSplitPolicy + // grows by the cube of the number of regions times flushsize each time. + long flushSize = maxSplitSize/8; conf.setLong(HConstants.HREGION_MEMSTORE_FLUSH_SIZE, flushSize); htd.setMemStoreFlushSize(flushSize); // If RegionServerService with no regions in it -- 'online regions' == 0 -- @@ -100,18 +100,18 @@ public void testIncreasingToUpperBoundRegionSplitPolicy() throws IOException { // Now test that we increase our split size as online regions for a table // grows. With one region, split size should be flushsize. regions.add(mockRegion); - Mockito.doReturn(flushSize/2).when(mockStore).getSize(); - // Should not split since store is 1/2 flush size. + Mockito.doReturn(flushSize).when(mockStore).getSize(); + // Should not split since store is flush size. assertFalse(policy.shouldSplit()); - // Set size of store to be > flush size and we should split - Mockito.doReturn(flushSize + 1).when(mockStore).getSize(); + // Set size of store to be > 2*flush size and we should split + Mockito.doReturn(flushSize*2 + 1).when(mockStore).getSize(); assertTrue(policy.shouldSplit()); // Add another region to the 'online regions' on this server and we should // now be no longer be splittable since split size has gone up. regions.add(mockRegion); assertFalse(policy.shouldSplit()); // Quadruple (2 squared) the store size and make sure its just over; verify it'll split - Mockito.doReturn((flushSize * 2 * 2) + 1).when(mockStore).getSize(); + Mockito.doReturn((flushSize * 2 * 2 * 2) + 1).when(mockStore).getSize(); assertTrue(policy.shouldSplit()); // Finally assert that even if loads of regions, we'll split at max size @@ -250,7 +250,43 @@ public void testGetSplitPoint() throws IOException { Bytes.toString(policy.getSplitPoint())); } + @Test + public void testDelimitedKeyPrefixRegionSplitPolicy() throws IOException { + HTableDescriptor myHtd = new HTableDescriptor(); + myHtd.setValue(HTableDescriptor.SPLIT_POLICY, + DelimitedKeyPrefixRegionSplitPolicy.class.getName()); + myHtd.setValue(DelimitedKeyPrefixRegionSplitPolicy.DELIMITER_KEY, ","); + + HRegion myMockRegion = Mockito.mock(HRegion.class); + Mockito.doReturn(myHtd).when(myMockRegion).getTableDesc(); + Mockito.doReturn(stores).when(myMockRegion).getStores(); + + Store mockStore = Mockito.mock(Store.class); + Mockito.doReturn(2000L).when(mockStore).getSize(); + Mockito.doReturn(true).when(mockStore).canSplit(); + Mockito.doReturn(Bytes.toBytes("ab,cd")).when(mockStore).getSplitPoint(); + stores.put(new byte[] { 1 }, mockStore); + + DelimitedKeyPrefixRegionSplitPolicy policy = (DelimitedKeyPrefixRegionSplitPolicy) RegionSplitPolicy + .create(myMockRegion, conf); + + assertEquals("ab", Bytes.toString(policy.getSplitPoint())); + + Mockito.doReturn(true).when(myMockRegion).shouldForceSplit(); + Mockito.doReturn(Bytes.toBytes("efg,h")).when(myMockRegion) + .getExplicitSplitPoint(); + + policy = (DelimitedKeyPrefixRegionSplitPolicy) RegionSplitPolicy + .create(myMockRegion, conf); + + assertEquals("efg", Bytes.toString(policy.getSplitPoint())); + + Mockito.doReturn(Bytes.toBytes("ijk")).when(myMockRegion) + .getExplicitSplitPoint(); + assertEquals("ijk", Bytes.toString(policy.getSplitPoint())); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); -} \ No newline at end of file +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java index c98efe50eec1..f51670e43074 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -68,31 +67,34 @@ public void testResettingCounters() throws Exception { } } HRegion region = HRegion.createHRegion(hri, path, conf, htd); + try { + Increment odd = new Increment(rows[0]); + Increment even = new Increment(rows[0]); + Increment all = new Increment(rows[0]); + for (int i=0;i 60000) { + assertTrue("Metric should have set incremented for "+metricName, + wasSet(metricName + "_num_ops")); + } + Thread.sleep(200); + } } public boolean wasSet(String name) { diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java index 13ed753a43a4..a7045e6c7267 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -110,6 +109,19 @@ public void testDelete_KeepDelete(){ assertEquals(false ,sdt.isEmpty()); } + public void testDelete_KeepVersionZero(){ + byte [] qualifier = Bytes.toBytes("qualifier"); + deleteType = KeyValue.Type.Delete.getCode(); + + long deleteTimestamp = 10; + long valueTimestamp = 0; + + sdt.reset(); + sdt.add(qualifier, 0, qualifier.length, deleteTimestamp, deleteType); + DeleteResult ret = sdt.isDeleted(qualifier, 0, qualifier.length, valueTimestamp); + assertEquals(DeleteResult.NOT_DELETED, ret); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java index 086a4082ff0b..7188471a43cb 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -55,7 +54,7 @@ public void testCheckColumn_Ok() throws IOException { List actual = new ArrayList(); for(byte [] qualifier : qualifiers) { - ScanQueryMatcher.MatchCode mc = tracker.checkColumn(qualifier, 0, + ScanQueryMatcher.MatchCode mc = ScanQueryMatcher.checkColumn(tracker, qualifier, 0, qualifier.length, 1, KeyValue.Type.Put.getCode(), false); actual.add(mc); } @@ -88,7 +87,7 @@ public void testCheckColumn_EnforceVersions() throws IOException { long timestamp = 0; for(byte [] qualifier : qualifiers) { - MatchCode mc = tracker.checkColumn(qualifier, 0, qualifier.length, + MatchCode mc = ScanQueryMatcher.checkColumn(tracker, qualifier, 0, qualifier.length, ++timestamp, KeyValue.Type.Put.getCode(), false); actual.add(mc); } @@ -112,7 +111,7 @@ public void DisabledTestCheckColumn_WrongOrder() { try { for(byte [] qualifier : qualifiers) { - tracker.checkColumn(qualifier, 0, qualifier.length, 1, + ScanQueryMatcher.checkColumn(tracker, qualifier, 0, qualifier.length, 1, KeyValue.Type.Put.getCode(), false); } } catch (Exception e) { diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java index bdb1231d97a8..2779302c9628 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java index 32e8d189fcff..eb26b9d72810 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java index 2a092e7f8833..104f99b1fb1a 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -142,6 +141,8 @@ public void setUp() { @Test public void testMultipleTimestampRanges() throws IOException { + // enable seek counting + StoreFileScanner.instrument(); region = TEST_UTIL.createTestRegion(TestSeekOptimizations.class.getName(), new HColumnDescriptor(FAMILY) .setCompressionType(comprAlgo) diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java index e99d2514a4d3..9af7d76b4bb1 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java index 26b9865fd0be..e20ef8171467 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -56,6 +55,7 @@ public class TestSplitLogWorker { static { Logger.getLogger("org.apache.hadoop.hbase").setLevel(Level.DEBUG); } + private static final int WAIT_TIME = 15000; private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); private ZooKeeperWatcher zkw; @@ -126,7 +126,7 @@ public Status exec(String name, CancelableProgressable p) { }; - @Test + @Test(timeout=60000) public void testAcquireTaskAtStartup() throws Exception { LOG.info("testAcquireTaskAtStartup"); ZKSplitLog.Counters.resetCounters(); @@ -139,7 +139,7 @@ public void testAcquireTaskAtStartup() throws Exception { "rs", neverEndingTask); slw.start(); try { - waitForCounter(tot_wkr_task_acquired, 0, 1, 100); + waitForCounter(tot_wkr_task_acquired, 0, 1, WAIT_TIME); assertTrue(TaskState.TASK_OWNED.equals(ZKUtil.getData(zkw, ZKSplitLog.getEncodedNodeName(zkw, "tatas")), "rs")); } finally { @@ -151,14 +151,14 @@ private void stopSplitLogWorker(final SplitLogWorker slw) throws InterruptedException { if (slw != null) { slw.stop(); - slw.worker.join(3000); + slw.worker.join(WAIT_TIME); if (slw.worker.isAlive()) { assertTrue(("Could not stop the worker thread slw=" + slw) == null); } } } - @Test + @Test(timeout=60000) public void testRaceForTask() throws Exception { LOG.info("testRaceForTask"); ZKSplitLog.Counters.resetCounters(); @@ -174,10 +174,10 @@ public void testRaceForTask() throws Exception { slw1.start(); slw2.start(); try { - waitForCounter(tot_wkr_task_acquired, 0, 1, 1000); + waitForCounter(tot_wkr_task_acquired, 0, 1, WAIT_TIME); // Assert that either the tot_wkr_failed_to_grab_task_owned count was set of if // not it, that we fell through to the next counter in line and it was set. - assertTrue(waitForCounterBoolean(tot_wkr_failed_to_grab_task_owned, 0, 1, 1000) || + assertTrue(waitForCounterBoolean(tot_wkr_failed_to_grab_task_owned, 0, 1, WAIT_TIME) || tot_wkr_failed_to_grab_task_lost_race.get() == 1); assertTrue(TaskState.TASK_OWNED.equals(ZKUtil.getData(zkw, ZKSplitLog.getEncodedNodeName(zkw, "trft")), "svr1") || @@ -189,7 +189,7 @@ public void testRaceForTask() throws Exception { } } - @Test + @Test(timeout=60000) public void testPreemptTask() throws Exception { LOG.info("testPreemptTask"); ZKSplitLog.Counters.resetCounters(); @@ -206,20 +206,20 @@ public void testPreemptTask() throws Exception { TaskState.TASK_UNASSIGNED.get("manager"), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); - waitForCounter(tot_wkr_task_acquired, 0, 1, 1000); + waitForCounter(tot_wkr_task_acquired, 0, 1, WAIT_TIME); assertEquals(1, slw.taskReadySeq); assertTrue(TaskState.TASK_OWNED.equals(ZKUtil.getData(zkw, ZKSplitLog.getEncodedNodeName(zkw, "tpt_task")), "tpt_svr")); ZKUtil.setData(zkw, ZKSplitLog.getEncodedNodeName(zkw, "tpt_task"), TaskState.TASK_UNASSIGNED.get("manager")); - waitForCounter(tot_wkr_preempt_task, 0, 1, 1000); + waitForCounter(tot_wkr_preempt_task, 0, 1, WAIT_TIME); } finally { stopSplitLogWorker(slw); } } - @Test + @Test(timeout=60000) public void testMultipleTasks() throws Exception { LOG.info("testMultipleTasks"); ZKSplitLog.Counters.resetCounters(); @@ -234,7 +234,7 @@ public void testMultipleTasks() throws Exception { TaskState.TASK_UNASSIGNED.get("manager"), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); - waitForCounter(tot_wkr_task_acquired, 0, 1, 1000); + waitForCounter(tot_wkr_task_acquired, 0, 1, WAIT_TIME); // now the worker is busy doing the above task // create another task @@ -245,9 +245,9 @@ public void testMultipleTasks() throws Exception { // preempt the first task, have it owned by another worker ZKUtil.setData(zkw, ZKSplitLog.getEncodedNodeName(zkw, "tmt_task"), TaskState.TASK_OWNED.get("another-worker")); - waitForCounter(tot_wkr_preempt_task, 0, 1, 1000); + waitForCounter(tot_wkr_preempt_task, 0, 1, WAIT_TIME); - waitForCounter(tot_wkr_task_acquired, 1, 2, 1000); + waitForCounter(tot_wkr_task_acquired, 1, 2, WAIT_TIME); assertEquals(2, slw.taskReadySeq); assertTrue(TaskState.TASK_OWNED.equals(ZKUtil.getData(zkw, ZKSplitLog.getEncodedNodeName(zkw, "tmt_task_2")), "tmt_svr")); @@ -256,7 +256,7 @@ public void testMultipleTasks() throws Exception { } } - @Test + @Test(timeout=60000) public void testRescan() throws Exception { LOG.info("testRescan"); ZKSplitLog.Counters.resetCounters(); @@ -264,19 +264,19 @@ public void testRescan() throws Exception { "svr", neverEndingTask); slw.start(); Thread.yield(); // let the worker start - Thread.sleep(100); + Thread.sleep(200); String task = ZKSplitLog.getEncodedNodeName(zkw, "task"); zkw.getRecoverableZooKeeper().create(task, TaskState.TASK_UNASSIGNED.get("manager"), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); - waitForCounter(tot_wkr_task_acquired, 0, 1, 1000); + waitForCounter(tot_wkr_task_acquired, 0, 1, WAIT_TIME); // now the worker is busy doing the above task // preempt the task, have it owned by another worker ZKUtil.setData(zkw, task, TaskState.TASK_UNASSIGNED.get("manager")); - waitForCounter(tot_wkr_preempt_task, 0, 1, 1000); + waitForCounter(tot_wkr_preempt_task, 0, 1, WAIT_TIME); // create a RESCAN node String rescan = ZKSplitLog.getEncodedNodeName(zkw, "RESCAN"); @@ -284,13 +284,13 @@ public void testRescan() throws Exception { TaskState.TASK_UNASSIGNED.get("manager"), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT_SEQUENTIAL); - waitForCounter(tot_wkr_task_acquired, 1, 2, 1000); + waitForCounter(tot_wkr_task_acquired, 1, 2, WAIT_TIME); // RESCAN node might not have been processed if the worker became busy // with the above task. preempt the task again so that now the RESCAN // node is processed ZKUtil.setData(zkw, task, TaskState.TASK_UNASSIGNED.get("manager")); - waitForCounter(tot_wkr_preempt_task, 1, 2, 1000); - waitForCounter(tot_wkr_task_acquired_rescan, 0, 1, 1000); + waitForCounter(tot_wkr_preempt_task, 1, 2, WAIT_TIME); + waitForCounter(tot_wkr_task_acquired_rescan, 0, 1, WAIT_TIME); List nodes = ZKUtil.listChildrenNoWatch(zkw, zkw.splitLogZNode); LOG.debug(nodes); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java index 8dc3d0c05e08..965a32fff31c 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -31,8 +30,18 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.Server; +import org.apache.hadoop.hbase.SmallTests; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.io.hfile.CacheConfig; +import org.apache.hadoop.hbase.io.hfile.HFile; +import org.apache.hadoop.hbase.io.hfile.LruBlockCache; import org.apache.hadoop.hbase.regionserver.wal.HLog; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.PairOfSameType; @@ -43,6 +52,8 @@ import org.junit.experimental.categories.Category; import org.mockito.Mockito; +import com.google.common.collect.ImmutableList; + /** * Test the {@link SplitTransaction} class against an HRegion (as opposed to * running cluster). @@ -137,6 +148,28 @@ private SplitTransaction prepareGOOD_SPLIT_ROW() { return st; } + /** + * Pass a reference store + */ + @Test public void testPrepareWithRegionsWithReference() throws IOException { + // create a mock that will act as a reference StoreFile + StoreFile storeFileMock = Mockito.mock(StoreFile.class); + when(storeFileMock.isReference()).thenReturn(true); + + // add the mock to the parent stores + Store storeMock = Mockito.mock(Store.class); + List storeFileList = new ArrayList(1); + storeFileList.add(storeFileMock); + when(storeMock.getStorefiles()).thenReturn(storeFileList); + when(storeMock.close()).thenReturn(ImmutableList.copyOf(storeFileList)); + this.parent.stores.put(Bytes.toBytes(""), storeMock); + + SplitTransaction st = new SplitTransaction(this.parent, GOOD_SPLIT_ROW); + + assertFalse("a region should not be splittable if it has instances of store file references", + st.prepare()); + } + /** * Pass an unreasonable split row. */ @@ -158,12 +191,32 @@ private SplitTransaction prepareGOOD_SPLIT_ROW() { assertFalse(st.prepare()); } + @Test public void testWholesomeSplitWithHFileV1() throws IOException { + int defaultVersion = TEST_UTIL.getConfiguration().getInt( + HFile.FORMAT_VERSION_KEY, 2); + TEST_UTIL.getConfiguration().setInt(HFile.FORMAT_VERSION_KEY, 1); + try { + for (Store store : this.parent.stores.values()) { + store.getFamily().setBloomFilterType(StoreFile.BloomType.ROW); + } + testWholesomeSplit(); + } finally { + TEST_UTIL.getConfiguration().setInt(HFile.FORMAT_VERSION_KEY, + defaultVersion); + } + } + @Test public void testWholesomeSplit() throws IOException { - final int rowcount = TEST_UTIL.loadRegion(this.parent, CF); + final int rowcount = TEST_UTIL.loadRegion(this.parent, CF, true); assertTrue(rowcount > 0); int parentRowCount = countRows(this.parent); assertEquals(rowcount, parentRowCount); + // Pretend region's blocks are not in the cache, used for + // testWholesomeSplitWithHFileV1 + CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration()); + ((LruBlockCache) cacheConf.getBlockCache()).clearCache(); + // Start transaction. SplitTransaction st = prepareGOOD_SPLIT_ROW(); @@ -193,7 +246,7 @@ private SplitTransaction prepareGOOD_SPLIT_ROW() { for (HRegion r: daughters) { // Open so can count its content. HRegion openRegion = HRegion.openHRegion(this.testdir, r.getRegionInfo(), - r.getTableDesc(), r.getLog(), r.getConf()); + r.getTableDesc(), r.getLog(), TEST_UTIL.getConfiguration()); try { int count = countRows(openRegion); assertTrue(count > 0 && count != rowcount); @@ -249,7 +302,7 @@ private SplitTransaction prepareGOOD_SPLIT_ROW() { for (HRegion r: daughters) { // Open so can count its content. HRegion openRegion = HRegion.openHRegion(this.testdir, r.getRegionInfo(), - r.getTableDesc(), r.getLog(), r.getConf()); + r.getTableDesc(), r.getLog(), TEST_UTIL.getConfiguration()); try { int count = countRows(openRegion); assertTrue(count > 0 && count != rowcount); @@ -295,7 +348,9 @@ HRegion createRegion(final Path testdir, final HLog wal) HColumnDescriptor hcd = new HColumnDescriptor(CF); htd.addFamily(hcd); HRegionInfo hri = new HRegionInfo(htd.getName(), STARTROW, ENDROW); - HRegion.createHRegion(hri, testdir, TEST_UTIL.getConfiguration(), htd); + HRegion r = HRegion.createHRegion(hri, testdir, TEST_UTIL.getConfiguration(), htd); + r.close(); + r.getLog().closeAndDelete(); return HRegion.openHRegion(testdir, hri, htd, wal, TEST_UTIL.getConfiguration()); } diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java index 1997abd531cb..fba778720313 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,25 +19,44 @@ package org.apache.hadoop.hbase.regionserver; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; import static org.junit.Assert.assertNotSame; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.fail; import java.io.IOException; import java.util.List; +import java.util.NavigableMap; +import java.util.concurrent.CountDownLatch; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.catalog.MetaReader; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.executor.EventHandler.EventType; import org.apache.hadoop.hbase.executor.RegionTransitionData; +import org.apache.hadoop.hbase.master.AssignmentManager; +import org.apache.hadoop.hbase.master.AssignmentManager.RegionState; +import org.apache.hadoop.hbase.master.HMaster; import org.apache.hadoop.hbase.master.handler.SplitRegionHandler; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.JVMClusterUtil.RegionServerThread; import org.apache.hadoop.hbase.util.Threads; import org.apache.hadoop.hbase.zookeeper.ZKAssign; +import org.apache.hadoop.hbase.zookeeper.ZKUtil; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.KeeperException.NodeExistsException; import org.apache.zookeeper.data.Stat; @@ -54,13 +72,18 @@ * is tests against a bare {@link HRegion}. */ @Category(LargeTests.class) +@SuppressWarnings("deprecation") public class TestSplitTransactionOnCluster { private static final Log LOG = LogFactory.getLog(TestSplitTransactionOnCluster.class); private HBaseAdmin admin = null; private MiniHBaseCluster cluster = null; private static final int NB_SERVERS = 2; - + private static CountDownLatch latch = new CountDownLatch(1); + private static volatile boolean secondSplit = false; + private static volatile boolean callRollBack = false; + private static volatile boolean firstSplitCompleted = false; + private static final HBaseTestingUtility TESTING_UTIL = new HBaseTestingUtility(); @@ -78,7 +101,7 @@ public class TestSplitTransactionOnCluster { } @Before public void setup() throws IOException { - TESTING_UTIL.ensureSomeRegionServersAvailable(NB_SERVERS); + TESTING_UTIL.ensureSomeNonStoppedRegionServersAvailable(NB_SERVERS); this.admin = new HBaseAdmin(TESTING_UTIL.getConfiguration()); this.cluster = TESTING_UTIL.getMiniHBaseCluster(); } @@ -88,6 +111,81 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { return regions.get(0).getRegionInfo(); } + @Test(timeout = 2000000) + public void testShouldFailSplitIfZNodeDoesNotExistDueToPrevRollBack() throws Exception { + final byte[] tableName = Bytes + .toBytes("testShouldFailSplitIfZNodeDoesNotExistDueToPrevRollBack"); + HBaseAdmin admin = new HBaseAdmin(TESTING_UTIL.getConfiguration()); + try { + // Create table then get the single region for our new table. + HTable t = createTableAndWait(tableName, Bytes.toBytes("cf")); + final List regions = cluster.getRegions(tableName); + HRegionInfo hri = getAndCheckSingleTableRegion(regions); + int regionServerIndex = cluster.getServerWith(regions.get(0).getRegionName()); + final HRegionServer regionServer = cluster.getRegionServer(regionServerIndex); + insertData(tableName, admin, t); + // Turn off balancer so it doesn't cut in and mess up our placements. + this.admin.setBalancerRunning(false, false); + // Turn off the meta scanner so it don't remove parent on us. + cluster.getMaster().setCatalogJanitorEnabled(false); + + new Thread() { + public void run() { + SplitTransaction st = null; + st = new MockedSplitTransaction(regions.get(0), Bytes.toBytes("row2")); + try { + st.prepare(); + st.execute(regionServer, regionServer); + } catch (IOException e) { + + } + } + }.start(); + for (int i = 0; !callRollBack && i < 100; i++) { + Thread.sleep(100); + } + assertTrue("Waited too long for rollback", callRollBack); + SplitTransaction st = null; + st = new MockedSplitTransaction(regions.get(0), Bytes.toBytes("row2")); + try { + secondSplit = true; + st.prepare(); + st.execute(regionServer, regionServer); + } catch (IOException e) { + LOG.debug("Rollback started :"+ e.getMessage()); + st.rollback(regionServer, regionServer); + } + for (int i=0; !firstSplitCompleted && i<100; i++) { + Thread.sleep(100); + } + assertTrue("fist split did not complete", firstSplitCompleted); + NavigableMap rit = cluster.getMaster().getAssignmentManager() + .getRegionsInTransition(); + for (int i=0; rit.containsKey(hri.getTableNameAsString()) && i<100; i++) { + Thread.sleep(100); + } + assertFalse("region still in transition", rit.containsKey(hri.getTableNameAsString())); + List onlineRegions = regionServer.getOnlineRegions(tableName); + // Region server side split is successful. + assertEquals("The parent region should be splitted", 2, onlineRegions.size()); + //Should be present in RIT + List regionsOfTable = cluster.getMaster().getAssignmentManager().getRegionsOfTable(tableName); + // Master side should also reflect the same + assertEquals("No of regions in master", 2, regionsOfTable.size()); + } finally { + admin.setBalancerRunning(true, false); + secondSplit = false; + firstSplitCompleted = false; + callRollBack = false; + cluster.getMaster().setCatalogJanitorEnabled(true); + } + if (admin.isTableAvailable(tableName) && admin.isTableEnabled(tableName)) { + admin.disableTable(tableName); + admin.deleteTable(tableName); + admin.close(); + } + } + /** * A test that intentionally has master fail the processing of the split message. * Tests that the regionserver split ephemeral node gets cleaned up if it @@ -104,15 +202,14 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { Bytes.toBytes("ephemeral"); // Create table then get the single region for our new table. - HTable t = TESTING_UTIL.createTable(tableName, HConstants.CATALOG_FAMILY); - + HTable t = createTableAndWait(tableName, HConstants.CATALOG_FAMILY); List regions = cluster.getRegions(tableName); HRegionInfo hri = getAndCheckSingleTableRegion(regions); int tableRegionIndex = ensureTableRegionNotOnSameServerAsMeta(admin, hri); // Turn off balancer so it doesn't cut in and mess up our placements. - this.admin.balanceSwitch(false); + this.admin.setBalancerRunning(false, true); // Turn off the meta scanner so it don't remove parent on us. cluster.getMaster().setCatalogJanitorEnabled(false); try { @@ -128,8 +225,7 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { // Now try splitting and it should work. split(hri, server, regionCount); // Get daughters - List daughters = cluster.getRegions(tableName); - assertTrue(daughters.size() >= 2); + List daughters = checkAndGetDaughters(tableName); // Assert the ephemeral node is up in zk. String path = ZKAssign.getNodeName(t.getConnection().getZooKeeperWatcher(), hri.getEncodedName()); @@ -145,26 +241,28 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { // Now crash the server cluster.abortRegionServer(tableRegionIndex); waitUntilRegionServerDead(); + awaitDaughters(tableName, daughters.size()); - // Wait till regions are back on line again. - while(cluster.getRegions(tableName).size() < daughters.size()) { - LOG.info("Waiting for repair to happen"); - Thread.sleep(1000); - } // Assert daughters are online. regions = cluster.getRegions(tableName); for (HRegion r: regions) { assertTrue(daughters.contains(r)); } // Finally assert that the ephemeral SPLIT znode was cleaned up. - stats = t.getConnection().getZooKeeperWatcher().getRecoverableZooKeeper().exists(path, false); + for (int i=0; i<100; i++) { + // wait a bit (10s max) for the node to disappear + stats = t.getConnection().getZooKeeperWatcher().getRecoverableZooKeeper().exists(path, false); + if (stats == null) break; + Thread.sleep(100); + } LOG.info("EPHEMERAL NODE AFTER SERVER ABORT, path=" + path + ", stats=" + stats); assertTrue(stats == null); } finally { // Set this flag back. SplitRegionHandler.TEST_SKIP = false; - admin.balanceSwitch(true); + admin.setBalancerRunning(true, false); cluster.getMaster().setCatalogJanitorEnabled(true); + t.close(); } } @@ -174,15 +272,14 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { Bytes.toBytes("testExistingZnodeBlocksSplitAndWeRollback"); // Create table then get the single region for our new table. - HTable t = TESTING_UTIL.createTable(tableName, HConstants.CATALOG_FAMILY); - + HTable t = createTableAndWait(tableName, HConstants.CATALOG_FAMILY); List regions = cluster.getRegions(tableName); HRegionInfo hri = getAndCheckSingleTableRegion(regions); int tableRegionIndex = ensureTableRegionNotOnSameServerAsMeta(admin, hri); // Turn off balancer so it doesn't cut in and mess up our placements. - this.admin.balanceSwitch(false); + this.admin.setBalancerRunning(false, true); // Turn off the meta scanner so it don't remove parent on us. cluster.getMaster().setCatalogJanitorEnabled(false); try { @@ -211,12 +308,12 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { // Now try splitting and it should work. split(hri, server, regionCount); // Get daughters - List daughters = cluster.getRegions(tableName); - assertTrue(daughters.size() >= 2); + checkAndGetDaughters(tableName); // OK, so split happened after we cleared the blocking node. } finally { - admin.balanceSwitch(true); + admin.setBalancerRunning(true, false); cluster.getMaster().setCatalogJanitorEnabled(true); + t.close(); } } @@ -233,15 +330,14 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { final byte [] tableName = Bytes.toBytes("testShutdownSimpleFixup"); // Create table then get the single region for our new table. - HTable t = TESTING_UTIL.createTable(tableName, HConstants.CATALOG_FAMILY); - + HTable t = createTableAndWait(tableName, HConstants.CATALOG_FAMILY); List regions = cluster.getRegions(tableName); HRegionInfo hri = getAndCheckSingleTableRegion(regions); int tableRegionIndex = ensureTableRegionNotOnSameServerAsMeta(admin, hri); // Turn off balancer so it doesn't cut in and mess up our placements. - this.admin.balanceSwitch(false); + this.admin.setBalancerRunning(false, true); // Turn off the meta scanner so it don't remove parent on us. cluster.getMaster().setCatalogJanitorEnabled(false); try { @@ -254,27 +350,23 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { // Now split. split(hri, server, regionCount); // Get daughters - List daughters = cluster.getRegions(tableName); - assertTrue(daughters.size() >= 2); + List daughters = checkAndGetDaughters(tableName); // Remove one of the daughters from .META. to simulate failed insert of // daughter region up into .META. removeDaughterFromMeta(daughters.get(0).getRegionName()); // Now crash the server cluster.abortRegionServer(tableRegionIndex); waitUntilRegionServerDead(); - // Wait till regions are back on line again. - while(cluster.getRegions(tableName).size() < daughters.size()) { - LOG.info("Waiting for repair to happen"); - Thread.sleep(1000); - } + awaitDaughters(tableName, daughters.size()); // Assert daughters are online. regions = cluster.getRegions(tableName); for (HRegion r: regions) { assertTrue(daughters.contains(r)); } } finally { - admin.balanceSwitch(true); + admin.setBalancerRunning(true, false); cluster.getMaster().setCatalogJanitorEnabled(true); + t.close(); } } @@ -290,15 +382,14 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { Bytes.toBytes("testShutdownFixupWhenDaughterHasSplit"); // Create table then get the single region for our new table. - HTable t = TESTING_UTIL.createTable(tableName, HConstants.CATALOG_FAMILY); - + HTable t = createTableAndWait(tableName, HConstants.CATALOG_FAMILY); List regions = cluster.getRegions(tableName); HRegionInfo hri = getAndCheckSingleTableRegion(regions); int tableRegionIndex = ensureTableRegionNotOnSameServerAsMeta(admin, hri); // Turn off balancer so it doesn't cut in and mess up our placements. - this.admin.balanceSwitch(false); + this.admin.setBalancerRunning(false, true); // Turn off the meta scanner so it don't remove parent on us. cluster.getMaster().setCatalogJanitorEnabled(false); try { @@ -311,8 +402,7 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { // Now split. split(hri, server, regionCount); // Get daughters - List daughters = cluster.getRegions(tableName); - assertTrue(daughters.size() >= 2); + List daughters = checkAndGetDaughters(tableName); // Now split one of the daughters. regionCount = server.getOnlineRegions().size(); HRegionInfo daughter = daughters.get(0).getRegionInfo(); @@ -325,21 +415,18 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { if (r.getRegionInfo().equals(daughter)) daughterRegion = r; } assertTrue(daughterRegion != null); - while (true) { + for (int i=0; i<100; i++) { if (!daughterRegion.hasReferences()) break; Threads.sleep(100); } + assertFalse("Waiting for refereces to be compacted", daughterRegion.hasReferences()); split(daughter, server, regionCount); // Get list of daughters daughters = cluster.getRegions(tableName); // Now crash the server cluster.abortRegionServer(tableRegionIndex); waitUntilRegionServerDead(); - // Wait till regions are back on line again. - while(cluster.getRegions(tableName).size() < daughters.size()) { - LOG.info("Waiting for repair to happen"); - Thread.sleep(1000); - } + awaitDaughters(tableName, daughters.size()); // Assert daughters are online and ONLY the original daughters -- that // fixup didn't insert one during server shutdown recover. regions = cluster.getRegions(tableName); @@ -348,19 +435,512 @@ private HRegionInfo getAndCheckSingleTableRegion(final List regions) { assertTrue(daughters.contains(r)); } } finally { - admin.balanceSwitch(true); + admin.setBalancerRunning(true, false); + cluster.getMaster().setCatalogJanitorEnabled(true); + t.close(); + } + } + + /** + * Verifies HBASE-5806. When splitting is partially done and the master goes down + * when the SPLIT node is in either SPLIT or SPLITTING state. + * + * @throws IOException + * @throws InterruptedException + * @throws NodeExistsException + * @throws KeeperException + */ + @Test(timeout = 300000) + public void testMasterRestartWhenSplittingIsPartial() + throws IOException, InterruptedException, NodeExistsException, + KeeperException { + final byte[] tableName = Bytes.toBytes("testMasterRestartWhenSplittingIsPartial"); + + // Create table then get the single region for our new table. + HTable t = createTableAndWait(tableName, HConstants.CATALOG_FAMILY); + List regions = cluster.getRegions(tableName); + HRegionInfo hri = getAndCheckSingleTableRegion(regions); + + int tableRegionIndex = ensureTableRegionNotOnSameServerAsMeta(admin, hri); + + // Turn off the meta scanner so it don't remove parent on us. + cluster.getMaster().setCatalogJanitorEnabled(false); + // Turn off balancer so it doesn't cut in and mess up our placements. + this.admin.setBalancerRunning(false, true); + + try { + // Add a bit of load up into the table so splittable. + TESTING_UTIL.loadTable(t, HConstants.CATALOG_FAMILY); + // Get region pre-split. + HRegionServer server = cluster.getRegionServer(tableRegionIndex); + printOutRegions(server, "Initial regions: "); + int regionCount = server.getOnlineRegions().size(); + // Now, before we split, set special flag in master, a flag that has + // it FAIL the processing of split. + SplitRegionHandler.TEST_SKIP = true; + // Now try splitting and it should work. + split(hri, server, regionCount); + // Get daughters + checkAndGetDaughters(tableName); + // Assert the ephemeral node is up in zk. + String path = ZKAssign.getNodeName(t.getConnection() + .getZooKeeperWatcher(), hri.getEncodedName()); + Stat stats = t.getConnection().getZooKeeperWatcher() + .getRecoverableZooKeeper().exists(path, false); + LOG.info("EPHEMERAL NODE BEFORE SERVER ABORT, path=" + path + ", stats=" + + stats); + RegionTransitionData rtd = ZKAssign.getData(t.getConnection() + .getZooKeeperWatcher(), hri.getEncodedName()); + // State could be SPLIT or SPLITTING. + assertTrue(rtd.getEventType().equals(EventType.RS_ZK_REGION_SPLIT) + || rtd.getEventType().equals(EventType.RS_ZK_REGION_SPLITTING)); + + + // abort and wait for new master. + MockMasterWithoutCatalogJanitor master = abortAndWaitForMaster(); + + this.admin = new HBaseAdmin(TESTING_UTIL.getConfiguration()); + + // update the hri to be offlined and splitted. + hri.setOffline(true); + hri.setSplit(true); + ServerName regionServerOfRegion = master.getAssignmentManager() + .getRegionServerOfRegion(hri); + assertTrue(regionServerOfRegion != null); + + } finally { + // Set this flag back. + SplitRegionHandler.TEST_SKIP = false; + admin.setBalancerRunning(true, false); cluster.getMaster().setCatalogJanitorEnabled(true); + t.close(); } } + + /** + * Verifies HBASE-5806. Here the case is that splitting is completed but before the + * CJ could remove the parent region the master is killed and restarted. + * @throws IOException + * @throws InterruptedException + * @throws NodeExistsException + * @throws KeeperException + */ + @Test (timeout = 300000) + public void testMasterRestartAtRegionSplitPendingCatalogJanitor() + throws IOException, InterruptedException, NodeExistsException, + KeeperException { + final byte[] tableName = Bytes.toBytes("testMasterRestartAtRegionSplitPendingCatalogJanitor"); + + // Create table then get the single region for our new table. + this.admin = new HBaseAdmin(TESTING_UTIL.getConfiguration()); + HTable t = createTableAndWait(tableName, HConstants.CATALOG_FAMILY); + List regions = cluster.getRegions(tableName); + HRegionInfo hri = getAndCheckSingleTableRegion(regions); + + int tableRegionIndex = ensureTableRegionNotOnSameServerAsMeta(admin, hri); + + // Turn off balancer so it doesn't cut in and mess up our placements. + this.admin.setBalancerRunning(false, true); + // Turn off the meta scanner so it don't remove parent on us. + cluster.getMaster().setCatalogJanitorEnabled(false); + try { + // Add a bit of load up into the table so splittable. + TESTING_UTIL.loadTable(t, HConstants.CATALOG_FAMILY); + // Get region pre-split. + HRegionServer server = cluster.getRegionServer(tableRegionIndex); + printOutRegions(server, "Initial regions: "); + int regionCount = server.getOnlineRegions().size(); + + split(hri, server, regionCount); + // Get daughters + checkAndGetDaughters(tableName); + // Assert the ephemeral node is up in zk. + String path = ZKAssign.getNodeName(t.getConnection() + .getZooKeeperWatcher(), hri.getEncodedName()); + Stat stats = t.getConnection().getZooKeeperWatcher() + .getRecoverableZooKeeper().exists(path, false); + LOG.info("EPHEMERAL NODE BEFORE SERVER ABORT, path=" + path + ", stats=" + + stats); + String node = ZKAssign.getNodeName(t.getConnection() + .getZooKeeperWatcher(), hri.getEncodedName()); + Stat stat = new Stat(); + byte[] data = ZKUtil.getDataNoWatch(t.getConnection() + .getZooKeeperWatcher(), node, stat); + // ZKUtil.create + for (int i=0; data != null && i<60; i++) { + Thread.sleep(1000); + data = ZKUtil.getDataNoWatch(t.getConnection().getZooKeeperWatcher(), + node, stat); + + } + assertNull("Waited too long for ZK node to be removed: "+node, data); + + MockMasterWithoutCatalogJanitor master = abortAndWaitForMaster(); + + this.admin = new HBaseAdmin(TESTING_UTIL.getConfiguration()); + + hri.setOffline(true); + hri.setSplit(true); + ServerName regionServerOfRegion = master.getAssignmentManager() + .getRegionServerOfRegion(hri); + assertTrue(regionServerOfRegion == null); + } finally { + // Set this flag back. + SplitRegionHandler.TEST_SKIP = false; + this.admin.setBalancerRunning(true, false); + cluster.getMaster().setCatalogJanitorEnabled(true); + t.close(); + } + } + + /** + * While transitioning node from RS_ZK_REGION_SPLITTING to + * RS_ZK_REGION_SPLITTING during region split,if zookeper went down split always + * fails for the region. HBASE-6088 fixes this scenario. + * This test case is to test the znode is deleted(if created) or not in roll back. + * + * @throws IOException + * @throws InterruptedException + * @throws KeeperException + */ + @Test + public void testSplitBeforeSettingSplittingInZK() throws Exception, + InterruptedException, KeeperException { + testSplitBeforeSettingSplittingInZK(true); + testSplitBeforeSettingSplittingInZK(false); + } + + private void testSplitBeforeSettingSplittingInZK(boolean nodeCreated) throws Exception { + final byte[] tableName = Bytes.toBytes("testSplitBeforeSettingSplittingInZK"); + + HBaseAdmin admin = new HBaseAdmin(TESTING_UTIL.getConfiguration()); + // Create table then get the single region for our new table. + HTableDescriptor htd = new HTableDescriptor(tableName); + htd.addFamily(new HColumnDescriptor("cf")); + admin.createTable(htd); + + List regions = null; + for (int i=0; i<100; i++) { + regions = cluster.getRegions(tableName); + if (regions.size() > 0) break; + Thread.sleep(100); + } + int regionServerIndex = cluster.getServerWith(regions.get(0).getRegionName()); + HRegionServer regionServer = cluster.getRegionServer(regionServerIndex); + SplitTransaction st = null; + if (nodeCreated) { + st = new MockedSplitTransaction(regions.get(0), null) { + @Override + int transitionNodeSplitting(ZooKeeperWatcher zkw, HRegionInfo parent, + ServerName serverName, int version) throws KeeperException, IOException { + throw new TransitionToSplittingFailedException(); + } + }; + } else { + st = new MockedSplitTransaction(regions.get(0), null) { + @Override + void createNodeSplitting(ZooKeeperWatcher zkw, HRegionInfo region, ServerName serverName) + throws KeeperException, IOException { + throw new SplittingNodeCreationFailedException (); + } + }; + } + String node = ZKAssign.getNodeName(regionServer.getZooKeeper(), regions.get(0) + .getRegionInfo().getEncodedName()); + // make sure the client is uptodate + regionServer.getZooKeeper().sync(node); + for (int i = 0; i < 100; i++) { + // We expect the znode to be deleted by this time. Here the znode could be in OPENED state and the + // master has not yet deleted the znode. + if (ZKUtil.checkExists(regionServer.getZooKeeper(), node) != -1) { + Thread.sleep(100); + } + } + + try { + st.execute(regionServer, regionServer); + } catch (IOException e) { + // check for the specific instance in case the Split failed due to the existence of the znode in OPENED state. + // This will at least make the test to fail; + if (nodeCreated) { + assertTrue("Should be instance of TransitionToSplittingFailedException", + e instanceof TransitionToSplittingFailedException); + } else { + assertTrue("Should be instance of CreateSplittingNodeFailedException", + e instanceof SplittingNodeCreationFailedException ); + } + node = ZKAssign.getNodeName(regionServer.getZooKeeper(), regions.get(0) + .getRegionInfo().getEncodedName()); + // make sure the client is uptodate + regionServer.getZooKeeper().sync(node); + if (nodeCreated) { + assertFalse(ZKUtil.checkExists(regionServer.getZooKeeper(), node) == -1); + } else { + assertTrue(ZKUtil.checkExists(regionServer.getZooKeeper(), node) == -1); + } + assertTrue(st.rollback(regionServer, regionServer)); + assertTrue(ZKUtil.checkExists(regionServer.getZooKeeper(), node) == -1); + } + if (admin.isTableAvailable(tableName) && admin.isTableEnabled(tableName)) { + admin.disableTable(tableName); + admin.deleteTable(tableName); + } + } + + @Test + public void testShouldClearRITWhenNodeFoundInSplittingState() throws Exception { + final byte[] tableName = Bytes.toBytes("testShouldClearRITWhenNodeFoundInSplittingState"); + HBaseAdmin admin = new HBaseAdmin(TESTING_UTIL.getConfiguration()); + // Create table then get the single region for our new table. + HTableDescriptor htd = new HTableDescriptor(tableName); + htd.addFamily(new HColumnDescriptor("cf")); + admin.createTable(htd); + for (int i = 0; cluster.getRegions(tableName).size() == 0 && i < 100; i++) { + Thread.sleep(100); + } + assertTrue("Table not online", cluster.getRegions(tableName).size() != 0); + + HRegion region = cluster.getRegions(tableName).get(0); + int regionServerIndex = cluster.getServerWith(region.getRegionName()); + HRegionServer regionServer = cluster.getRegionServer(regionServerIndex); + SplitTransaction st = null; + + st = new MockedSplitTransaction(region, null) { + @Override + void createSplitDir(FileSystem fs, Path splitdir) throws IOException { + throw new IOException(""); + } + }; + + try { + st.execute(regionServer, regionServer); + } catch (IOException e) { + String node = ZKAssign.getNodeName(regionServer.getZooKeeper(), region + .getRegionInfo().getEncodedName()); + + assertFalse(ZKUtil.checkExists(regionServer.getZooKeeper(), node) == -1); + AssignmentManager am = cluster.getMaster().getAssignmentManager(); + for (int i = 0; !am.getRegionsInTransition().containsKey( + region.getRegionInfo().getEncodedName()) + && i < 100; i++) { + Thread.sleep(200); + } + assertTrue("region is not in transition "+region, + am.getRegionsInTransition().containsKey(region.getRegionInfo().getEncodedName())); + RegionState regionState = am.getRegionsInTransition().get(region.getRegionInfo() + .getEncodedName()); + assertTrue(regionState.getState() == RegionState.State.SPLITTING); + assertTrue(st.rollback(regionServer, regionServer)); + assertTrue(ZKUtil.checkExists(regionServer.getZooKeeper(), node) == -1); + for (int i=0; am.getRegionsInTransition().containsKey(region.getRegionInfo().getEncodedName()) && i<100; i++) { + // Just in case the nodeDeleted event did not get executed. + Thread.sleep(200); + } + assertFalse("region is still in transition", + am.getRegionsInTransition().containsKey(region.getRegionInfo().getEncodedName())); + } + if (admin.isTableAvailable(tableName) && admin.isTableEnabled(tableName)) { + admin.disableTable(tableName); + admin.deleteTable(tableName); + admin.close(); + } + } + + @Test(timeout = 60000) + public void testTableExistsIfTheSpecifiedTableRegionIsSplitParent() throws Exception { + ZooKeeperWatcher zkw = HBaseTestingUtility.getZooKeeperWatcher(TESTING_UTIL); + final byte[] tableName = + Bytes.toBytes("testTableExistsIfTheSpecifiedTableRegionIsSplitParent"); + HRegionServer regionServer = null; + List regions = null; + HBaseAdmin admin = new HBaseAdmin(TESTING_UTIL.getConfiguration()); + try { + // Create table then get the single region for our new table. + HTable t = createTableAndWait(tableName, Bytes.toBytes("cf")); + regions = cluster.getRegions(tableName); + int regionServerIndex = cluster.getServerWith(regions.get(0).getRegionName()); + regionServer = cluster.getRegionServer(regionServerIndex); + insertData(tableName, admin, t); + // Turn off balancer so it doesn't cut in and mess up our placements. + cluster.getMaster().setCatalogJanitorEnabled(false); + boolean tableExists = MetaReader.tableExists(regionServer.getCatalogTracker(), + Bytes.toString(tableName)); + assertEquals("The specified table should present.", true, tableExists); + SplitTransaction st = new SplitTransaction(regions.get(0), Bytes.toBytes("row2")); + try { + st.prepare(); + st.createDaughters(regionServer, regionServer); + } catch (IOException e) { + + } + tableExists = MetaReader.tableExists(regionServer.getCatalogTracker(), + Bytes.toString(tableName)); + assertEquals("The specified table should present.", true, tableExists); + } finally { + if (regions != null) { + String node = ZKAssign.getNodeName(zkw, regions.get(0).getRegionInfo() + .getEncodedName()); + ZKUtil.deleteNodeFailSilent(zkw, node); + } + cluster.getMaster().setCatalogJanitorEnabled(true); + admin.close(); + } + } + + @Test(timeout = 180000) + public void testSplitShouldNotThrowNPEEvenARegionHasEmptySplitFiles() throws Exception { + Configuration conf = TESTING_UTIL.getConfiguration(); + String userTableName = "testSplitShouldNotThrowNPEEvenARegionHasEmptySplitFiles"; + HTableDescriptor htd = new HTableDescriptor(userTableName); + HColumnDescriptor hcd = new HColumnDescriptor("col"); + htd.addFamily(hcd); + admin.createTable(htd); + HTable table = new HTable(conf, userTableName); + try { + for (int i = 0; i <= 5; i++) { + String row = "row" + i; + Put p = new Put(row.getBytes()); + String val = "Val" + i; + p.add("col".getBytes(), "ql".getBytes(), val.getBytes()); + table.put(p); + admin.flush(userTableName); + Delete d = new Delete(row.getBytes()); + // Do a normal delete + table.delete(d); + admin.flush(userTableName); + } + admin.majorCompact(userTableName); + List regionsOfTable = TESTING_UTIL.getMiniHBaseCluster() + .getMaster().getAssignmentManager() + .getRegionsOfTable(userTableName.getBytes()); + HRegionInfo hRegionInfo = regionsOfTable.get(0); + Put p = new Put("row6".getBytes()); + p.add("col".getBytes(), "ql".getBytes(), "val".getBytes()); + table.put(p); + p = new Put("row7".getBytes()); + p.add("col".getBytes(), "ql".getBytes(), "val".getBytes()); + table.put(p); + p = new Put("row8".getBytes()); + p.add("col".getBytes(), "ql".getBytes(), "val".getBytes()); + table.put(p); + admin.flush(userTableName); + admin.split(hRegionInfo.getRegionName(), "row7".getBytes()); + regionsOfTable = TESTING_UTIL.getMiniHBaseCluster().getMaster() + .getAssignmentManager().getRegionsOfTable(userTableName.getBytes()); + + while (regionsOfTable.size() != 2) { + Thread.sleep(2000); + regionsOfTable = TESTING_UTIL.getMiniHBaseCluster().getMaster() + .getAssignmentManager().getRegionsOfTable(userTableName.getBytes()); + } + assertEquals(2, regionsOfTable.size()); + Scan s = new Scan(); + ResultScanner scanner = table.getScanner(s); + int mainTableCount = 0; + for (Result rr = scanner.next(); rr != null; rr = scanner.next()) { + mainTableCount++; + } + assertEquals(3, mainTableCount); + } finally { + table.close(); + } + } + + private void insertData(final byte[] tableName, HBaseAdmin admin, HTable t) throws IOException, + InterruptedException { + Put p = new Put(Bytes.toBytes("row1")); + p.add(Bytes.toBytes("cf"), Bytes.toBytes("q1"), Bytes.toBytes("1")); + t.put(p); + p = new Put(Bytes.toBytes("row2")); + p.add(Bytes.toBytes("cf"), Bytes.toBytes("q1"), Bytes.toBytes("2")); + t.put(p); + p = new Put(Bytes.toBytes("row3")); + p.add(Bytes.toBytes("cf"), Bytes.toBytes("q1"), Bytes.toBytes("3")); + t.put(p); + p = new Put(Bytes.toBytes("row4")); + p.add(Bytes.toBytes("cf"), Bytes.toBytes("q1"), Bytes.toBytes("4")); + t.put(p); + admin.flush(tableName); + } + + public static class MockedSplitTransaction extends SplitTransaction { + + private HRegion currentRegion; + public MockedSplitTransaction(HRegion r, byte[] splitrow) { + super(r, splitrow); + this.currentRegion = r; + } + + @Override + void transitionZKNode(Server server, RegionServerServices services, HRegion a, HRegion b) + throws IOException { + if (this.currentRegion.getRegionInfo().getTableNameAsString() + .equals("testShouldFailSplitIfZNodeDoesNotExistDueToPrevRollBack")) { + try { + if (!secondSplit){ + callRollBack = true; + latch.await(); + } + } catch (InterruptedException e) { + } + + } + super.transitionZKNode(server, services, a, b); + if (this.currentRegion.getRegionInfo().getTableNameAsString() + .equals("testShouldFailSplitIfZNodeDoesNotExistDueToPrevRollBack")) { + firstSplitCompleted = true; + } + } + @Override + public boolean rollback(Server server, RegionServerServices services) throws IOException { + if (this.currentRegion.getRegionInfo().getTableNameAsString() + .equals("testShouldFailSplitIfZNodeDoesNotExistDueToPrevRollBack")) { + if(secondSplit){ + super.rollback(server, services); + latch.countDown(); + return true; + } + } + return super.rollback(server, services); + } + + } + + private List checkAndGetDaughters(byte[] tableName) + throws InterruptedException { + List daughters = null; + // try up to 10s + for (int i=0; i<100; i++) { + daughters = cluster.getRegions(tableName); + if (daughters.size() >= 2) break; + Thread.sleep(100); + } + assertTrue(daughters.size() >= 2); + return daughters; + } + + private MockMasterWithoutCatalogJanitor abortAndWaitForMaster() + throws IOException, InterruptedException { + cluster.abortMaster(0); + cluster.waitOnMaster(0); + cluster.getConfiguration().setClass(HConstants.MASTER_IMPL, + MockMasterWithoutCatalogJanitor.class, HMaster.class); + MockMasterWithoutCatalogJanitor master = null; + master = (MockMasterWithoutCatalogJanitor) cluster.startMaster().getMaster(); + cluster.waitForActiveAndReadyMaster(); + return master; + } + private void split(final HRegionInfo hri, final HRegionServer server, final int regionCount) throws IOException, InterruptedException { this.admin.split(hri.getRegionNameAsString()); - while (server.getOnlineRegions().size() <= regionCount) { + for (int i=0; server.getOnlineRegions().size() <= regionCount && i<300; i++) { LOG.debug("Waiting on region to split"); Thread.sleep(100); } + assertFalse("Waited too long for split", server.getOnlineRegions().size() <= regionCount); } private void removeDaughterFromMeta(final byte [] regionName) throws IOException { @@ -398,19 +978,29 @@ private int ensureTableRegionNotOnSameServerAsMeta(final HBaseAdmin admin, HRegionServer tableRegionServer = cluster.getRegionServer(tableRegionIndex); if (metaRegionServer.getServerName().equals(tableRegionServer.getServerName())) { HRegionServer hrs = getOtherRegionServer(cluster, metaRegionServer); - LOG.info("Moving " + hri.getRegionNameAsString() + " to " + + assertNotNull(hrs); + assertNotNull(hri); + LOG. + info("Moving " + hri.getRegionNameAsString() + " to " + hrs.getServerName() + "; metaServerIndex=" + metaServerIndex); + for (int i = 0; cluster.getMaster().getAssignmentManager() + .getRegionServerOfRegion(hri) == null + && i < 100; i++) { + Thread.sleep(10); + } admin.move(hri.getEncodedNameAsBytes(), Bytes.toBytes(hrs.getServerName().toString())); } // Wait till table region is up on the server that is NOT carrying .META.. - while (true) { + for (int i=0; i<100; i++) { tableRegionIndex = cluster.getServerWith(hri.getRegionName()); if (tableRegionIndex != -1 && tableRegionIndex != metaServerIndex) break; LOG.debug("Waiting on region move off the .META. server; current index " + tableRegionIndex + " and metaServerIndex=" + metaServerIndex); Thread.sleep(100); } + assertTrue("Region not moved off .META. server", tableRegionIndex != -1 + && tableRegionIndex != metaServerIndex); // Verify for sure table region is not on same server as .META. tableRegionIndex = cluster.getServerWith(hri.getRegionName()); assertTrue(tableRegionIndex != -1); @@ -448,15 +1038,66 @@ private void printOutRegions(final HRegionServer hrs, final String prefix) private void waitUntilRegionServerDead() throws InterruptedException { // Wait until the master processes the RS shutdown - while (cluster.getMaster().getClusterStatus(). - getServers().size() == NB_SERVERS) { + for (int i=0; cluster.getMaster().getClusterStatus(). + getServers().size() == NB_SERVERS && i<100; i++) { LOG.info("Waiting on server to go down"); Thread.sleep(100); } + assertFalse("Waited too long for RS to die", cluster.getMaster().getClusterStatus(). + getServers().size() == NB_SERVERS); + } + + private void awaitDaughters(byte[] tableName, int numDaughters) throws InterruptedException { + // Wait till regions are back on line again. + for (int i=0; cluster.getRegions(tableName).size() < numDaughters && i<60; i++) { + LOG.info("Waiting for repair to happen"); + Thread.sleep(1000); + } + if (cluster.getRegions(tableName).size() < numDaughters) { + fail("Waiting too long for daughter regions"); + } + } + + private HTable createTableAndWait(byte[] tableName, byte[] cf) throws IOException, + InterruptedException { + HTable t = TESTING_UTIL.createTable(tableName, cf); + for (int i = 0; cluster.getRegions(tableName).size() == 0 && i < 100; i++) { + Thread.sleep(100); + } + assertTrue("Table not online: "+Bytes.toString(tableName), cluster.getRegions(tableName).size() != 0); + return t; } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); + + public static class MockMasterWithoutCatalogJanitor extends HMaster { + + public MockMasterWithoutCatalogJanitor(Configuration conf) throws IOException, KeeperException, + InterruptedException { + super(conf); + } + + protected void startCatalogJanitorChore() { + LOG.debug("Customised master executed."); + } + } + + private static class TransitionToSplittingFailedException extends IOException { + private static final long serialVersionUID = 7025885032995944524L; + + public TransitionToSplittingFailedException() { + super(); + } + } + + private static class SplittingNodeCreationFailedException extends IOException { + private static final long serialVersionUID = 1652404976265623004L; + + public SplittingNodeCreationFailedException () { + super(); + } + } } diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java index bd7204c968e1..4b8208968706 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -151,8 +150,10 @@ private void init(String methodName, Configuration conf, } public void testDeleteExpiredStoreFiles() throws Exception { + ManualEnvironmentEdge mee = new ManualEnvironmentEdge(); + EnvironmentEdgeManagerTestHelper.injectEdge(mee); int storeFileNum = 4; - int ttl = 1; + int ttl = 4; Configuration conf = HBaseConfiguration.create(); // Enable the expired store file deletion @@ -172,8 +173,10 @@ public void testDeleteExpiredStoreFiles() throws Exception { this.store.add(new KeyValue(row, family, qf2, timeStamp, (byte[]) null)); this.store.add(new KeyValue(row, family, qf3, timeStamp, (byte[]) null)); flush(i); - Thread.sleep(sleepTime); + mee.incValue(sleepTime); } + // move time forward a bit more, so that the first file is expired + mee.incValue(1); // Verify the total number of store files assertEquals(storeFileNum, this.store.getStorefiles().size()); @@ -183,15 +186,21 @@ public void testDeleteExpiredStoreFiles() throws Exception { for (int i = 1; i <= storeFileNum; i++) { // verify the expired store file. CompactionRequest cr = this.store.requestCompaction(); - assertEquals(1, cr.getFiles().size()); - assertTrue(cr.getFiles().get(0).getReader().getMaxTimestamp() < - (System.currentTimeMillis() - this.store.scanInfo.getTtl())); - // Verify that the expired the store has been deleted. + // the first is expired normally. + // If not the first compaction, there is another empty store file, + assertEquals(Math.min(i, 2), cr.getFiles().size()); + for (int j = 0; i < cr.getFiles().size(); j++) { + assertTrue(cr.getFiles().get(j).getReader().getMaxTimestamp() < (EnvironmentEdgeManager + .currentTimeMillis() - this.store.scanInfo.getTtl())); + } + // Verify that the expired store file is compacted to an empty store file. this.store.compact(cr); - assertEquals(storeFileNum - i, this.store.getStorefiles().size()); + // It is an empty store file. + assertEquals(0, this.store.getStorefiles().get(0).getReader() + .getEntries()); // Let the next store file expired. - Thread.sleep(sleepTime); + mee.incValue(sleepTime); } } diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java index 5b3b96234649..8cef69b87ec4 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -28,16 +27,23 @@ import java.util.List; import java.util.Map; import java.util.TreeSet; +import java.util.regex.Pattern; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseTestCase; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.SmallTests; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.io.HFileLink; +import org.apache.hadoop.hbase.io.HalfStoreFileReader; +import org.apache.hadoop.hbase.io.Reference; import org.apache.hadoop.hbase.io.Reference.Range; import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; import org.apache.hadoop.hbase.io.hfile.BlockCache; @@ -52,6 +58,8 @@ import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; import org.apache.hadoop.hbase.util.BloomFilterFactory; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.ChecksumType; +import org.apache.hadoop.hbase.util.FSUtils; import org.junit.experimental.categories.Category; import org.mockito.Mockito; @@ -69,6 +77,9 @@ public class TestStoreFile extends HBaseTestCase { private String ROOT_DIR; private Map startingMetrics; + private static final ChecksumType CKTYPE = ChecksumType.CRC32; + private static final int CKBYTES = 512; + @Override public void setUp() throws Exception { super.setUp(); @@ -88,8 +99,8 @@ public void tearDown() throws Exception { * @throws Exception */ public void testBasicHalfMapFile() throws Exception { - // Make up a directory hierarchy that has a regiondir and familyname. - Path outputDir = new Path(new Path(this.testDir, "regionname"), + // Make up a directory hierarchy that has a regiondir ("7e0102") and familyname. + Path outputDir = new Path(new Path(this.testDir, "7e0102"), "familyname"); StoreFile.Writer writer = new StoreFile.WriterBuilder(conf, cacheConf, this.fs, 2 * 1024) @@ -103,6 +114,10 @@ public void testBasicHalfMapFile() throws Exception { private void writeStoreFile(final StoreFile.Writer writer) throws IOException { writeStoreFile(writer, Bytes.toBytes(getName()), Bytes.toBytes(getName())); } + + // pick an split point (roughly halfway) + byte[] SPLITKEY = new byte[] { (LAST_CHAR + FIRST_CHAR)/2, FIRST_CHAR}; + /* * Writes HStoreKey and ImmutableBytes data to passed writer and * then closes it. @@ -131,12 +146,12 @@ public static void writeStoreFile(final StoreFile.Writer writer, byte[] fam, byt */ public void testReference() throws IOException { - Path storedir = new Path(new Path(this.testDir, "regionname"), "familyname"); - Path dir = new Path(storedir, "1234567890"); + // Make up a directory hierarchy that has a regiondir ("7e0102") and familyname. + Path storedir = new Path(new Path(this.testDir, "7e0102"), "familyname"); // Make a store file and write data to it. StoreFile.Writer writer = new StoreFile.WriterBuilder(conf, cacheConf, this.fs, 8 * 1024) - .withOutputDir(dir) + .withOutputDir(storedir) .build(); writeStoreFile(writer); StoreFile hsf = new StoreFile(this.fs, writer.getPath(), conf, cacheConf, @@ -150,7 +165,7 @@ public void testReference() kv = KeyValue.createKeyValueFromKey(reader.getLastKey()); byte [] finalRow = kv.getRow(); // Make a reference - Path refPath = StoreFile.split(fs, dir, hsf, midRow, Range.top); + Path refPath = StoreFile.split(fs, storedir, hsf, midRow, Range.top); StoreFile refHsf = new StoreFile(this.fs, refPath, conf, cacheConf, StoreFile.BloomType.NONE, NoOpDataBlockEncoder.INSTANCE); // Now confirm that I can read from the reference and that it only gets @@ -167,6 +182,150 @@ public void testReference() assertTrue(Bytes.equals(kv.getRow(), finalRow)); } + public void testHFileLink() throws IOException { + final String columnFamily = "f"; + + Configuration testConf = new Configuration(this.conf); + FSUtils.setRootDir(testConf, this.testDir); + + HRegionInfo hri = new HRegionInfo(Bytes.toBytes("table-link")); + Path storedir = new Path(new Path(this.testDir, + new Path(hri.getTableNameAsString(), hri.getEncodedName())), columnFamily); + + // Make a store file and write data to it. + StoreFile.Writer writer = new StoreFile.WriterBuilder(testConf, cacheConf, + this.fs, 8 * 1024) + .withOutputDir(storedir) + .build(); + Path storeFilePath = writer.getPath(); + writeStoreFile(writer); + writer.close(); + + Path dstPath = new Path(this.testDir, new Path("test-region", columnFamily)); + HFileLink.create(testConf, this.fs, dstPath, hri, storeFilePath.getName()); + Path linkFilePath = new Path(dstPath, + HFileLink.createHFileLinkName(hri, storeFilePath.getName())); + + // Try to open store file from link + StoreFile hsf = new StoreFile(this.fs, linkFilePath, testConf, cacheConf, + StoreFile.BloomType.NONE, NoOpDataBlockEncoder.INSTANCE); + assertTrue(hsf.isLink()); + + // Now confirm that I can read from the link + int count = 1; + HFileScanner s = hsf.createReader().getScanner(false, false); + s.seekTo(); + while (s.next()) { + count++; + } + assertEquals((LAST_CHAR - FIRST_CHAR + 1) * (LAST_CHAR - FIRST_CHAR + 1), count); + } + + /** + * Validate that we can handle valid tables with '.', '_', and '-' chars. + */ + public void testStoreFileNames() { + String[] legalHFileLink = { "MyTable_02=abc012-def345", "MyTable_02.300=abc012-def345", + "MyTable_02-400=abc012-def345", "MyTable_02-400.200=abc012-def345", + "MyTable_02=abc012-def345_SeqId_1_", "MyTable_02=abc012-def345_SeqId_20_" }; + for (String name: legalHFileLink) { + assertTrue("should be a valid link: " + name, HFileLink.isHFileLink(name)); + assertTrue("should be a valid StoreFile" + name, StoreFile.validateStoreFileName(name)); + assertFalse("should not be a valid reference: " + name, StoreFile.isReference(name)); + + String refName = name + ".6789"; + assertTrue("should be a valid link reference: " + refName, StoreFile.isReference(refName)); + assertTrue("should be a valid StoreFile" + refName, StoreFile.validateStoreFileName(refName)); + } + + String[] illegalHFileLink = { ".MyTable_02=abc012-def345", "-MyTable_02.300=abc012-def345", + "MyTable_02-400=abc0_12-def345", "MyTable_02-400.200=abc012-def345...." }; + for (String name: illegalHFileLink) { + assertFalse("should not be a valid link: " + name, HFileLink.isHFileLink(name)); + } + } + + /** + * This test creates an hfile and then the dir structures and files to verify that references + * to hfilelinks (created by snapshot clones) can be properly interpreted. + */ + public void testReferenceToHFileLink() throws IOException { + final String columnFamily = "f"; + + Path rootDir = FSUtils.getRootDir(conf); + + String tablename = "_original-evil-name"; // adding legal table name chars to verify regex handles it. + HRegionInfo hri = new HRegionInfo(Bytes.toBytes(tablename)); + // store dir = /// + Path storedir = new Path(new Path(rootDir, + new Path(hri.getTableNameAsString(), hri.getEncodedName())), columnFamily); + + // Make a store file and write data to it. //// + StoreFile.Writer writer = new StoreFile.WriterBuilder(conf, cacheConf, + this.fs, 8 * 1024) + .withOutputDir(storedir) + .build(); + Path storeFilePath = writer.getPath(); + writeStoreFile(writer); + writer.close(); + + // create link to store file. /clone/region//-- + String target = "clone"; + Path dstPath = new Path(rootDir, new Path(new Path(target, "7e0102"), columnFamily)); + HFileLink.create(conf, this.fs, dstPath, hri, storeFilePath.getName()); + Path linkFilePath = new Path(dstPath, + HFileLink.createHFileLinkName(hri, storeFilePath.getName())); + + // create splits of the link. + // /clone/splitA//, + // /clone/splitB// + Path splitDirA = new Path(new Path(rootDir, + new Path(target, "571A")), columnFamily); + Path splitDirB = new Path(new Path(rootDir, + new Path(target, "571B")), columnFamily); + StoreFile f = new StoreFile(fs, linkFilePath, conf, cacheConf, BloomType.NONE, + NoOpDataBlockEncoder.INSTANCE); + byte[] splitRow = SPLITKEY; + Path pathA = StoreFile.split(fs, splitDirA, f, splitRow, Range.top); // top + Path pathB = StoreFile.split(fs, splitDirB, f, splitRow, Range.bottom); // bottom + + // OK test the thing + FSUtils.logFileSystemState(fs, rootDir, LOG); + + // There is a case where a file with the hfilelink pattern is actually a daughter + // reference to a hfile link. This code in StoreFile that handles this case. + + // Try to open store file from link + StoreFile hsfA = new StoreFile(this.fs, pathA, conf, cacheConf, + StoreFile.BloomType.NONE, NoOpDataBlockEncoder.INSTANCE); + + // Now confirm that I can read from the ref to link + int count = 1; + HFileScanner s = hsfA.createReader().getScanner(false, false); + s.seekTo(); + while (s.next()) { + count++; + } + assertTrue(count > 0); // read some rows here + + // Try to open store file from link + StoreFile hsfB = new StoreFile(this.fs, pathB, conf, cacheConf, + StoreFile.BloomType.NONE, NoOpDataBlockEncoder.INSTANCE); + + // Now confirm that I can read from the ref to link + HFileScanner sB = hsfB.createReader().getScanner(false, false); + sB.seekTo(); + + //count++ as seekTo() will advance the scanner + count++; + while (sB.next()) { + count++; + } + + // read the rest of the rows + assertEquals((LAST_CHAR - FIRST_CHAR + 1) * (LAST_CHAR - FIRST_CHAR + 1), count); + } + private void checkHalfHFile(final StoreFile f) throws IOException { byte [] midkey = f.createReader().midkey(); @@ -248,20 +407,12 @@ private void checkHalfHFile(final StoreFile f) topPath = StoreFile.split(this.fs, topDir, f, badmidkey, Range.top); bottomPath = StoreFile.split(this.fs, bottomDir, f, badmidkey, Range.bottom); + + assertNull(bottomPath); + top = new StoreFile(this.fs, topPath, conf, cacheConf, StoreFile.BloomType.NONE, NoOpDataBlockEncoder.INSTANCE).createReader(); - bottom = new StoreFile(this.fs, bottomPath, conf, cacheConf, - StoreFile.BloomType.NONE, - NoOpDataBlockEncoder.INSTANCE).createReader(); - bottomScanner = bottom.getScanner(false, false); - int count = 0; - while ((!bottomScanner.isSeeked() && bottomScanner.seekTo()) || - bottomScanner.next()) { - count++; - } - // When badkey is < than the bottom, should return no values. - assertTrue(count == 0); // Now read from the top. first = true; topScanner = top.getScanner(false, false); @@ -288,16 +439,15 @@ private void checkHalfHFile(final StoreFile f) } // Remove references. this.fs.delete(topPath, false); - this.fs.delete(bottomPath, false); // Test when badkey is > than last key in file ('||' > 'zz'). badmidkey = Bytes.toBytes("|||"); topPath = StoreFile.split(this.fs, topDir, f, badmidkey, Range.top); bottomPath = StoreFile.split(this.fs, bottomDir, f, badmidkey, Range.bottom); - top = new StoreFile(this.fs, topPath, conf, cacheConf, - StoreFile.BloomType.NONE, - NoOpDataBlockEncoder.INSTANCE).createReader(); + + assertNull(topPath); + bottom = new StoreFile(this.fs, bottomPath, conf, cacheConf, StoreFile.BloomType.NONE, NoOpDataBlockEncoder.INSTANCE).createReader(); @@ -321,14 +471,6 @@ private void checkHalfHFile(final StoreFile f) for (int i = 0; i < tmp.length(); i++) { assertTrue(Bytes.toString(keyKV.getRow()).charAt(i) == 'z'); } - count = 0; - topScanner = top.getScanner(false, false); - while ((!topScanner.isSeeked() && topScanner.seekTo()) || - (topScanner.isSeeked() && topScanner.next())) { - count++; - } - // When badkey is < than the bottom, should return no values. - assertTrue(count == 0); } finally { if (top != null) { top.close(true); // evict since we are about to delete the file @@ -367,7 +509,7 @@ private void bloomWriteRead(StoreFile.Writer writer, FileSystem fs) int falseNeg = 0; for (int i = 0; i < 2000; i++) { String row = String.format(localFormatter, i); - TreeSet columns = new TreeSet(); + TreeSet columns = new TreeSet(Bytes.BYTES_COMPARATOR); columns.add("family:col".getBytes()); Scan scan = new Scan(row.getBytes(),row.getBytes()); @@ -401,6 +543,8 @@ public void testBloomFilter() throws Exception { .withFilePath(f) .withBloomType(StoreFile.BloomType.ROW) .withMaxKeyCount(2000) + .withChecksumType(CKTYPE) + .withBytesPerChecksum(CKBYTES) .build(); bloomWriteRead(writer, fs); } @@ -420,6 +564,8 @@ public void testDeleteFamilyBloomFilter() throws Exception { fs, StoreFile.DEFAULT_BLOCKSIZE_SMALL) .withFilePath(f) .withMaxKeyCount(2000) + .withChecksumType(CKTYPE) + .withBytesPerChecksum(CKBYTES) .build(); // add delete family @@ -462,6 +608,33 @@ public void testDeleteFamilyBloomFilter() throws Exception { + ", expected no more than " + maxFalsePos, falsePos <= maxFalsePos); } + /** + * Test for HBASE-8012 + */ + public void testReseek() throws Exception { + // write the file + Path f = new Path(ROOT_DIR, getName()); + + // Make a store file and write data to it. + StoreFile.Writer writer = new StoreFile.WriterBuilder(conf, cacheConf, + this.fs, 8 * 1024) + .withFilePath(f) + .build(); + + writeStoreFile(writer); + writer.close(); + + StoreFile.Reader reader = new StoreFile.Reader(fs, f, cacheConf, DataBlockEncoding.NONE); + + // Now do reseek with empty KV to position to the beginning of the file + + KeyValue k = KeyValue.createFirstOnRow(HConstants.EMPTY_BYTE_ARRAY); + StoreFileScanner s = reader.getStoreFileScanner(false, false); + s.reseek(k); + + assertNotNull("Intial reseek should position at the beginning of the file", s.peek()); + } + public void testBloomTypes() throws Exception { float err = (float) 0.01; FileSystem fs = FileSystem.getLocal(conf); @@ -490,6 +663,8 @@ public void testBloomTypes() throws Exception { .withFilePath(f) .withBloomType(bt[x]) .withMaxKeyCount(expKeys[x]) + .withChecksumType(CKTYPE) + .withBytesPerChecksum(CKBYTES) .build(); long now = System.currentTimeMillis(); @@ -521,7 +696,7 @@ public void testBloomTypes() throws Exception { for (int j = 0; j < colCount*2; ++j) { // column qualifiers String row = String.format(localFormatter, i); String col = String.format(localFormatter, j); - TreeSet columns = new TreeSet(); + TreeSet columns = new TreeSet(Bytes.BYTES_COMPARATOR); columns.add(("col" + col).getBytes()); Scan scan = new Scan(row.getBytes(),row.getBytes()); @@ -565,6 +740,8 @@ public void testBloomEdgeCases() throws Exception { .withFilePath(f) .withBloomType(StoreFile.BloomType.ROW) .withMaxKeyCount(2000) + .withChecksumType(CKTYPE) + .withBytesPerChecksum(CKBYTES) .build(); assertFalse(writer.hasGeneralBloom()); writer.close(); @@ -592,14 +769,16 @@ public void testBloomEdgeCases() throws Exception { .withFilePath(f) .withBloomType(StoreFile.BloomType.ROW) .withMaxKeyCount(Integer.MAX_VALUE) + .withChecksumType(CKTYPE) + .withBytesPerChecksum(CKBYTES) .build(); assertFalse(writer.hasGeneralBloom()); writer.close(); fs.delete(f, true); } - public void testFlushTimeComparator() { - assertOrdering(StoreFile.Comparators.FLUSH_TIME, + public void testSeqIdComparator() { + assertOrdering(StoreFile.Comparators.SEQ_ID, mockStoreFile(true, 1000, -1, "/foo/123"), mockStoreFile(true, 1000, -1, "/foo/126"), mockStoreFile(true, 2000, -1, "/foo/126"), @@ -630,13 +809,7 @@ private StoreFile mockStoreFile(boolean bulkLoad, long bulkTimestamp, StoreFile mock = Mockito.mock(StoreFile.class); Mockito.doReturn(bulkLoad).when(mock).isBulkLoadResult(); Mockito.doReturn(bulkTimestamp).when(mock).getBulkLoadTimestamp(); - if (bulkLoad) { - // Bulk load files will throw if you ask for their sequence ID - Mockito.doThrow(new IllegalAccessError("bulk load")) - .when(mock).getMaxSequenceId(); - } else { - Mockito.doReturn(seqId).when(mock).getMaxSequenceId(); - } + Mockito.doReturn(seqId).when(mock).getMaxSequenceId(); Mockito.doReturn(new Path(path)).when(mock).getPath(); String name = "mock storefile, bulkLoad=" + bulkLoad + " bulkTimestamp=" + bulkTimestamp + @@ -680,8 +853,8 @@ public void testMultipleTimestamps() throws IOException { long[] timestamps = new long[] {20,10,5,1}; Scan scan = new Scan(); - Path storedir = new Path(new Path(this.testDir, "regionname"), - "familyname"); + // Make up a directory hierarchy that has a regiondir ("7e0102") and familyname. + Path storedir = new Path(new Path(this.testDir, "7e0102"), "familyname"); Path dir = new Path(storedir, "1234567890"); StoreFile.Writer writer = new StoreFile.WriterBuilder(conf, cacheConf, this.fs, 8 * 1024) @@ -701,7 +874,7 @@ public void testMultipleTimestamps() throws IOException { StoreFile.BloomType.NONE, NoOpDataBlockEncoder.INSTANCE); StoreFile.Reader reader = hsf.createReader(); StoreFileScanner scanner = reader.getStoreFileScanner(false, false); - TreeSet columns = new TreeSet(); + TreeSet columns = new TreeSet(Bytes.BYTES_COMPARATOR); columns.add(qualifier); scan.setTimeRange(20, 100); @@ -724,8 +897,8 @@ public void testMultipleTimestamps() throws IOException { public void testCacheOnWriteEvictOnClose() throws Exception { Configuration conf = this.conf; - // Find a home for our files - Path baseDir = new Path(new Path(this.testDir, "regionname"),"twoCOWEOC"); + // Find a home for our files (regiondir ("7e0102") and familyname). + Path baseDir = new Path(new Path(this.testDir, "7e0102"),"twoCOWEOC"); // Grab the block cache and get the initial hit/miss counts BlockCache bc = new CacheConfig(conf).getBlockCache(); @@ -796,7 +969,7 @@ public void testCacheOnWriteEvictOnClose() throws Exception { kv2 = scannerTwo.next(); assertTrue(kv1.equals(kv2)); assertTrue(Bytes.compareTo( - kv1.getBuffer(), kv1.getKeyOffset(), kv1.getKeyLength(), + kv1.getBuffer(), kv1.getKeyOffset(), kv1.getKeyLength(), kv2.getBuffer(), kv2.getKeyOffset(), kv2.getKeyLength()) == 0); assertTrue(Bytes.compareTo( kv1.getBuffer(), kv1.getValueOffset(), kv1.getValueLength(), @@ -859,6 +1032,8 @@ private StoreFile.Writer writeStoreFile(Configuration conf, blockSize) .withFilePath(path) .withMaxKeyCount(2000) + .withChecksumType(CKTYPE) + .withBytesPerChecksum(CKBYTES) .build(); // We'll write N-1 KVs to ensure we don't write an extra block kvs.remove(kvs.size()-1); @@ -875,7 +1050,8 @@ private StoreFile.Writer writeStoreFile(Configuration conf, * file info. */ public void testDataBlockEncodingMetaData() throws IOException { - Path dir = new Path(new Path(this.testDir, "regionname"), "familyname"); + // Make up a directory hierarchy that has a regiondir ("7e0102") and familyname. + Path dir = new Path(new Path(this.testDir, "7e0102"), "familyname"); Path path = new Path(dir, "1234567890"); DataBlockEncoding dataBlockEncoderAlgo = @@ -890,15 +1066,17 @@ public void testDataBlockEncodingMetaData() throws IOException { .withFilePath(path) .withDataBlockEncoder(dataBlockEncoder) .withMaxKeyCount(2000) + .withChecksumType(CKTYPE) + .withBytesPerChecksum(CKBYTES) .build(); writer.close(); - + StoreFile storeFile = new StoreFile(fs, writer.getPath(), conf, cacheConf, BloomType.NONE, dataBlockEncoder); StoreFile.Reader reader = storeFile.createReader(); - + Map fileInfo = reader.loadFileInfo(); - byte[] value = fileInfo.get(StoreFile.DATA_BLOCK_ENCODING); + byte[] value = fileInfo.get(HFileDataBlockEncoder.DATA_BLOCK_ENCODING); assertEquals(dataBlockEncoderAlgo.getNameInBytes(), value); } diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java index 0ffb8113b7af..3cf81fd995e1 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java index 3c582338e69b..f0b2688b4c04 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -38,7 +37,6 @@ import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.regionserver.Store.ScanInfo; -import org.apache.hadoop.hbase.regionserver.StoreScanner.ScanType; import org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.EnvironmentEdge; @@ -559,7 +557,7 @@ public long currentTimeMillis() { KeyValue.COMPARATOR); StoreScanner scanner = new StoreScanner(scan, scanInfo, - StoreScanner.ScanType.MAJOR_COMPACT, null, scanners, + ScanType.MAJOR_COMPACT, null, scanners, HConstants.OLDEST_TIMESTAMP); List results = new ArrayList(); results = new ArrayList(); diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java b/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java index 09d3151cd212..82033c12ebac 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java b/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java index e205acb403e0..10ccbacd5e22 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java @@ -99,28 +99,32 @@ public void setupHRI() { HRegion region = HRegion.createHRegion(hri, HTU.getDataTestDir(), HTU.getConfiguration(), htd); - assertNotNull(region); - // Spy on the region so can throw exception when close is called. - HRegion spy = Mockito.spy(region); - final boolean abort = false; - Mockito.when(spy.close(abort)). - thenThrow(new RuntimeException("Mocked failed close!")); - // The CloseRegionHandler will try to get an HRegion that corresponds - // to the passed hri -- so insert the region into the online region Set. - rss.addToOnlineRegions(spy); - // Assert the Server is NOT stopped before we call close region. - assertFalse(server.isStopped()); - CloseRegionHandler handler = - new CloseRegionHandler(server, rss, hri, false, false, -1); - boolean throwable = false; try { - handler.process(); - } catch (Throwable t) { - throwable = true; + assertNotNull(region); + // Spy on the region so can throw exception when close is called. + HRegion spy = Mockito.spy(region); + final boolean abort = false; + Mockito.when(spy.close(abort)). + thenThrow(new RuntimeException("Mocked failed close!")); + // The CloseRegionHandler will try to get an HRegion that corresponds + // to the passed hri -- so insert the region into the online region Set. + rss.addToOnlineRegions(spy); + // Assert the Server is NOT stopped before we call close region. + assertFalse(server.isStopped()); + CloseRegionHandler handler = + new CloseRegionHandler(server, rss, hri, false, false, -1); + boolean throwable = false; + try { + handler.process(); + } catch (Throwable t) { + throwable = true; + } finally { + assertTrue(throwable); + // Abort calls stop so stopped flag should be set. + assertTrue(server.isStopped()); + } } finally { - assertTrue(throwable); - // Abort calls stop so stopped flag should be set. - assertTrue(server.isStopped()); + HRegion.closeHRegion(region); } } @@ -133,7 +137,8 @@ public void setupHRI() { @Test public void testZKClosingNodeVersionMismatch() throws IOException, NodeExistsException, KeeperException { final Server server = new MockServer(HTU); - final RegionServerServices rss = new MockRegionServerServices(); + final MockRegionServerServices rss = new MockRegionServerServices(); + rss.setFileSystem(HTU.getTestFileSystem()); HTableDescriptor htd = TEST_HTD; final HRegionInfo hri = TEST_HRI; @@ -169,7 +174,8 @@ public void setupHRI() { @Test public void testCloseRegion() throws IOException, NodeExistsException, KeeperException { final Server server = new MockServer(HTU); - final RegionServerServices rss = new MockRegionServerServices(); + final MockRegionServerServices rss = new MockRegionServerServices(); + rss.setFileSystem(HTU.getTestFileSystem()); HTableDescriptor htd = TEST_HTD; HRegionInfo hri = TEST_HRI; @@ -198,18 +204,18 @@ private void OpenRegion(Server server, RegionServerServices rss, HTableDescriptor htd, HRegionInfo hri) throws IOException, NodeExistsException, KeeperException { // Create it OFFLINE node, which is what Master set before sending OPEN RPC - ZKAssign.createNodeOffline(server.getZooKeeper(), hri, - server.getServerName()); - OpenRegionHandler openHandler = new OpenRegionHandler(server, rss, hri, - htd); - openHandler.process(); - RegionTransitionData data = - ZKAssign.getData(server.getZooKeeper(), hri.getEncodedName()); - - // delete the node, which is what Master do after the region is opened - ZKAssign.deleteNode(server.getZooKeeper(), hri.getEncodedName(), - EventType.RS_ZK_REGION_OPENED); - } + + + ZKAssign.createNodeOffline(server.getZooKeeper(), hri, server.getServerName()); + int version = ZKAssign.transitionNodeOpening(server.getZooKeeper(), hri, server.getServerName()); + OpenRegionHandler openHandler = new OpenRegionHandler(server, rss, hri, htd, version); + openHandler.process(); + RegionTransitionData data = ZKAssign.getData(server.getZooKeeper(), hri.getEncodedName()); + + // delete the node, which is what Master do after the region is opened + ZKAssign.deleteNode(server.getZooKeeper(), hri.getEncodedName(), + EventType.RS_ZK_REGION_OPENED); + } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java b/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java index 8988f83775e3..1648f4fcfebf 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,16 +18,26 @@ */ package org.apache.hadoop.hbase.regionserver.handler; -import static org.junit.Assert.*; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; import java.io.IOException; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.executor.RegionTransitionData; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.Server; import org.apache.hadoop.hbase.executor.EventHandler.EventType; +import org.apache.hadoop.hbase.executor.RegionTransitionData; import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException; import org.apache.hadoop.hbase.regionserver.RegionServerServices; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.MockRegionServerServices; @@ -57,13 +66,20 @@ public class TestOpenRegionHandler { private int testIndex = 0; @BeforeClass public static void before() throws Exception { - HTU.startMiniZKCluster(); + Configuration c = HTU.getConfiguration(); + c.setClass(HConstants.REGION_SERVER_IMPL, TestOpenRegionHandlerRegionServer.class, + HRegionServer.class); + HTU.startMiniCluster(); TEST_HTD = new HTableDescriptor("TestOpenRegionHandler.java"); } @AfterClass public static void after() throws IOException { TEST_HTD = null; - HTU.shutdownMiniZKCluster(); + try { + HTU.shutdownMiniCluster(); + } catch (Exception e) { + throw new IOException(e); + } } /** @@ -98,30 +114,34 @@ public void setupHRI() { HRegion.createHRegion(hri, HTU.getDataTestDir(), HTU .getConfiguration(), htd); assertNotNull(region); - OpenRegionHandler handler = new OpenRegionHandler(server, rss, hri, htd) { - HRegion openRegion() { - // Open region first, then remove znode as though it'd been hijacked. - HRegion region = super.openRegion(); - - // Don't actually open region BUT remove the znode as though it'd - // been hijacked on us. - ZooKeeperWatcher zkw = this.server.getZooKeeper(); - String node = ZKAssign.getNodeName(zkw, hri.getEncodedName()); - try { - ZKUtil.deleteNodeFailSilent(zkw, node); - } catch (KeeperException e) { - throw new RuntimeException("Ugh failed delete of " + node, e); + try { + OpenRegionHandler handler = new OpenRegionHandler(server, rss, hri, htd) { + HRegion openRegion() { + // Open region first, then remove znode as though it'd been hijacked. + HRegion region = super.openRegion(); + + // Don't actually open region BUT remove the znode as though it'd + // been hijacked on us. + ZooKeeperWatcher zkw = this.server.getZooKeeper(); + String node = ZKAssign.getNodeName(zkw, hri.getEncodedName()); + try { + ZKUtil.deleteNodeFailSilent(zkw, node); + } catch (KeeperException e) { + throw new RuntimeException("Ugh failed delete of " + node, e); + } + return region; } - return region; - } - }; - // Call process without first creating OFFLINE region in zk, see if - // exception or just quiet return (expected). - handler.process(); - ZKAssign.createNodeOffline(server.getZooKeeper(), hri, server.getServerName()); - // Call process again but this time yank the zk znode out from under it - // post OPENING; again will expect it to come back w/o NPE or exception. - handler.process(); + }; + // Call process without first creating OFFLINE region in zk, see if + // exception or just quiet return (expected). + handler.process(); + ZKAssign.createNodeOffline(server.getZooKeeper(), hri, server.getServerName()); + // Call process again but this time yank the zk znode out from under it + // post OPENING; again will expect it to come back w/o NPE or exception. + handler.process(); + } finally { + HRegion.closeHRegion(region); + } } @Test @@ -131,6 +151,7 @@ public void testFailedOpenRegion() throws Exception { // Create it OFFLINE, which is what it expects ZKAssign.createNodeOffline(server.getZooKeeper(), TEST_HRI, server.getServerName()); + ZKAssign.transitionNodeOpening(server.getZooKeeper(), TEST_HRI, server.getServerName()); // Create the handler OpenRegionHandler handler = @@ -156,7 +177,7 @@ public void testFailedUpdateMeta() throws Exception { // Create it OFFLINE, which is what it expects ZKAssign.createNodeOffline(server.getZooKeeper(), TEST_HRI, server.getServerName()); - + ZKAssign.transitionNodeOpening(server.getZooKeeper(), TEST_HRI, server.getServerName()); // Create the handler OpenRegionHandler handler = new OpenRegionHandler(server, rsServices, TEST_HRI, TEST_HTD) { @@ -174,6 +195,49 @@ boolean updateMeta(final HRegion r) { assertEquals(EventType.RS_ZK_REGION_FAILED_OPEN, data.getEventType()); } + public static class TestOpenRegionHandlerRegionServer extends HRegionServer { + public TestOpenRegionHandlerRegionServer(Configuration conf) + throws IOException, InterruptedException { + super(conf); + } + @Override + public boolean addRegionsInTransition(HRegionInfo region, + String currentAction) throws RegionAlreadyInTransitionException { + return super.addRegionsInTransition(region, currentAction); + } + } + + @Test + public void testTransitionToFailedOpenEvenIfCleanupFails() throws Exception { + MiniHBaseCluster cluster = HTU.getHBaseCluster(); + HRegionServer server = + cluster.getLiveRegionServerThreads().get(0).getRegionServer(); + // Create it OFFLINE, which is what it expects + ZKAssign.createNodeOffline(server.getZooKeeper(), TEST_HRI, server.getServerName()); + ZKAssign.transitionNodeOpening(server.getZooKeeper(), TEST_HRI, server.getServerName()); + // Create the handler + OpenRegionHandler handler = new OpenRegionHandler(server, server, TEST_HRI, TEST_HTD) { + @Override + boolean updateMeta(HRegion r) { + return false; + }; + + @Override + void cleanupFailedOpen(HRegion region) throws IOException { + throw new IOException("FileSystem got closed."); + } + }; + ((TestOpenRegionHandlerRegionServer)server).addRegionsInTransition(TEST_HRI, "OPEN"); + try { + handler.process(); + } catch (Exception e) { + // Ignore the IOException that we have thrown from cleanupFailedOpen + } + RegionTransitionData data = + ZKAssign.getData(server.getZooKeeper(), TEST_HRI.getEncodedName()); + assertEquals(EventType.RS_ZK_REGION_FAILED_OPEN, data.getEventType()); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultySequenceFileLogReader.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultySequenceFileLogReader.java index 16db1675ae1f..f926952225c7 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultySequenceFileLogReader.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultySequenceFileLogReader.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -39,6 +38,8 @@ FailureType getFailureType() { return FailureType.valueOf(conf.get("faultysequencefilelogreader.failuretype", "NONE")); } + WALEditCodec codec = new WALEditCodec(); + @Override public HLog.Entry next(HLog.Entry reuse) throws IOException { this.entryStart = this.reader.getPosition(); @@ -49,6 +50,11 @@ public HLog.Entry next(HLog.Entry reuse) throws IOException { HLogKey key = HLog.newKey(conf); WALEdit val = new WALEdit(); HLog.Entry e = new HLog.Entry(key, val); + codec.setCompression(compressionContext); + e.getEdit().setCodec(codec); + if (compressionContext != null) { + e.getKey().setCompressionContext(compressionContext); + } b = this.reader.next(e.getKey(), e.getEdit()); nextQueue.offer(e); numberOfFileEntries++; diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java new file mode 100644 index 000000000000..e534cc28a752 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java @@ -0,0 +1,361 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.regionserver.wal; + +import java.util.Map; +import java.util.List; +import java.util.Random; +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; + +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; +import org.apache.hadoop.conf.Configured; + +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.wal.HLog.Entry; + +/** + * This class runs performance benchmarks for {@link HLog}. + * See usage for this tool by running: + * $ hbase org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation -h + */ +public final class HLogPerformanceEvaluation extends Configured implements Tool { + static final Log LOG = LogFactory.getLog(HLogPerformanceEvaluation.class.getName()); + + private final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + static final String TABLE_NAME = "HLogPerformanceEvaluation"; + static final String QUALIFIER_PREFIX = "q"; + static final String FAMILY_PREFIX = "cf"; + + private int numQualifiers = 1; + private int valueSize = 512; + private int keySize = 16; + + /** + * Perform HLog.append() of Put object, for the number of iterations requested. + * Keys and Vaues are generated randomly, the number of column familes, + * qualifiers and key/value size is tunable by the user. + */ + class HLogPutBenchmark implements Runnable { + private final long numIterations; + private final int numFamilies; + private final boolean noSync; + private final HRegion region; + private final HTableDescriptor htd; + + HLogPutBenchmark(final HRegion region, final HTableDescriptor htd, + final long numIterations, final boolean noSync) { + this.numIterations = numIterations; + this.noSync = noSync; + this.numFamilies = htd.getColumnFamilies().length; + this.region = region; + this.htd = htd; + } + + public void run() { + byte[] key = new byte[keySize]; + byte[] value = new byte[valueSize]; + Random rand = new Random(Thread.currentThread().getId()); + HLog hlog = region.getLog(); + + try { + long startTime = System.currentTimeMillis(); + for (int i = 0; i < numIterations; ++i) { + Put put = setupPut(rand, key, value, numFamilies); + long now = System.currentTimeMillis(); + WALEdit walEdit = new WALEdit(); + addFamilyMapToWALEdit(put.getFamilyMap(), walEdit); + HRegionInfo hri = region.getRegionInfo(); + if (this.noSync) { + hlog.appendNoSync(hri, hri.getTableName(), walEdit, + HConstants.DEFAULT_CLUSTER_ID, now, htd); + } else { + hlog.append(hri, hri.getTableName(), walEdit, now, htd); + } + } + long totalTime = (System.currentTimeMillis() - startTime); + logBenchmarkResult(Thread.currentThread().getName(), numIterations, totalTime); + } catch (Exception e) { + LOG.error(getClass().getSimpleName() + " Thread failed", e); + } + } + } + + @Override + public int run(String[] args) throws Exception { + Path rootRegionDir = null; + int numThreads = 1; + long numIterations = 10000; + int numFamilies = 1; + boolean noSync = false; + boolean verify = false; + boolean verbose = false; + long roll = Long.MAX_VALUE; + // Process command line args + for (int i = 0; i < args.length; i++) { + String cmd = args[i]; + try { + if (cmd.equals("-threads")) { + numThreads = Integer.parseInt(args[++i]); + } else if (cmd.equals("-iterations")) { + numIterations = Long.parseLong(args[++i]); + } else if (cmd.equals("-path")) { + rootRegionDir = new Path(args[++i]); + } else if (cmd.equals("-families")) { + numFamilies = Integer.parseInt(args[++i]); + } else if (cmd.equals("-qualifiers")) { + numQualifiers = Integer.parseInt(args[++i]); + } else if (cmd.equals("-keySize")) { + keySize = Integer.parseInt(args[++i]); + } else if (cmd.equals("-valueSize")) { + valueSize = Integer.parseInt(args[++i]); + } else if (cmd.equals("-nosync")) { + noSync = true; + } else if (cmd.equals("-verify")) { + verify = true; + } else if (cmd.equals("-verbose")) { + verbose = true; + } else if (cmd.equals("-roll")) { + roll = Long.parseLong(args[++i]); + } else if (cmd.equals("-h")) { + printUsageAndExit(); + } else if (cmd.equals("--help")) { + printUsageAndExit(); + } else { + System.err.println("UNEXPECTED: " + cmd); + printUsageAndExit(); + } + } catch (Exception e) { + printUsageAndExit(); + } + } + + // Run HLog Performance Evaluation + FileSystem fs = FileSystem.get(getConf()); + LOG.info("" + fs); + try { + if (rootRegionDir == null) { + rootRegionDir = TEST_UTIL.getDataTestDir("HLogPerformanceEvaluation"); + } + rootRegionDir = rootRegionDir.makeQualified(fs); + cleanRegionRootDir(fs, rootRegionDir); + // Initialize Table Descriptor + HTableDescriptor htd = createHTableDescriptor(numFamilies); + final long whenToRoll = roll; + HLog hlog = new HLog(fs, new Path(rootRegionDir, "wals"), + new Path(rootRegionDir, "old.wals"), getConf()) { + int appends = 0; + protected void doWrite(HRegionInfo info, HLogKey logKey, WALEdit logEdit, + HTableDescriptor htd) + throws IOException { + this.appends++; + if (this.appends % whenToRoll == 0) { + LOG.info("Rolling after " + appends + " edits"); + rollWriter(); + } + super.doWrite(info, logKey, logEdit, htd); + }; + }; + hlog.rollWriter(); + HRegion region = null; + try { + region = openRegion(fs, rootRegionDir, htd, hlog); + long putTime = runBenchmark(new HLogPutBenchmark(region, htd, numIterations, noSync), numThreads); + logBenchmarkResult("Summary: threads=" + numThreads + ", iterations=" + numIterations, + numIterations * numThreads, putTime); + if (region != null) { + closeRegion(region); + region = null; + } + if (verify) { + Path dir = hlog.getDir(); + long editCount = 0; + for (FileStatus fss: fs.listStatus(dir)) { + editCount += verify(fss.getPath(), verbose); + } + long expected = numIterations * numThreads; + if (editCount != expected) { + throw new IllegalStateException("Counted=" + editCount + ", expected=" + expected); + } + } + } finally { + if (region != null) closeRegion(region); + // Remove the root dir for this test region + cleanRegionRootDir(fs, rootRegionDir); + } + } finally { + fs.close(); + } + + return(0); + } + + private static HTableDescriptor createHTableDescriptor(final int numFamilies) { + HTableDescriptor htd = new HTableDescriptor(TABLE_NAME); + for (int i = 0; i < numFamilies; ++i) { + HColumnDescriptor colDef = new HColumnDescriptor(FAMILY_PREFIX + i); + htd.addFamily(colDef); + } + return htd; + } + + /** + * Verify the content of the WAL file. + * Verify that sequenceids are ascending and that the file has expected number + * of edits. + * @param wal + * @return Count of edits. + * @throws IOException + */ + private long verify(final Path wal, final boolean verbose) throws IOException { + HLog.Reader reader = HLog.getReader(wal.getFileSystem(getConf()), wal, getConf()); + long previousSeqid = -1; + long count = 0; + try { + while (true) { + Entry e = reader.next(); + if (e == null) break; + count++; + long seqid = e.getKey().getLogSeqNum(); + if (verbose) LOG.info("seqid=" + seqid); + if (previousSeqid >= seqid) { + throw new IllegalStateException("wal=" + wal.getName() + + ", previousSeqid=" + previousSeqid + ", seqid=" + seqid); + } + previousSeqid = seqid; + } + } finally { + reader.close(); + } + return count; + } + + private static void logBenchmarkResult(String testName, long numTests, long totalTime) { + float tsec = totalTime / 1000.0f; + LOG.info(String.format("%s took %.3fs %.3fops/s", testName, tsec, numTests / tsec)); + } + + private void printUsageAndExit() { + System.err.printf("Usage: bin/hbase %s [options]\n", getClass().getName()); + System.err.println(" where [options] are:"); + System.err.println(" -h|-help Show this help and exit."); + System.err.println(" -threads Number of threads writing on the WAL."); + System.err.println(" -iterations Number of iterations per thread."); + System.err.println(" -path Path where region's root directory is created."); + System.err.println(" -families Number of column families to write."); + System.err.println(" -qualifiers Number of qualifiers to write."); + System.err.println(" -keySize Row key size in byte."); + System.err.println(" -valueSize Row/Col value size in byte."); + System.err.println(" -nosync Append without syncing"); + System.err.println(" -verify Verify edits written in sequence"); + System.err.println(" -verbose Output extra info; e.g. all edit seq ids when verifying"); + System.err.println(" -roll Roll the way every N appends"); + System.err.println(""); + System.err.println("Examples:"); + System.err.println(""); + System.err.println(" To run 100 threads on hdfs with log rolling every 10k edits and verification afterward do:"); + System.err.println(" $ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation \\"); + System.err.println(" -conf ./core-site.xml -path hdfs://example.org:7000/tmp -threads 100 -roll 10000 -verify"); + System.exit(1); + } + + private HRegion openRegion(final FileSystem fs, final Path dir, final HTableDescriptor htd, final HLog hlog) + throws IOException { + // Initialize HRegion + HRegionInfo regionInfo = new HRegionInfo(htd.getName()); + return HRegion.createHRegion(regionInfo, dir, getConf(), htd, hlog); + } + + private void closeRegion(final HRegion region) throws IOException { + if (region != null) { + region.close(); + HLog wal = region.getLog(); + if (wal != null) wal.close(); + } + } + + private void cleanRegionRootDir(final FileSystem fs, final Path dir) throws IOException { + if (fs.exists(dir)) { + fs.delete(dir, true); + } + } + + private Put setupPut(Random rand, byte[] key, byte[] value, final int numFamilies) { + rand.nextBytes(key); + Put put = new Put(key); + for (int cf = 0; cf < numFamilies; ++cf) { + for (int q = 0; q < numQualifiers; ++q) { + rand.nextBytes(value); + put.add(Bytes.toBytes(FAMILY_PREFIX + cf), Bytes.toBytes(QUALIFIER_PREFIX + q), value); + } + } + return put; + } + + private void addFamilyMapToWALEdit(Map> familyMap, WALEdit walEdit) { + for (List edits : familyMap.values()) { + for (KeyValue kv : edits) { + walEdit.add(kv); + } + } + } + + private long runBenchmark(Runnable runnable, final int numThreads) throws InterruptedException { + Thread[] threads = new Thread[numThreads]; + long startTime = System.currentTimeMillis(); + for (int i = 0; i < numThreads; ++i) { + threads[i] = new Thread(runnable); + threads[i].start(); + } + for (Thread t : threads) t.join(); + long endTime = System.currentTimeMillis(); + return(endTime - startTime); + } + + /** + * The guts of the {@link #main} method. + * Call this method to avoid the {@link #main(String[])} System.exit. + * @param args + * @return errCode + * @throws Exception + */ + static int innerMain(final String [] args) throws Exception { + return ToolRunner.run(HBaseConfiguration.create(), new HLogPerformanceEvaluation(), args); + } + + public static void main(String[] args) throws Exception { + System.exit(innerMain(args)); + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java index 33a6b6b5d16b..f56c2156e1ed 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,9 +18,6 @@ */ package org.apache.hadoop.hbase.regionserver.wal; -import org.apache.hadoop.hbase.HBaseTestingUtility; -import org.apache.hadoop.hbase.HConstants; - /** * An Utility testcase that returns the number of log files that * were rolled to be accessed from outside packages. diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedSequenceFileLogWriter.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedSequenceFileLogWriter.java index bf9bfc42d679..1e669a4cf356 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedSequenceFileLogWriter.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedSequenceFileLogWriter.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestCompressor.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestCompressor.java new file mode 100644 index 000000000000..dad681d85d42 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestCompressor.java @@ -0,0 +1,87 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License + */ +package org.apache.hadoop.hbase.regionserver.wal; + + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.DataInputStream; +import java.io.DataOutputStream; +import java.io.IOException; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test our compressor class. + */ +@Category(SmallTests.class) +public class TestCompressor { + @BeforeClass + public static void setUpBeforeClass() throws Exception { + } + + @Test + public void testToShort() { + short s = 1; + assertEquals(s, Compressor.toShort((byte)0, (byte)1)); + s <<= 8; + assertEquals(s, Compressor.toShort((byte)1, (byte)0)); + } + + @Test (expected = IllegalArgumentException.class) + public void testNegativeToShort() { + Compressor.toShort((byte)0xff, (byte)0xff); + } + + @Test + public void testCompressingWithNullDictionaries() throws IOException { + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + DataOutputStream dos = new DataOutputStream(baos); + byte [] blahBytes = Bytes.toBytes("blah"); + Compressor.writeCompressed(blahBytes, 0, blahBytes.length, dos, null); + dos.close(); + byte [] dosbytes = baos.toByteArray(); + DataInputStream dis = + new DataInputStream(new ByteArrayInputStream(dosbytes)); + byte [] product = Compressor.readCompressed(dis, null); + assertTrue(Bytes.equals(blahBytes, product)); + } + + @Test + public void testCompressingWithClearDictionaries() throws IOException { + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + DataOutputStream dos = new DataOutputStream(baos); + Dictionary dictionary = new LRUDictionary(); + byte [] blahBytes = Bytes.toBytes("blah"); + Compressor.writeCompressed(blahBytes, 0, blahBytes.length, dos, dictionary); + dos.close(); + byte [] dosbytes = baos.toByteArray(); + DataInputStream dis = + new DataInputStream(new ByteArrayInputStream(dosbytes)); + dictionary = new LRUDictionary(); + byte [] product = Compressor.readCompressed(dis, dictionary); + assertTrue(Bytes.equals(blahBytes, product)); + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestCustomWALEditCodec.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestCustomWALEditCodec.java new file mode 100644 index 000000000000..dbc440d5451d --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestCustomWALEditCodec.java @@ -0,0 +1,61 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver.wal; + +import static org.junit.Assert.assertTrue; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test that we can create, load, setup our own custom codec + */ +@Category(SmallTests.class) +public class TestCustomWALEditCodec { + + public static class CustomWALEditCodec extends WALEditCodec { + public boolean initialized = false; + public boolean compressionSet = false; + + @Override + public void init(Configuration conf) { + this.initialized = true; + } + + @Override + public void setCompression(CompressionContext compression) { + this.compressionSet = true; + } + } + + /** + * Test that a custom WALEditCodec will be completely setup when it is instantiated via + * {@link WALEditCodec} + * @throws Exception on failure + */ + @Test + public void testCreatePreparesCodec() throws Exception { + Configuration conf = new Configuration(false); + conf.setClass(WALEditCodec.WAL_EDIT_CODEC_CLASS_KEY, CustomWALEditCodec.class, WALEditCodec.class); + CustomWALEditCodec codec = (CustomWALEditCodec) WALEditCodec.create(conf, null); + assertTrue("Custom codec didn't get initialized", codec.initialized); + assertTrue("Custom codec didn't have compression set", codec.compressionSet); + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java new file mode 100644 index 000000000000..53086364543f --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java @@ -0,0 +1,168 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.regionserver.wal; + +import static org.junit.Assert.*; + +import java.io.IOException; +import java.util.List; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.Durability; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Tests for HLog write durability + */ +@Category(MediumTests.class) +public class TestDurability { + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static FileSystem FS; + private static MiniDFSCluster CLUSTER; + private static Configuration CONF; + private static final Path DIR = TEST_UTIL.getDataTestDir("TestDurability"); + + private static byte[] FAMILY = Bytes.toBytes("family"); + private static byte[] ROW = Bytes.toBytes("row"); + private static byte[] COL = Bytes.toBytes("col"); + + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + CONF = TEST_UTIL.getConfiguration(); + CONF.setLong("hbase.regionserver.optionallogflushinterval", 500*1000); + TEST_UTIL.startMiniDFSCluster(1); + + CLUSTER = TEST_UTIL.getDFSCluster(); + FS = CLUSTER.getFileSystem(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + @Test + public void testDurability() throws Exception { + HLog wal = new HLog(FS, new Path(DIR, "hlogdir"), + new Path(DIR, "hlogdir_archive"), CONF); + byte[] tableName = Bytes.toBytes("TestDurability"); + HRegion region = createHRegion(tableName, "region", wal, false); + HRegion deferredRegion = createHRegion(tableName, "deferredRegion", wal, true); + + region.put(newPut(null)); + + verifyHLogCount(wal, 1); + + // a put through the deferred table does not write to the wal immdiately + deferredRegion.put(newPut(null)); + verifyHLogCount(wal, 1); + // but will after we sync the wal + wal.sync(); + verifyHLogCount(wal, 2); + + // a put through a deferred table will be sync with the put sync'ed put + deferredRegion.put(newPut(null)); + verifyHLogCount(wal, 2); + region.put(newPut(null)); + verifyHLogCount(wal, 4); + + // a put through a deferred table will be sync with the put sync'ed put + deferredRegion.put(newPut(Durability.USE_DEFAULT)); + verifyHLogCount(wal, 4); + region.put(newPut(Durability.USE_DEFAULT)); + verifyHLogCount(wal, 6); + + // SKIP_WAL never writes to the wal + region.put(newPut(Durability.SKIP_WAL)); + deferredRegion.put(newPut(Durability.SKIP_WAL)); + verifyHLogCount(wal, 6); + wal.sync(); + verifyHLogCount(wal, 6); + + // async overrides sync table default + region.put(newPut(Durability.ASYNC_WAL)); + deferredRegion.put(newPut(Durability.ASYNC_WAL)); + verifyHLogCount(wal, 6); + wal.sync(); + verifyHLogCount(wal, 8); + + // sync overrides async table default + region.put(newPut(Durability.SYNC_WAL)); + deferredRegion.put(newPut(Durability.SYNC_WAL)); + verifyHLogCount(wal, 10); + + // fsync behaves like sync + region.put(newPut(Durability.FSYNC_WAL)); + deferredRegion.put(newPut(Durability.FSYNC_WAL)); + verifyHLogCount(wal, 12); + } + + private Put[] newPut(Durability durability) { + Put p = new Put(ROW); + p.add(FAMILY, COL, COL); + if (durability != null) { + p.setDurability(durability); + } + return new Put[]{p}; + } + + private void verifyHLogCount(HLog log, int expected) throws Exception { + Path walPath = log.computeFilename(); + HLog.Reader reader = HLog.getReader(FS, walPath, CONF); + int count = 0; + HLog.Entry entry = new HLog.Entry(); + while (reader.next(entry) != null) count++; + reader.close(); + assertEquals(expected, count); + } + + // lifted from TestAtomicOperation + private HRegion createHRegion (byte [] tableName, String callingMethod, HLog log, boolean isDeferredLogFlush) + throws IOException { + HTableDescriptor htd = new HTableDescriptor(tableName); + htd.setDeferredLogFlush(isDeferredLogFlush); + HColumnDescriptor hcd = new HColumnDescriptor(FAMILY); + htd.addFamily(hcd); + HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false); + Path path = new Path(DIR + callingMethod); + if (FS.exists(path)) { + if (!FS.delete(path, true)) { + throw new IOException("Failed delete of " + path); + } + } + return HRegion.createHRegion(info, path, HBaseConfiguration.create(), htd, log); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java index 9034844f0131..e59dac9708cb 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,11 +19,13 @@ package org.apache.hadoop.hbase.regionserver.wal; import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; import java.io.IOException; import java.lang.reflect.Method; +import java.net.BindException; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -38,21 +39,25 @@ import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.regionserver.wal.HLog.Reader; -import org.apache.hadoop.hbase.util.Bytes; -import org.apache.hadoop.hbase.util.FSUtils; import org.apache.hadoop.hbase.Coprocessor; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.LargeTests; import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; import org.apache.hadoop.hbase.coprocessor.SampleRegionWALObserver; +import org.apache.hadoop.hbase.regionserver.wal.HLog.Reader; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; import org.apache.hadoop.hdfs.DFSClient; import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.protocol.FSConstants; import org.apache.hadoop.hdfs.server.datanode.DataNode; -import org.apache.hadoop.hdfs.server.namenode.FSNamesystem; import org.apache.hadoop.hdfs.server.namenode.LeaseManager; -import org.apache.hadoop.io.SequenceFile; import org.apache.log4j.Level; import org.junit.After; import org.junit.AfterClass; @@ -101,6 +106,7 @@ public static void setUpBeforeClass() throws Exception { // Make block sizes small. TEST_UTIL.getConfiguration().setInt("dfs.blocksize", 1024 * 1024); // needed for testAppendClose() + TEST_UTIL.getConfiguration().setBoolean("dfs.support.broken.append", true); TEST_UTIL.getConfiguration().setBoolean("dfs.support.append", true); // quicker heartbeat interval for faster DN death notification TEST_UTIL.getConfiguration().setInt("heartbeat.recheck.interval", 5000); @@ -135,6 +141,21 @@ private static String getName() { return "TestHLog"; } + /** + * Test that with three concurrent threads we still write edits in sequence + * edit id order. + * @throws Exception + */ + @Test + public void testMaintainOrderWithConcurrentWrites() throws Exception { + // Run the HPE tool with three threads writing 3000 edits each concurrently. + // When done, verify that all edits were written and that the order in the + // WALs is of ascending edit sequence ids. + int errCode = + HLogPerformanceEvaluation.innerMain(new String [] {"-threads", "3", "-verify", "-iterations", "3000"}); + assertEquals(0, errCode); + } + /** * Just write multiple logs then split. Before fix for HADOOP-2283, this * would fail. @@ -181,7 +202,7 @@ public void testSplit() throws IOException { } log.close(); HLogSplitter logSplitter = HLogSplitter.createLogSplitter(conf, - hbaseDir, logdir, this.oldLogDir, this.fs); + hbaseDir, logdir, oldLogDir, fs); List splits = logSplitter.splitLog(); verifySplits(splits, howmany); @@ -204,7 +225,18 @@ public void Broken_testSync() throws Exception { Path p = new Path(dir, getName() + ".fsdos"); FSDataOutputStream out = fs.create(p); out.write(bytes); - out.sync(); + Method syncMethod = null; + try { + syncMethod = out.getClass().getMethod("hflush", new Class []{}); + } catch (NoSuchMethodException e) { + try { + syncMethod = out.getClass().getMethod("sync", new Class []{}); + } catch (NoSuchMethodException ex) { + fail("This version of Hadoop supports neither Syncable.sync() " + + "nor Syncable.hflush()."); + } + } + syncMethod.invoke(out, new Object[]{}); FSDataInputStream in = fs.open(p); assertTrue(in.available() > 0); byte [] buffer = new byte [1024]; @@ -343,12 +375,16 @@ private void verifySplits(List splits, final int howmany) } } } - - // For this test to pass, requires: - // 1. HDFS-200 (append support) - // 2. HDFS-988 (SafeMode should freeze file operations - // [FSNamesystem.nextGenerationStampForBlock]) - // 3. HDFS-142 (on restart, maintain pendingCreates) + + /* + * We pass different values to recoverFileLease() so that different code paths are covered + * + * For this test to pass, requires: + * 1. HDFS-200 (append support) + * 2. HDFS-988 (SafeMode should freeze file operations + * [FSNamesystem.nextGenerationStampForBlock]) + * 3. HDFS-142 (on restart, maintain pendingCreates) + */ @Test public void testAppendClose() throws Exception { byte [] tableName = Bytes.toBytes(getName()); @@ -371,7 +407,7 @@ public void testAppendClose() throws Exception { wal.sync(); int namenodePort = cluster.getNameNodePort(); final Path walPath = wal.computeFilename(); - + // Stop the cluster. (ensure restart since we're sharing MiniDFSCluster) try { @@ -400,7 +436,17 @@ public void testAppendClose() throws Exception { // the idle time threshold configured in the conf above Thread.sleep(2000); - cluster = new MiniDFSCluster(namenodePort, conf, 5, false, true, true, null, null, null, null); + cluster = null; + // retry a few times if the port is not freed, yet. + for (int i = 0; i < 30; i++) { + try { + cluster = new MiniDFSCluster(namenodePort, conf, 5, false, true, true, null, null, null, null); + break; + } catch (BindException e) { + LOG.info("Sleeping. BindException bringing up new cluster"); + Thread.sleep(1000); + } + } TEST_UTIL.setDFSCluster(cluster); cluster.waitActive(); fs = cluster.getFileSystem(); @@ -412,18 +458,17 @@ public void testAppendClose() throws Exception { Method setLeasePeriod = cluster.getClass() .getDeclaredMethod("setLeasePeriod", new Class[]{Long.TYPE, Long.TYPE}); setLeasePeriod.setAccessible(true); - setLeasePeriod.invoke(cluster, - new Object[]{new Long(1000), new Long(1000)}); + setLeasePeriod.invoke(cluster, 1000L, 1000L); try { Thread.sleep(1000); } catch (InterruptedException e) { LOG.info(e); } - + // Now try recovering the log, like the HMaster would do final FileSystem recoveredFs = fs; final Configuration rlConf = conf; - + class RecoverLogThread extends Thread { public Exception exception = null; public void run() { @@ -449,18 +494,19 @@ public void run() { throw t.exception; // Make sure you can read all the content - SequenceFile.Reader reader - = new SequenceFile.Reader(this.fs, walPath, this.conf); + HLog.Reader reader = HLog.getReader(this.fs, walPath, this.conf); int count = 0; - HLogKey key = HLog.newKey(conf); - WALEdit val = new WALEdit(); - while (reader.next(key, val)) { + HLog.Entry entry = new HLog.Entry(); + while (reader.next(entry) != null) { count++; assertTrue("Should be one KeyValue per WALEdit", - val.getKeyValues().size() == 1); + entry.getEdit().getKeyValues().size() == 1); } assertEquals(total, count); reader.close(); + + // Reset the lease period + setLeasePeriod.invoke(cluster, new Object[]{new Long(60000), new Long(3600000)}); } /** @@ -661,12 +707,12 @@ public void testLogCleaning() throws Exception { // Before HBASE-3198 it used to delete it addEdits(log, hri, tableName, 1); log.rollWriter(); - assertEquals(1, log.getNumLogFiles()); + assertEquals(2, log.getNumLogFiles()); // See if there's anything wrong with more than 1 edit addEdits(log, hri, tableName, 2); log.rollWriter(); - assertEquals(2, log.getNumLogFiles()); + assertEquals(3, log.getNumLogFiles()); // Now mix edits from 2 regions, still no flushing addEdits(log, hri, tableName, 1); @@ -674,14 +720,14 @@ public void testLogCleaning() throws Exception { addEdits(log, hri, tableName, 1); addEdits(log, hri2, tableName2, 1); log.rollWriter(); - assertEquals(3, log.getNumLogFiles()); + assertEquals(4, log.getNumLogFiles()); // Flush the first region, we expect to see the first two files getting // archived long seqId = log.startCacheFlush(hri.getEncodedNameAsBytes()); log.completeCacheFlush(hri.getEncodedNameAsBytes(), tableName, seqId, false); log.rollWriter(); - assertEquals(2, log.getNumLogFiles()); + assertEquals(3, log.getNumLogFiles()); // Flush the second region, which removes all the remaining output files // since the oldest was completely flushed and the two others only contain @@ -689,7 +735,7 @@ public void testLogCleaning() throws Exception { seqId = log.startCacheFlush(hri2.getEncodedNameAsBytes()); log.completeCacheFlush(hri2.getEncodedNameAsBytes(), tableName2, seqId, false); log.rollWriter(); - assertEquals(0, log.getNumLogFiles()); + assertEquals(1, log.getNumLogFiles()); } finally { if (log != null) log.closeAndDelete(); } @@ -763,7 +809,7 @@ public void postLogArchive(Path oldFile, Path newFile) { @Override public void logRollRequested() { // TODO Auto-generated method stub - + } @Override diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java index 0b0a7cee11ad..c6584bbcf0fc 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java index 13f77ec33558..1da80c576a84 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java index f1ea70114b06..95d7009aa55b 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -27,11 +26,13 @@ import java.io.FileNotFoundException; import java.io.IOException; +import java.lang.reflect.Method; import java.util.ArrayList; import java.util.Collections; import java.util.List; import java.util.Map; import java.util.NavigableSet; +import java.util.concurrent.CountDownLatch; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; @@ -85,7 +86,7 @@ public class TestHLogSplit { private Configuration conf; private FileSystem fs; - private final static HBaseTestingUtility + protected final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); @@ -118,14 +119,11 @@ static enum Corruptions { @BeforeClass public static void setUpBeforeClass() throws Exception { - TEST_UTIL.getConfiguration(). - setBoolean("dfs.support.append", true); - TEST_UTIL.getConfiguration(). - setStrings("hbase.rootdir", hbaseDir.toString()); - TEST_UTIL.getConfiguration(). - setClass("hbase.regionserver.hlog.writer.impl", - InstrumentedSequenceFileLogWriter.class, HLog.Writer.class); - + TEST_UTIL.getConfiguration().setStrings("hbase.rootdir", hbaseDir.toString()); + TEST_UTIL.getConfiguration().setClass("hbase.regionserver.hlog.writer.impl", + InstrumentedSequenceFileLogWriter.class, HLog.Writer.class); + TEST_UTIL.getConfiguration().setBoolean("dfs.support.broken.append", true); + TEST_UTIL.getConfiguration().setBoolean("dfs.support.append", true); TEST_UTIL.startMiniDFSCluster(2); } @@ -190,16 +188,44 @@ public void testSplitFailsIfNewHLogGetsCreatedAfterSplitStarted() generateHLogs(-1); + CountDownLatch latch = new CountDownLatch(1); try { - (new ZombieNewLogWriterRegionServer(stop)).start(); - HLogSplitter logSplitter = HLogSplitter.createLogSplitter(conf, - hbaseDir, hlogDir, oldLogDir, fs); - logSplitter.splitLog(); + (new ZombieNewLogWriterRegionServer(latch, stop)).start(); + HLogSplitter logSplitter = HLogSplitter.createLogSplitter(conf, hbaseDir, hlogDir, oldLogDir, + fs); + logSplitter.splitLog(latch); } finally { stop.set(true); } } + /** + * Test old recovered edits file doesn't break HLogSplitter. + * This is useful in upgrading old instances. + */ + @Test + public void testOldRecoveredEditsFileSidelined() throws IOException { + FileSystem fs = FileSystem.get(TEST_UTIL.getConfiguration()); + byte [] encoded = HRegionInfo.FIRST_META_REGIONINFO.getEncodedNameAsBytes(); + Path tdir = new Path(hbaseDir, Bytes.toString(HConstants.META_TABLE_NAME)); + Path regiondir = new Path(tdir, + HRegionInfo.FIRST_META_REGIONINFO.getEncodedName()); + fs.mkdirs(regiondir); + long now = System.currentTimeMillis(); + HLog.Entry entry = + new HLog.Entry(new HLogKey(encoded, + HConstants.META_TABLE_NAME, 1, now, HConstants.DEFAULT_CLUSTER_ID), + new WALEdit()); + Path parent = HLog.getRegionDirRecoveredEditsDir(regiondir); + assertEquals(parent.getName(), HLog.RECOVERED_EDITS_DIR); + fs.createNewFile(parent); // create a recovered.edits file + + Path p = HLogSplitter.getRegionSplitEditsPath(fs, entry, hbaseDir, true); + String parentOfParent = p.getParent().getParent().getName(); + assertEquals(parentOfParent, HRegionInfo.FIRST_META_REGIONINFO.getEncodedName()); + HLog.createWriter(fs, p, conf).close(); + } + @Test public void testSplitPreservesEdits() throws IOException{ final String REGION = "region__1"; @@ -554,16 +580,23 @@ public void testSplitWillNotTouchLogsIfNewHLogGetsCreatedAfterSplitStarted() AtomicBoolean stop = new AtomicBoolean(false); generateHLogs(-1); fs.initialize(fs.getUri(), conf); - Thread zombie = new ZombieNewLogWriterRegionServer(stop); + CountDownLatch latch = new CountDownLatch(1); + Thread zombie = new ZombieNewLogWriterRegionServer(latch, stop); + List splits = null; try { zombie.start(); try { HLogSplitter logSplitter = HLogSplitter.createLogSplitter(conf, hbaseDir, hlogDir, oldLogDir, fs); - logSplitter.splitLog(); - } catch (IOException ex) {/* expected */} - int logFilesNumber = fs.listStatus(hlogDir).length; + splits = logSplitter.splitLog(latch); + } catch (IOException ex) { + /* expected */ + LOG.warn("testSplitWillNotTouchLogsIfNewHLogGetsCreatedAfterSplitStarted", ex); + } + FileStatus[] files = fs.listStatus(hlogDir); + if (files == null) fail("no files in " + hlogDir + " with splits " + splits); + int logFilesNumber = files.length; assertEquals("Log files should not be archived if there's an extra file after split", NUM_WRITERS + 1, logFilesNumber); @@ -960,8 +993,10 @@ public void run() { */ class ZombieNewLogWriterRegionServer extends Thread { AtomicBoolean stop; - public ZombieNewLogWriterRegionServer(AtomicBoolean stop) { + CountDownLatch latch; + public ZombieNewLogWriterRegionServer(CountDownLatch latch, AtomicBoolean stop) { super("ZombieNewLogWriterRegionServer"); + this.latch = latch; this.stop = stop; } @@ -978,7 +1013,7 @@ public void run() { try { while (!fs.exists(recoveredEdits) && !stop.get()) { - flushToConsole("Juliet: split not started, sleeping a bit..."); + LOG.info("Juliet: split not started, sleeping a bit..."); Threads.sleep(10); } @@ -988,8 +1023,10 @@ public void run() { appendEntry(writer, "juliet".getBytes(), ("juliet").getBytes(), ("r").getBytes(), FAMILY, QUALIFIER, VALUE, 0); writer.close(); - flushToConsole("Juliet file creator: created file " + julietLog); + LOG.info("Juliet file creator: created file " + julietLog); + latch.countDown(); } catch (IOException e1) { + LOG.error("Failed to create file " + julietLog, e1); assertTrue("Failed to create file " + julietLog, false); } } @@ -1017,10 +1054,9 @@ public void testSplitLogFileWithOneRegion() throws IOException { generateHLogs(1, 10, -1); FileStatus logfile = fs.listStatus(hlogDir)[0]; fs.initialize(fs.getUri(), conf); - HLogSplitter.splitLogFileToTemp(hbaseDir, "tmpdir", logfile, fs, - conf, reporter); - HLogSplitter.moveRecoveredEditsFromTemp("tmpdir", hbaseDir, oldLogDir, - logfile.getPath().toString(), conf); + HLogSplitter.splitLogFile(hbaseDir, logfile, fs, conf, reporter); + HLogSplitter.finishSplitLogFile(hbaseDir, oldLogDir, logfile.getPath() + .toString(), conf); Path originalLog = (fs.listStatus(oldLogDir))[0].getPath(); @@ -1047,10 +1083,9 @@ public void testSplitLogFileDeletedRegionDir() LOG.info("Region directory is" + regiondir); fs.delete(regiondir, true); - HLogSplitter.splitLogFileToTemp(hbaseDir, "tmpdir", logfile, fs, - conf, reporter); - HLogSplitter.moveRecoveredEditsFromTemp("tmpdir", hbaseDir, oldLogDir, - logfile.getPath().toString(), conf); + HLogSplitter.splitLogFile(hbaseDir, logfile, fs, conf, reporter); + HLogSplitter.finishSplitLogFile(hbaseDir, oldLogDir, logfile.getPath() + .toString(), conf); assertTrue(!fs.exists(regiondir)); assertTrue(true); @@ -1066,10 +1101,9 @@ public void testSplitLogFileEmpty() throws IOException { fs.initialize(fs.getUri(), conf); - HLogSplitter.splitLogFileToTemp(hbaseDir, "tmpdir", logfile, fs, - conf, reporter); - HLogSplitter.moveRecoveredEditsFromTemp("tmpdir", hbaseDir, oldLogDir, - logfile.getPath().toString(), conf); + HLogSplitter.splitLogFile(hbaseDir, logfile, fs, conf, reporter); + HLogSplitter.finishSplitLogFile(hbaseDir, oldLogDir, logfile.getPath() + .toString(), conf); Path tdir = HTableDescriptor.getTableDir(hbaseDir, TABLE_NAME); assertFalse(fs.exists(tdir)); @@ -1083,10 +1117,9 @@ public void testSplitLogFileMultipleRegions() throws IOException { FileStatus logfile = fs.listStatus(hlogDir)[0]; fs.initialize(fs.getUri(), conf); - HLogSplitter.splitLogFileToTemp(hbaseDir, "tmpdir", logfile, fs, - conf, reporter); - HLogSplitter.moveRecoveredEditsFromTemp("tmpdir", hbaseDir, oldLogDir, - logfile.getPath().toString(), conf); + HLogSplitter.splitLogFile(hbaseDir, logfile, fs, conf, reporter); + HLogSplitter.finishSplitLogFile(hbaseDir, oldLogDir, logfile.getPath() + .toString(), conf); for (String region : regions) { Path recovered = getLogForRegion(hbaseDir, TABLE_NAME, region); assertEquals(10, countHLog(recovered, fs, conf)); @@ -1104,10 +1137,9 @@ public void testSplitLogFileFirstLineCorruptionLog() Corruptions.INSERT_GARBAGE_ON_FIRST_LINE, true, fs); fs.initialize(fs.getUri(), conf); - HLogSplitter.splitLogFileToTemp(hbaseDir, "tmpdir", logfile, fs, - conf, reporter); - HLogSplitter.moveRecoveredEditsFromTemp("tmpdir", hbaseDir, oldLogDir, - logfile.getPath().toString(), conf); + HLogSplitter.splitLogFile(hbaseDir, logfile, fs, conf, reporter); + HLogSplitter.finishSplitLogFile(hbaseDir, oldLogDir, logfile.getPath() + .toString(), conf); final Path corruptDir = new Path(conf.get(HConstants.HBASE_DIR), conf.get( "hbase.regionserver.hlog.splitlog.corrupt.dir", ".corrupt")); @@ -1198,7 +1230,7 @@ private void generateHLogs(int writers, int entries, int leaveOpen) throws IOExc } if (i != leaveOpen) { writer[i].close(); - flushToConsole("Closing writer " + i); + LOG.info("Closing writer " + i); } } } @@ -1226,7 +1258,9 @@ private void corruptHLog(Path path, Corruptions corruption, boolean close, switch (corruption) { case APPEND_GARBAGE: - out = fs.append(path); + fs.delete(path, false); + out = fs.create(path); + out.write(corrupted_bytes); out.write("-----".getBytes()); closeOrFlush(close, out); break; @@ -1266,7 +1300,22 @@ private void closeOrFlush(boolean close, FSDataOutputStream out) if (close) { out.close(); } else { - out.sync(); + Method syncMethod = null; + try { + syncMethod = out.getClass().getMethod("hflush", new Class []{}); + } catch (NoSuchMethodException e) { + try { + syncMethod = out.getClass().getMethod("sync", new Class []{}); + } catch (NoSuchMethodException ex) { + throw new IOException("This version of Hadoop supports " + + "neither Syncable.sync() nor Syncable.hflush()."); + } + } + try { + syncMethod.invoke(out, new Object[]{}); + } catch (Exception e) { + throw new IOException(e); + } // Not in 0out.hflush(); } } @@ -1294,8 +1343,9 @@ public long appendEntry(HLog.Writer writer, byte[] table, byte[] region, byte[] row, byte[] family, byte[] qualifier, byte[] value, long seq) throws IOException { - + LOG.info(Thread.currentThread().getName() + " append"); writer.append(createTestEntry(table, region, row, family, qualifier, value, seq)); + LOG.info(Thread.currentThread().getName() + " sync"); writer.sync(); return seq; } diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplitCompressed.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplitCompressed.java new file mode 100644 index 000000000000..101678a9a21a --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplitCompressed.java @@ -0,0 +1,35 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.regionserver.wal; + + +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.LargeTests; +import org.junit.BeforeClass; +import org.junit.experimental.categories.Category; + +@Category(LargeTests.class) +public class TestHLogSplitCompressed extends TestHLogSplit { + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TestHLogSplit.setUpBeforeClass(); + TEST_UTIL.getConfiguration().setBoolean(HConstants.ENABLE_WAL_COMPRESSION, true); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestKeyValueCompression.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestKeyValueCompression.java new file mode 100644 index 000000000000..8fa7fe883b68 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestKeyValueCompression.java @@ -0,0 +1,81 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License + */ +package org.apache.hadoop.hbase.regionserver.wal; + +import java.io.ByteArrayInputStream; +import java.io.DataInputStream; +import java.util.List; + +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.io.DataOutputBuffer; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import static org.junit.Assert.*; + +import com.google.common.collect.Lists; + +@Category(SmallTests.class) +public class TestKeyValueCompression { + private static final byte[] VALUE = Bytes.toBytes("fake value"); + private static final int BUF_SIZE = 256*1024; + + @Test + public void testCountingKVs() throws Exception { + List kvs = Lists.newArrayList(); + for (int i = 0; i < 400; i++) { + byte[] row = Bytes.toBytes("row" + i); + byte[] fam = Bytes.toBytes("fam" + i); + byte[] qual = Bytes.toBytes("qual" + i); + kvs.add(new KeyValue(row, fam, qual, 12345L, VALUE)); + } + + runTestCycle(kvs); + } + + @Test + public void testRepeatingKVs() throws Exception { + List kvs = Lists.newArrayList(); + for (int i = 0; i < 400; i++) { + byte[] row = Bytes.toBytes("row" + (i % 10)); + byte[] fam = Bytes.toBytes("fam" + (i % 127)); + byte[] qual = Bytes.toBytes("qual" + (i % 128)); + kvs.add(new KeyValue(row, fam, qual, 12345L, VALUE)); + } + + runTestCycle(kvs); + } + + private void runTestCycle(List kvs) throws Exception { + CompressionContext ctx = new CompressionContext(LRUDictionary.class); + DataOutputBuffer buf = new DataOutputBuffer(BUF_SIZE); + for (KeyValue kv : kvs) { + KeyValueCompression.writeKV(buf, kv, ctx); + } + + ctx.clear(); + DataInputStream in = new DataInputStream(new ByteArrayInputStream( + buf.getData(), 0, buf.getLength())); + for (KeyValue kv : kvs) { + KeyValue readBack = KeyValueCompression.readKV(in, ctx); + assertEquals(kv, readBack); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLRUDictionary.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLRUDictionary.java new file mode 100644 index 000000000000..99983a2f5cc1 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLRUDictionary.java @@ -0,0 +1,155 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.regionserver.wal; + +import static org.junit.Assert.*; + +import java.math.BigInteger; +import java.util.Arrays; +import java.util.Random; + +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Tests LRUDictionary + */ +@Category(SmallTests.class) +public class TestLRUDictionary { + LRUDictionary testee; + + @Before + public void setUp() throws Exception { + testee = new LRUDictionary(); + } + + @Test + public void TestContainsNothing() { + assertTrue(isDictionaryEmpty(testee)); + } + + /** + * Assert can't add empty array. + */ + @Test + public void testPassingEmptyArrayToFindEntry() { + assertEquals(Dictionary.NOT_IN_DICTIONARY, + testee.findEntry(HConstants.EMPTY_BYTE_ARRAY, 0, 0)); + assertEquals(Dictionary.NOT_IN_DICTIONARY, + testee.addEntry(HConstants.EMPTY_BYTE_ARRAY, 0, 0)); + } + + @Test + public void testPassingSameArrayToAddEntry() { + // Add random predefined byte array, in this case a random byte array from + // HConstants. Assert that when we add, we get new index. Thats how it + // works. + int len = HConstants.CATALOG_FAMILY.length; + int index = testee.addEntry(HConstants.CATALOG_FAMILY, 0, len); + assertFalse(index == testee.addEntry(HConstants.CATALOG_FAMILY, 0, len)); + assertFalse(index == testee.addEntry(HConstants.CATALOG_FAMILY, 0, len)); + } + + @Test + public void testBasic() { + Random rand = new Random(); + byte[] testBytes = new byte[10]; + rand.nextBytes(testBytes); + + // Verify that our randomly generated array doesn't exist in the dictionary + assertEquals(testee.findEntry(testBytes, 0, testBytes.length), -1); + + // now since we looked up an entry, we should have added it to the + // dictionary, so it isn't empty + + assertFalse(isDictionaryEmpty(testee)); + + // Check if we can find it using findEntry + short t = testee.findEntry(testBytes, 0, testBytes.length); + + // Making sure we do find what we're looking for + assertTrue(t != -1); + + byte[] testBytesCopy = new byte[20]; + + Bytes.putBytes(testBytesCopy, 10, testBytes, 0, testBytes.length); + + // copy byte arrays, make sure that we check that equal byte arrays are + // equal without just checking the reference + assertEquals(testee.findEntry(testBytesCopy, 10, testBytes.length), t); + + // make sure the entry retrieved is the same as the one put in + assertTrue(Arrays.equals(testBytes, testee.getEntry(t))); + + testee.clear(); + + // making sure clear clears the dictionary + assertTrue(isDictionaryEmpty(testee)); + } + + @Test + public void TestLRUPolicy(){ + //start by filling the dictionary up with byte arrays + for (int i = 0; i < LRUDictionary.BidirectionalLRUMap.MAX_SIZE; i++) { + testee.findEntry((BigInteger.valueOf(i)).toByteArray(), 0, + (BigInteger.valueOf(i)).toByteArray().length); + } + + // check we have the first element added + assertTrue(testee.findEntry(BigInteger.ZERO.toByteArray(), 0, + BigInteger.ZERO.toByteArray().length) != -1); + + // check for an element we know isn't there + assertTrue(testee.findEntry(BigInteger.valueOf(Integer.MAX_VALUE).toByteArray(), 0, + BigInteger.valueOf(Integer.MAX_VALUE).toByteArray().length) == -1); + + // since we just checked for this element, it should be there now. + assertTrue(testee.findEntry(BigInteger.valueOf(Integer.MAX_VALUE).toByteArray(), 0, + BigInteger.valueOf(Integer.MAX_VALUE).toByteArray().length) != -1); + + // test eviction, that the least recently added or looked at element is + // evicted. We looked at ZERO so it should be in the dictionary still. + assertTrue(testee.findEntry(BigInteger.ZERO.toByteArray(), 0, + BigInteger.ZERO.toByteArray().length) != -1); + // Now go from beyond 1 to the end. + for(int i = 1; i < LRUDictionary.BidirectionalLRUMap.MAX_SIZE; i++) { + assertTrue(testee.findEntry(BigInteger.valueOf(i).toByteArray(), 0, + BigInteger.valueOf(i).toByteArray().length) == -1); + } + + // check we can find all of these. + for (int i = 0; i < LRUDictionary.BidirectionalLRUMap.MAX_SIZE; i++) { + assertTrue(testee.findEntry(BigInteger.valueOf(i).toByteArray(), 0, + BigInteger.valueOf(i).toByteArray().length) != -1); + } + } + + static private boolean isDictionaryEmpty(LRUDictionary dict) { + try { + dict.getEntry((short)0); + return false; + } catch (IndexOutOfBoundsException ioobe) { + return true; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollPeriod.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollPeriod.java new file mode 100644 index 000000000000..a7d9ccb90a46 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollPeriod.java @@ -0,0 +1,179 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver.wal; + +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertFalse; + +import java.util.List; +import java.util.ArrayList; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Tests that verifies that the log is forced to be rolled every "hbase.regionserver.logroll.period" + */ +@Category(MediumTests.class) +public class TestLogRollPeriod { + private static final Log LOG = LogFactory.getLog(TestLogRolling.class); + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + private final static long LOG_ROLL_PERIOD = 4000; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + // disable the ui + TEST_UTIL.getConfiguration().setInt("hbase.regionsever.info.port", -1); + + TEST_UTIL.getConfiguration().setLong("hbase.regionserver.logroll.period", LOG_ROLL_PERIOD); + + TEST_UTIL.startMiniCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * Tests that the LogRoller perform the roll even if there are no edits + */ + @Test + public void testNoEdits() throws Exception { + final byte[] tableName = Bytes.toBytes("TestLogRollPeriodNoEdits"); + + TEST_UTIL.createTable(tableName, Bytes.toBytes("cf")); + try { + HTable table = new HTable(TEST_UTIL.getConfiguration(), tableName); + try { + HRegionServer server = TEST_UTIL.getRSForFirstRegionInTable(tableName); + HLog log = server.getWAL(); + checkMinLogRolls(log, 5); + } finally { + table.close(); + } + } finally { + TEST_UTIL.deleteTable(tableName); + } + } + + /** + * Tests that the LogRoller perform the roll with some data in the log + */ + @Test(timeout=60000) + public void testWithEdits() throws Exception { + final byte[] tableName = Bytes.toBytes("TestLogRollPeriodWithEdits"); + final byte[] family = Bytes.toBytes("cf"); + + TEST_UTIL.createTable(tableName, family); + try { + HRegionServer server = TEST_UTIL.getRSForFirstRegionInTable(tableName); + HLog log = server.getWAL(); + final HTable table = new HTable(TEST_UTIL.getConfiguration(), tableName); + + Thread writerThread = new Thread("writer") { + @Override + public void run() { + try { + long row = 0; + while (!interrupted()) { + Put p = new Put(Bytes.toBytes(String.format("row%d", row))); + p.add(family, Bytes.toBytes("col"), Bytes.toBytes(row)); + table.put(p); + row++; + + Thread.sleep(LOG_ROLL_PERIOD / 16); + } + } catch (Exception e) { + LOG.warn(e); + } + } + }; + + try { + writerThread.start(); + checkMinLogRolls(log, 5); + } finally { + writerThread.interrupt(); + writerThread.join(); + table.close(); + } + } finally { + TEST_UTIL.deleteTable(tableName); + } + } + + private void checkMinLogRolls(final HLog log, final int minRolls) + throws Exception { + final List paths = new ArrayList(); + log.registerWALActionsListener(new WALActionsListener() { + @Override + public void preLogRoll(Path oldFile, Path newFile) {} + @Override + public void postLogRoll(Path oldFile, Path newFile) { + LOG.debug("postLogRoll: oldFile="+oldFile+" newFile="+newFile); + paths.add(newFile); + } + @Override + public void preLogArchive(Path oldFile, Path newFile) {} + @Override + public void postLogArchive(Path oldFile, Path newFile) {} + @Override + public void logRollRequested() {} + @Override + public void logCloseRequested() {} + @Override + public void visitLogEntryBeforeWrite(HRegionInfo info, HLogKey logKey, WALEdit logEdit) {} + @Override + public void visitLogEntryBeforeWrite(HTableDescriptor htd, HLogKey logKey, WALEdit logEdit) {} + }); + + // Sleep until we should get at least min-LogRoll events + long wtime = System.currentTimeMillis(); + Thread.sleep((minRolls + 1) * LOG_ROLL_PERIOD); + // Do some extra sleep in case the machine is slow, + // and the log-roll is not triggered exactly on LOG_ROLL_PERIOD. + final int NUM_RETRIES = 1 + 8 * (minRolls - paths.size()); + for (int retry = 0; paths.size() < minRolls && retry < NUM_RETRIES; ++retry) { + Thread.sleep(LOG_ROLL_PERIOD / 4); + } + wtime = System.currentTimeMillis() - wtime; + LOG.info(String.format("got %d rolls after %dms (%dms each) - expected at least %d rolls", + paths.size(), wtime, wtime / paths.size(), minRolls)); + assertFalse(paths.size() < minRolls); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java index cb7efc3d0918..73842264b53b 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -209,7 +208,7 @@ public void testLogRolling() throws FailedLogCloseException, IOException { int count = log.getNumLogFiles(); LOG.info("after flushing all regions and rolling logs there are " + log.getNumLogFiles() + " log files"); - assertTrue(("actual count: " + count), count <= 2); + assertTrue(("actual count: " + count), count <= 3); } private static String getName() { @@ -394,7 +393,7 @@ public void testLogRollOnDatanodeDeath() throws Exception { * restarted. * @throws Exception */ - @Test + //DISABLED BECAUSE FLAKEY @Test public void testLogRollOnPipelineRestart() throws Exception { LOG.info("Starting testLogRollOnPipelineRestart"); assertTrue("This test requires HLog file replication.", diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java new file mode 100644 index 000000000000..3f467363bda1 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java @@ -0,0 +1,142 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver.wal; + +import static org.junit.Assert.assertFalse; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test many concurrent appenders to an {@link #HLog} while rolling the log. + */ +@Category(MediumTests.class) +public class TestLogRollingNoCluster { + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private final static byte [] EMPTY_1K_ARRAY = new byte[1024]; + private static final int THREAD_COUNT = 100; // Spin up this many threads + + /** + * Spin up a bunch of threads and have them all append to a WAL. Roll the + * WAL frequently to try and trigger NPE. + * @throws IOException + * @throws InterruptedException + */ + @Test + public void testContendedLogRolling() throws IOException, InterruptedException { + FileSystem fs = FileSystem.get(TEST_UTIL.getConfiguration()); + Path dir = TEST_UTIL.getDataTestDir(); + HLog wal = new HLog(fs, new Path(dir, "logs"), new Path(dir, "oldlogs"), + TEST_UTIL.getConfiguration()); + Appender [] appenders = null; + + final int count = THREAD_COUNT; + appenders = new Appender[count]; + try { + for (int i = 0; i < count; i++) { + // Have each appending thread write 'count' entries + appenders[i] = new Appender(wal, i, count); + } + for (int i = 0; i < count; i++) { + appenders[i].start(); + } + for (int i = 0; i < count; i++) { + //ensure that all threads are joined before closing the wal + appenders[i].join(); + } + } finally { + wal.close(); + } + for (int i = 0; i < count; i++) { + assertFalse(appenders[i].isException()); + } + } + + /** + * Appender thread. Appends to passed wal file. + */ + static class Appender extends Thread { + private final Log log; + private final HLog wal; + private final int count; + private Exception e = null; + + Appender(final HLog wal, final int index, final int count) { + super("" + index); + this.wal = wal; + this.count = count; + this.log = LogFactory.getLog("Appender:" + getName()); + } + + /** + * @return Call when the thread is done. + */ + boolean isException() { + return !isAlive() && this.e != null; + } + + Exception getException() { + return this.e; + } + + @Override + public void run() { + this.log.info(getName() +" started"); + try { + for (int i = 0; i < this.count; i++) { + long now = System.currentTimeMillis(); + // Roll every ten edits if the log has anything in it. + if (i % 10 == 0 && this.wal.getNumEntries() > 0) { + this.wal.rollWriter(); + } + WALEdit edit = new WALEdit(); + byte[] bytes = Bytes.toBytes(i); + edit.add(new KeyValue(bytes, bytes, bytes, now, EMPTY_1K_ARRAY)); + + this.wal.append(HRegionInfo.FIRST_META_REGIONINFO, + HTableDescriptor.META_TABLEDESC.getName(), + edit, now, HTableDescriptor.META_TABLEDESC); + } + String msg = getName() + " finished"; + if (isException()) + this.log.info(msg, getException()); + else + this.log.info(msg); + } catch (Exception e) { + this.e = e; + log.info("Caught exception from Appender:" + getName(), e); + } + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java index de81e9170c7b..ee1789bb5938 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java index a11899c6e37e..cad123c21f3c 100644 --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -21,31 +20,56 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; import java.io.IOException; import java.security.PrivilegedExceptionAction; import java.util.ArrayList; import java.util.List; +import java.util.SortedSet; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MasterNotRunningException; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.ZooKeeperConnectionException; +import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; -import org.apache.hadoop.hbase.io.hfile.HFile; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.master.HMaster; import org.apache.hadoop.hbase.monitoring.MonitoredTask; import org.apache.hadoop.hbase.regionserver.FlushRequester; import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.regionserver.RegionScanner; +import org.apache.hadoop.hbase.regionserver.RegionServerServices; import org.apache.hadoop.hbase.regionserver.Store; +import org.apache.hadoop.hbase.regionserver.TimeRangeTracker; import org.apache.hadoop.hbase.security.User; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.EnvironmentEdge; import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; +import org.apache.hadoop.hbase.util.HFileTestUtil; import org.apache.hadoop.hbase.util.Pair; import org.junit.After; import org.junit.AfterClass; @@ -61,7 +85,7 @@ @Category(MediumTests.class) public class TestWALReplay { public static final Log LOG = LogFactory.getLog(TestWALReplay.class); - private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); private final EnvironmentEdge ee = EnvironmentEdgeManager.getDelegate(); private Path hbaseRootDir = null; private Path oldLogDir; @@ -75,7 +99,7 @@ public static void setUpBeforeClass() throws Exception { conf.setBoolean("dfs.support.append", true); // The below config supported by 0.20-append and CDH3b2 conf.setInt("dfs.client.block.recovery.retries", 2); - TEST_UTIL.startMiniDFSCluster(3); + TEST_UTIL.startMiniCluster(3); Path hbaseRootDir = TEST_UTIL.getDFSCluster().getFileSystem().makeQualified(new Path("/hbase")); LOG.info("hbase.rootdir=" + hbaseRootDir); @@ -84,7 +108,7 @@ public static void setUpBeforeClass() throws Exception { @AfterClass public static void tearDownAfterClass() throws Exception { - TEST_UTIL.shutdownMiniDFSCluster(); + TEST_UTIL.shutdownMiniCluster(); } @Before @@ -115,6 +139,100 @@ private void deleteDir(final Path p) throws IOException { } } + /** + * + * @throws Exception + */ + @Test + public void testReplayEditsAfterRegionMovedWithMultiCF() throws Exception { + final byte[] tableName = Bytes + .toBytes("testReplayEditsAfterRegionMovedWithMultiCF"); + byte[] family1 = Bytes.toBytes("cf1"); + byte[] family2 = Bytes.toBytes("cf2"); + byte[] qualifier = Bytes.toBytes("q"); + byte[] value = Bytes.toBytes("testV"); + byte[][] familys = { family1, family2 }; + TEST_UTIL.createTable(tableName, familys); + HTable htable = new HTable(TEST_UTIL.getConfiguration(), tableName); + Put put = new Put(Bytes.toBytes("r1")); + put.add(family1, qualifier, value); + htable.put(put); + ResultScanner resultScanner = htable.getScanner(new Scan()); + int count = 0; + while (resultScanner.next() != null) { + count++; + } + resultScanner.close(); + assertEquals(1, count); + + MiniHBaseCluster hbaseCluster = TEST_UTIL.getMiniHBaseCluster(); + List regions = hbaseCluster.getRegions(tableName); + assertEquals(1, regions.size()); + + // move region to another regionserver + HRegion destRegion = regions.get(0); + int originServerNum = hbaseCluster + .getServerWith(destRegion.getRegionName()); + assertTrue("Please start more than 1 regionserver", hbaseCluster + .getRegionServerThreads().size() > 1); + int destServerNum = 0; + while (destServerNum == originServerNum) { + destServerNum++; + } + HRegionServer originServer = hbaseCluster.getRegionServer(originServerNum); + HRegionServer destServer = hbaseCluster.getRegionServer(destServerNum); + // move region to destination regionserver + moveRegionAndWait(destRegion, destServer); + + // delete the row + Delete del = new Delete(Bytes.toBytes("r1")); + htable.delete(del); + resultScanner = htable.getScanner(new Scan()); + count = 0; + while (resultScanner.next() != null) { + count++; + } + resultScanner.close(); + assertEquals(0, count); + + // flush region and make major compaction + destServer.getOnlineRegion(destRegion.getRegionName()).flushcache(); + // wait to complete major compaction + for (Store store : destServer.getOnlineRegion(destRegion.getRegionName()) + .getStores().values()) { + store.triggerMajorCompaction(); + } + destServer.getOnlineRegion(destRegion.getRegionName()).compactStores(); + + // move region to origin regionserver + moveRegionAndWait(destRegion, originServer); + // abort the origin regionserver + originServer.abort("testing"); + + // see what we get + Result result = htable.get(new Get(Bytes.toBytes("r1"))); + if (result != null) { + assertTrue("Row is deleted, but we get" + result.toString(), + (result == null) || result.isEmpty()); + } + resultScanner.close(); + } + + private void moveRegionAndWait(HRegion destRegion, HRegionServer destServer) + throws InterruptedException, MasterNotRunningException, + ZooKeeperConnectionException, IOException { + HMaster master = TEST_UTIL.getMiniHBaseCluster().getMaster(); + TEST_UTIL.getHBaseAdmin().move( + destRegion.getRegionInfo().getEncodedNameAsBytes(), + Bytes.toBytes(destServer.getServerName().getServerName())); + while (true) { + ServerName serverName = master.getAssignmentManager() + .getRegionServerOfRegion(destRegion.getRegionInfo()); + if (serverName != null && serverName.equals(destServer.getServerName())) break; + Thread.sleep(10); + } + } + /** * Tests for hbase-2727. * @throws Exception @@ -133,7 +251,8 @@ public void test2727() throws Exception { HTableDescriptor htd = createBasic3FamilyHTD(tableNameStr); HRegion region2 = HRegion.createHRegion(hri, hbaseRootDir, this.conf, htd); - + region2.close(); + region2.getLog().closeAndDelete(); final byte [] tableName = Bytes.toBytes(tableNameStr); final byte [] rowName = tableName; @@ -186,28 +305,33 @@ public void test2727() throws Exception { public void testRegionMadeOfBulkLoadedFilesOnly() throws IOException, SecurityException, IllegalArgumentException, NoSuchFieldException, IllegalAccessException, InterruptedException { - final String tableNameStr = "testReplayEditsWrittenViaHRegion"; + final String tableNameStr = "testRegionMadeOfBulkLoadedFilesOnly"; final HRegionInfo hri = createBasic3FamilyHRegionInfo(tableNameStr); final Path basedir = new Path(this.hbaseRootDir, tableNameStr); deleteDir(basedir); final HTableDescriptor htd = createBasic3FamilyHTD(tableNameStr); HRegion region2 = HRegion.createHRegion(hri, hbaseRootDir, this.conf, htd); + region2.close(); + region2.getLog().closeAndDelete(); HLog wal = createWAL(this.conf); HRegion region = HRegion.openHRegion(hri, htd, wal, this.conf); - Path f = new Path(basedir, "hfile"); - HFile.Writer writer = - HFile.getWriterFactoryNoCache(conf).withPath(fs, f).create(); + byte [] family = htd.getFamilies().iterator().next().getName(); - byte [] row = Bytes.toBytes(tableNameStr); - writer.append(new KeyValue(row, family, family, row)); - writer.close(); + Path f = new Path(basedir, "hfile"); + HFileTestUtil.createHFile(this.conf, fs, f, family, family, Bytes.toBytes(""), + Bytes.toBytes("z"), 10); List > hfs= new ArrayList>(1); hfs.add(Pair.newPair(family, f.toString())); - region.bulkLoadHFiles(hfs); + region.bulkLoadHFiles(hfs, true); + // Add an edit so something in the WAL + byte [] row = Bytes.toBytes(tableNameStr); region.put((new Put(row)).add(family, family, family)); wal.sync(); + final int rowsInsertedCount = 11; + + assertEquals(rowsInsertedCount, getScannedCount(region.getScanner(new Scan()))); // Now 'crash' the region by stealing its wal final Configuration newConf = HBaseConfiguration.create(this.conf); @@ -221,6 +345,77 @@ public Object run() throws Exception { newConf, hri, htd, null); long seqid2 = region2.initialize(); assertTrue(seqid2 > -1); + assertEquals(rowsInsertedCount, getScannedCount(region2.getScanner(new Scan()))); + + // I can't close wal1. Its been appropriated when we split. + region2.close(); + wal2.closeAndDelete(); + return null; + } + }); + } + + /** + * HRegion test case that is made of a major compacted HFile (created with three bulk loaded + * files) and an edit in the memstore. + * This is for HBASE-10958 "[dataloss] Bulk loading with seqids can prevent some log entries + * from being replayed" + * @throws IOException + * @throws IllegalAccessException + * @throws NoSuchFieldException + * @throws IllegalArgumentException + * @throws SecurityException + */ + @Test + public void testCompactedBulkLoadedFiles() + throws IOException, SecurityException, IllegalArgumentException, + NoSuchFieldException, IllegalAccessException, InterruptedException { + final String tableNameStr = "testCompactedBulkLoadedFiles"; + final HRegionInfo hri = createBasic3FamilyHRegionInfo(tableNameStr); + final Path basedir = new Path(this.hbaseRootDir, tableNameStr); + deleteDir(basedir); + final HTableDescriptor htd = createBasic3FamilyHTD(tableNameStr); + HRegion region2 = HRegion.createHRegion(hri, + hbaseRootDir, this.conf, htd); + region2.close(); + region2.getLog().closeAndDelete(); + HLog wal = createWAL(this.conf); + HRegion region = HRegion.openHRegion(hri, htd, wal, this.conf); + + // Add an edit so something in the WAL + byte [] row = Bytes.toBytes(tableNameStr); + byte [] family = htd.getFamilies().iterator().next().getName(); + region.put((new Put(row)).add(family, family, family)); + wal.sync(); + + List > hfs= new ArrayList>(1); + for (int i = 0; i < 3; i++) { + Path f = new Path(basedir, "hfile"+i); + HFileTestUtil.createHFile(this.conf, fs, f, family, family, Bytes.toBytes(i + "00"), + Bytes.toBytes(i + "50"), 10); + hfs.add(Pair.newPair(family, f.toString())); + } + region.bulkLoadHFiles(hfs, true); + final int rowsInsertedCount = 31; + assertEquals(rowsInsertedCount, getScannedCount(region.getScanner(new Scan()))); + + // major compact to turn all the bulk loaded files into one normal file + region.compactStores(true); + assertEquals(rowsInsertedCount, getScannedCount(region.getScanner(new Scan()))); + + // Now 'crash' the region by stealing its wal + final Configuration newConf = HBaseConfiguration.create(this.conf); + User user = HBaseTestingUtility.getDifferentUser(newConf, + tableNameStr); + user.runAs(new PrivilegedExceptionAction() { + public Object run() throws Exception { + runWALSplit(newConf); + HLog wal2 = createWAL(newConf); + HRegion region2 = new HRegion(basedir, wal2, FileSystem.get(newConf), + newConf, hri, htd, null); + long seqid2 = region2.initialize(); + assertTrue(seqid2 > -1); + assertEquals(rowsInsertedCount, getScannedCount(region2.getScanner(new Scan()))); // I can't close wal1. Its been appropriated when we split. region2.close(); @@ -252,7 +447,8 @@ public void testReplayEditsWrittenViaHRegion() final HTableDescriptor htd = createBasic3FamilyHTD(tableNameStr); HRegion region3 = HRegion.createHRegion(hri, hbaseRootDir, this.conf, htd); - + region3.close(); + region3.getLog().closeAndDelete(); // Write countPerFamily edits into the three families. Do a flush on one // of the families during the load of edits so its seqid is not same as // others to test we do right thing when different seqids. @@ -369,7 +565,8 @@ public void testReplayEditsAfterPartialFlush() final HTableDescriptor htd = createBasic3FamilyHTD(tableNameStr); HRegion region3 = HRegion.createHRegion(hri, hbaseRootDir, this.conf, htd); - + region3.close(); + region3.getLog().closeAndDelete(); // Write countPerFamily edits into the three families. Do a flush on one // of the families during the load of edits so its seqid is not same as // others to test we do right thing when different seqids. @@ -420,6 +617,129 @@ public void testReplayEditsAfterPartialFlush() assertEquals(result.size(), result1b.size()); } + /** + * Test that we could recover the data correctly after aborting flush. In the + * test, first we abort flush after writing some data, then writing more data + * and flush again, at last verify the data. + * @throws IOException + */ + @Test + public void testReplayEditsAfterAbortingFlush() throws IOException { + final String tableNameStr = "testReplayEditsAfterAbortingFlush"; + final HRegionInfo hri = createBasic3FamilyHRegionInfo(tableNameStr); + final Path basedir = new Path(this.hbaseRootDir, tableNameStr); + deleteDir(basedir); + final HTableDescriptor htd = createBasic3FamilyHTD(tableNameStr); + HRegion region3 = HRegion.createHRegion(hri, hbaseRootDir, this.conf, htd); + region3.close(); + region3.getLog().closeAndDelete(); + // Write countPerFamily edits into the three families. Do a flush on one + // of the families during the load of edits so its seqid is not same as + // others to test we do right thing when different seqids. + HLog wal = createWAL(this.conf); + final AtomicBoolean throwExceptionWhenFlushing = new AtomicBoolean(false); + RegionServerServices rsServices = Mockito.mock(RegionServerServices.class); + Mockito.doReturn(false).when(rsServices).isAborted(); + HRegion region = new HRegion(basedir, wal, this.fs, this.conf, hri, htd, + rsServices) { + @Override + protected Store instantiateHStore(Path tableDir, HColumnDescriptor c) + throws IOException { + return new Store(tableDir, this, c, fs, conf) { + @Override + protected Path flushCache(final long logCacheFlushId, + SortedSet snapshot, + TimeRangeTracker snapshotTimeRangeTracker, + AtomicLong flushedSize, MonitoredTask status) throws IOException { + if (throwExceptionWhenFlushing.get()) { + throw new IOException("Simulated exception by tests"); + } + return super.flushCache(logCacheFlushId, snapshot, + snapshotTimeRangeTracker, flushedSize, status); + } + }; + } + }; + long seqid = region.initialize(); + // HRegionServer usually does this. It knows the largest seqid across all + // regions. + wal.setSequenceNumber(seqid); + + int writtenRowCount = 10; + List families = new ArrayList( + htd.getFamilies()); + for (int i = 0; i < writtenRowCount; i++) { + Put put = new Put(Bytes.toBytes(tableNameStr + Integer.toString(i))); + put.add(families.get(i % families.size()).getName(), Bytes.toBytes("q"), + Bytes.toBytes("val")); + region.put(put); + } + + // Now assert edits made it in. + RegionScanner scanner = region.getScanner(new Scan()); + assertEquals(writtenRowCount, getScannedCount(scanner)); + + // Let us flush the region + throwExceptionWhenFlushing.set(true); + try { + region.flushcache(); + fail("Injected exception hasn't been thrown"); + } catch (Throwable t) { + LOG.info("Expected simulated exception when flushing region," + + t.getMessage()); + // simulated to abort server + Mockito.doReturn(true).when(rsServices).isAborted(); + } + // writing more data + int moreRow = 10; + for (int i = writtenRowCount; i < writtenRowCount + moreRow; i++) { + Put put = new Put(Bytes.toBytes(tableNameStr + Integer.toString(i))); + put.add(families.get(i % families.size()).getName(), Bytes.toBytes("q"), + Bytes.toBytes("val")); + region.put(put); + } + writtenRowCount += moreRow; + // call flush again + throwExceptionWhenFlushing.set(false); + try { + region.flushcache(); + } catch (IOException t) { + LOG.info("Expected exception when flushing region because server is stopped," + + t.getMessage()); + } + + region.close(true); + wal.close(); + + // Let us try to split and recover + runWALSplit(this.conf); + HLog wal2 = createWAL(this.conf); + Mockito.doReturn(false).when(rsServices).isAborted(); + HRegion region2 = new HRegion(basedir, wal2, this.fs, this.conf, hri, htd, + rsServices); + long seqid2 = region2.initialize(); + // HRegionServer usually does this. It knows the largest seqid across all + // regions. + wal2.setSequenceNumber(seqid2); + + scanner = region2.getScanner(new Scan()); + assertEquals(writtenRowCount, getScannedCount(scanner)); + } + + private int getScannedCount(RegionScanner scanner) throws IOException { + int scannedCount = 0; + List results = new ArrayList(); + while (true) { + boolean existMore = scanner.next(results); + if (!results.isEmpty()) + scannedCount++; + if (!existMore) + break; + results.clear(); + } + return scannedCount; + } + /** * Create an HRegion with the result of a HLog split and test we only see the * good edits @@ -435,7 +755,8 @@ public void testReplayEditsWrittenIntoWAL() throws Exception { final HTableDescriptor htd = createBasic3FamilyHTD(tableNameStr); HRegion region2 = HRegion.createHRegion(hri, hbaseRootDir, this.conf, htd); - + region2.close(); + region2.getLog().closeAndDelete(); final HLog wal = createWAL(this.conf); final byte[] tableName = Bytes.toBytes(tableNameStr); final byte[] rowName = tableName; @@ -488,14 +809,14 @@ public Object run() throws Exception { try { final HRegion region = new HRegion(basedir, newWal, newFS, newConf, hri, htd, null) { - protected boolean internalFlushcache( + protected FlushResult internalFlushcache( final HLog wal, final long myseqid, MonitoredTask status) throws IOException { LOG.info("InternalFlushCache Invoked"); - boolean b = super.internalFlushcache(wal, myseqid, + FlushResult fs = super.internalFlushcache(wal, myseqid, Mockito.mock(MonitoredTask.class)); flushcount.incrementAndGet(); - return b; + return fs; }; }; long seqid = region.initialize(); @@ -517,6 +838,91 @@ protected boolean internalFlushcache( }); } + @Test + public void testSequentialEditLogSeqNum() throws IOException { + final String tableNameStr = "testSequentialEditLogSeqNum"; + final HRegionInfo hri = createBasic3FamilyHRegionInfo(tableNameStr); + final Path basedir = new Path(this.hbaseRootDir, tableNameStr); + deleteDir(basedir); + final byte[] rowName = Bytes.toBytes(tableNameStr); + final int countPerFamily = 10; + final HTableDescriptor htd = createBasic1FamilyHTD(tableNameStr); + + // Mock the HLog + MockHLog wal = createMockWAL(this.conf); + + HRegion region = new HRegion(basedir, wal, this.fs, this.conf, hri, htd, null); + long seqid = region.initialize(); + // HRegionServer usually does this. It knows the largest seqid across all + // regions. + wal.setSequenceNumber(seqid); + for (HColumnDescriptor hcd : htd.getFamilies()) { + addRegionEdits(rowName, hcd.getName(), countPerFamily, this.ee, region, "x"); + } + // get the seq no after first set of entries. + long sequenceNumber = wal.getSequenceNumber(); + + // Let us flush the region + // But this time completeflushcache is not yet done + region.flushcache(); + for (HColumnDescriptor hcd : htd.getFamilies()) { + addRegionEdits(rowName, hcd.getName(), 5, this.ee, region, "x"); + } + long lastestSeqNumber = wal.getSequenceNumber(); + // get the current seq no + wal.doCompleteCacheFlush = true; + // allow complete cache flush with the previous seq number got after first + // set of edits. + wal.completeCacheFlush(hri.getEncodedNameAsBytes(), hri.getTableName(), sequenceNumber, false); + wal.close(); + FileStatus[] listStatus = this.fs.listStatus(wal.getDir()); + HLogSplitter.splitLogFile(hbaseRootDir, listStatus[0], this.fs, this.conf, + null); + FileStatus[] listStatus1 = this.fs.listStatus(new Path(hbaseRootDir + "/" + + tableNameStr + "/" + hri.getEncodedName() + "/recovered.edits")); + int editCount = 0; + for (FileStatus fileStatus : listStatus1) { + editCount = Integer.parseInt(fileStatus.getPath().getName()); + } + // The sequence number should be same + assertEquals( + "The sequence number of the recoverd.edits and the current edit seq should be same", + lastestSeqNumber, editCount); + } + + static class MockHLog extends HLog { + boolean doCompleteCacheFlush = false; + + public MockHLog(FileSystem fs, Path dir, Path oldLogDir, Configuration conf) throws IOException { + super(fs, dir, oldLogDir, conf); + } + + @Override + public void completeCacheFlush(byte[] encodedRegionName, byte[] tableName, long logSeqId, + boolean isMetaRegion) throws IOException { + if (!doCompleteCacheFlush) { + return; + } + super.completeCacheFlush(encodedRegionName, tableName, logSeqId, isMetaRegion); + } + } + + private HTableDescriptor createBasic1FamilyHTD(final String tableName) { + HTableDescriptor htd = new HTableDescriptor(tableName); + HColumnDescriptor a = new HColumnDescriptor(Bytes.toBytes("a")); + htd.addFamily(a); + return htd; + } + + private MockHLog createMockWAL(Configuration conf) throws IOException { + MockHLog wal = new MockHLog(FileSystem.get(conf), logDir, oldLogDir, conf); + // Set down maximum recovery so we dfsclient doesn't linger retrying something + // long gone. + HBaseTestingUtility.setMaxRecoveryErrorCount(wal.getOutputStream(), 1); + return wal; + } + + // Flusher used in this test. Keep count of how often we are called and // actually run the flush inside here. class TestFlusher implements FlushRequester { @@ -532,6 +938,12 @@ public void requestFlush(HRegion region) { throw new RuntimeException("Exception flushing", e); } } + + @Override + public void requestDelayedFlush(HRegion region, long when) { + // TODO Auto-generated method stub + + } } private void addWALEdits (final byte [] tableName, final HRegionInfo hri, diff --git a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplayCompressed.java b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplayCompressed.java new file mode 100644 index 000000000000..7e5735967204 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplayCompressed.java @@ -0,0 +1,39 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver.wal; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; +import org.junit.BeforeClass; +import org.junit.experimental.categories.Category; + +/** + * Enables compression and runs the TestWALReplay tests. + */ +@Category(MediumTests.class) +public class TestWALReplayCompressed extends TestWALReplay { + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TestWALReplay.setUpBeforeClass(); + Configuration conf = TestWALReplay.TEST_UTIL.getConfiguration(); + conf.setBoolean(HConstants.ENABLE_WAL_COMPRESSION, true); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java b/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java index 9d3e8620ed7d..3a5baa992757 100644 --- a/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java +++ b/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,6 +18,9 @@ */ package org.apache.hadoop.hbase.replication; +import java.io.IOException; +import java.util.concurrent.atomic.AtomicBoolean; + import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; @@ -26,9 +28,6 @@ import org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceInterface; import org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager; -import java.io.IOException; -import java.util.concurrent.atomic.AtomicBoolean; - /** * Source that does nothing at all, helpful to test ReplicationSourceManager */ @@ -81,10 +80,4 @@ public String getPeerClusterZnode() { public String getPeerClusterId() { return peerClusterId; } - - @Override - public void setSourceEnabled(boolean status) { - - } - } diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java b/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java index 9d7c9c186c12..c90c321a23e2 100644 --- a/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -23,8 +22,10 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.fail; +import java.io.Closeable; import java.io.IOException; import java.util.List; +import java.util.Random; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -53,17 +54,13 @@ @Category(LargeTests.class) public class TestMasterReplication { - private static final Log LOG = LogFactory.getLog(TestReplication.class); + private static final Log LOG = LogFactory.getLog(TestReplicationBase.class); - private Configuration conf1; - private Configuration conf2; - private Configuration conf3; + private Configuration baseConfiguration; - private HBaseTestingUtility utility1; - private HBaseTestingUtility utility2; - private HBaseTestingUtility utility3; - - private MiniZooKeeperCluster miniZK; + private HBaseTestingUtility[] utilities; + private Configuration[] configurations; + private MiniZooKeeperCluster miniZK; private static final long SLEEP_TIME = 500; private static final int NB_RETRIES = 10; @@ -73,6 +70,8 @@ public class TestMasterReplication { private static final byte[] row = Bytes.toBytes("row"); private static final byte[] row1 = Bytes.toBytes("row1"); private static final byte[] row2 = Bytes.toBytes("row2"); + private static final byte[] row3 = Bytes.toBytes("row3"); + private static final byte[] row4 = Bytes.toBytes("row4"); private static final byte[] noRepfamName = Bytes.toBytes("norep"); private static final byte[] count = Bytes.toBytes("count"); @@ -83,44 +82,21 @@ public class TestMasterReplication { @Before public void setUp() throws Exception { - conf1 = HBaseConfiguration.create(); - conf1.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/1"); + baseConfiguration = HBaseConfiguration.create(); // smaller block size and capacity to trigger more operations // and test them - conf1.setInt("hbase.regionserver.hlog.blocksize", 1024*20); - conf1.setInt("replication.source.size.capacity", 1024); - conf1.setLong("replication.source.sleepforretries", 100); - conf1.setInt("hbase.regionserver.maxlogs", 10); - conf1.setLong("hbase.master.logcleaner.ttl", 10); - conf1.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true); - conf1.setBoolean("dfs.support.append", true); - conf1.setLong(HConstants.THREAD_WAKE_FREQUENCY, 100); - conf1.setStrings(CoprocessorHost.USER_REGION_COPROCESSOR_CONF_KEY, + baseConfiguration.setInt("hbase.regionserver.hlog.blocksize", 1024 * 20); + baseConfiguration.setInt("replication.source.size.capacity", 1024); + baseConfiguration.setLong("replication.source.sleepforretries", 100); + baseConfiguration.setInt("hbase.regionserver.maxlogs", 10); + baseConfiguration.setLong("hbase.master.logcleaner.ttl", 10); + baseConfiguration.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true); + baseConfiguration.setBoolean("dfs.support.append", true); + baseConfiguration.setLong(HConstants.THREAD_WAKE_FREQUENCY, 100); + baseConfiguration.setStrings( + CoprocessorHost.USER_REGION_COPROCESSOR_CONF_KEY, CoprocessorCounter.class.getName()); - utility1 = new HBaseTestingUtility(conf1); - utility1.startMiniZKCluster(); - miniZK = utility1.getZkCluster(); - // By setting the mini ZK cluster through this method, even though this is - // already utility1's mini ZK cluster, we are telling utility1 not to shut - // the mini ZK cluster when we shut down the HBase cluster. - utility1.setZkCluster(miniZK); - new ZooKeeperWatcher(conf1, "cluster1", null, true); - - conf2 = new Configuration(conf1); - conf2.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/2"); - - utility2 = new HBaseTestingUtility(conf2); - utility2.setZkCluster(miniZK); - new ZooKeeperWatcher(conf2, "cluster2", null, true); - - conf3 = new Configuration(conf1); - conf3.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/3"); - - utility3 = new HBaseTestingUtility(conf3); - utility3.setZkCluster(miniZK); - new ZooKeeperWatcher(conf3, "cluster3", null, true); - table = new HTableDescriptor(tableName); HColumnDescriptor fam = new HColumnDescriptor(famName); fam.setScope(HConstants.REPLICATION_SCOPE_GLOBAL); @@ -129,159 +105,297 @@ public void setUp() throws Exception { table.addFamily(fam); } - @After - public void tearDown() throws IOException { - miniZK.shutdown(); + /** + * It tests the replication scenario involving 0 -> 1 -> 0. It does it by + * adding and deleting a row to a table in each cluster, checking if it's + * replicated. It also tests that the puts and deletes are not replicated back + * to the originating cluster. + */ + @Test(timeout = 300000) + public void testCyclicReplication1() throws Exception { + LOG.info("testSimplePutDelete"); + int numClusters = 2; + HTable[] htables = null; + try { + startMiniClusters(numClusters); + createTableOnClusters(table); + + htables = getHTablesOnClusters(tableName); + + // Test the replication scenarios of 0 -> 1 -> 0 + addPeer("1", 0, 1); + addPeer("1", 1, 0); + + int[] expectedCounts = new int[] { 2, 2 }; + + // add rows to both clusters, + // make sure they are both replication + putAndWait(row, famName, htables[0], htables[1]); + putAndWait(row1, famName, htables[1], htables[0]); + validateCounts(htables, put, expectedCounts); + + deleteAndWait(row, htables[0], htables[1]); + deleteAndWait(row1, htables[1], htables[0]); + validateCounts(htables, delete, expectedCounts); + } finally { + close(htables); + shutDownMiniClusters(); + } } - @Test(timeout=300000) - public void testCyclicReplication() throws Exception { - LOG.info("testCyclicReplication"); - utility1.startMiniCluster(); - utility2.startMiniCluster(); - utility3.startMiniCluster(); - ReplicationAdmin admin1 = new ReplicationAdmin(conf1); - ReplicationAdmin admin2 = new ReplicationAdmin(conf2); - ReplicationAdmin admin3 = new ReplicationAdmin(conf3); - - new HBaseAdmin(conf1).createTable(table); - new HBaseAdmin(conf2).createTable(table); - new HBaseAdmin(conf3).createTable(table); - HTable htable1 = new HTable(conf1, tableName); - htable1.setWriteBufferSize(1024); - HTable htable2 = new HTable(conf2, tableName); - htable2.setWriteBufferSize(1024); - HTable htable3 = new HTable(conf3, tableName); - htable3.setWriteBufferSize(1024); - - admin1.addPeer("1", utility2.getClusterKey()); - admin2.addPeer("1", utility3.getClusterKey()); - admin3.addPeer("1", utility1.getClusterKey()); - - // put "row" and wait 'til it got around - putAndWait(row, famName, htable1, htable3); - // it should have passed through table2 - check(row,famName,htable2); - - putAndWait(row1, famName, htable2, htable1); - check(row,famName,htable3); - putAndWait(row2, famName, htable3, htable2); - check(row,famName,htable1); - - deleteAndWait(row,htable1,htable3); - deleteAndWait(row1,htable2,htable1); - deleteAndWait(row2,htable3,htable2); - - assertEquals("Puts were replicated back ", 3, getCount(htable1, put)); - assertEquals("Puts were replicated back ", 3, getCount(htable2, put)); - assertEquals("Puts were replicated back ", 3, getCount(htable3, put)); - assertEquals("Deletes were replicated back ", 3, getCount(htable1, delete)); - assertEquals("Deletes were replicated back ", 3, getCount(htable2, delete)); - assertEquals("Deletes were replicated back ", 3, getCount(htable3, delete)); - utility3.shutdownMiniCluster(); - utility2.shutdownMiniCluster(); - utility1.shutdownMiniCluster(); + /** + * Tests the cyclic replication scenario of 0 -> 1 -> 2 -> 0 by adding and + * deleting rows to a table in each clusters and ensuring that the each of + * these clusters get the appropriate mutations. It also tests the grouping + * scenario where a cluster needs to replicate the edits originating from + * itself and also the edits that it received using replication from a + * different cluster. The scenario is explained in HBASE-9158 + */ + @Test(timeout = 300000) + public void testCyclicReplication2() throws Exception { + LOG.info("testCyclicReplication1"); + int numClusters = 3; + HTable[] htables = null; + try { + startMiniClusters(numClusters); + createTableOnClusters(table); + + // Test the replication scenario of 0 -> 1 -> 2 -> 0 + addPeer("1", 0, 1); + addPeer("1", 1, 2); + addPeer("1", 2, 0); + + htables = getHTablesOnClusters(tableName); + + // put "row" and wait 'til it got around + putAndWait(row, famName, htables[0], htables[2]); + putAndWait(row1, famName, htables[1], htables[0]); + putAndWait(row2, famName, htables[2], htables[1]); + + deleteAndWait(row, htables[0], htables[2]); + deleteAndWait(row1, htables[1], htables[0]); + deleteAndWait(row2, htables[2], htables[1]); + + int[] expectedCounts = new int[] { 3, 3, 3 }; + validateCounts(htables, put, expectedCounts); + validateCounts(htables, delete, expectedCounts); + + // Test HBASE-9158 + disablePeer("1", 2); + // we now have an edit that was replicated into cluster originating from + // cluster 0 + putAndWait(row3, famName, htables[0], htables[1]); + // now add a local edit to cluster 1 + htables[1].put(new Put(row4).add(famName, row4, row4)); + // re-enable replication from cluster 2 to cluster 0 + enablePeer("1", 2); + // without HBASE-9158 the edit for row4 would have been marked with + // cluster 0's id + // and hence not replicated to cluster 0 + wait(row4, htables[0], true); + } finally { + close(htables); + shutDownMiniClusters(); + } } /** - * Add a row to a table in each cluster, check it's replicated, - * delete it, check's gone - * Also check the puts and deletes are not replicated back to - * the originating cluster. + * Tests cyclic replication scenario of 0 -> 1 -> 2 -> 1. */ - @Test(timeout=300000) - public void testSimplePutDelete() throws Exception { - LOG.info("testSimplePutDelete"); - utility1.startMiniCluster(); - utility2.startMiniCluster(); + @Test(timeout = 300000) + public void testCyclicReplication3() throws Exception { + LOG.info("testCyclicReplication2"); + int numClusters = 3; + HTable[] htables = null; + try { + startMiniClusters(numClusters); + createTableOnClusters(table); + + // Test the replication scenario of 0 -> 1 -> 2 -> 1 + addPeer("1", 0, 1); + addPeer("1", 1, 2); + addPeer("1", 2, 1); + + htables = getHTablesOnClusters(tableName); + + // put "row" and wait 'til it got around + putAndWait(row, famName, htables[0], htables[2]); + putAndWait(row1, famName, htables[1], htables[2]); + putAndWait(row2, famName, htables[2], htables[1]); + + deleteAndWait(row, htables[0], htables[2]); + deleteAndWait(row1, htables[1], htables[2]); + deleteAndWait(row2, htables[2], htables[1]); + + int[] expectedCounts = new int[] { 1, 3, 3 }; + validateCounts(htables, put, expectedCounts); + validateCounts(htables, delete, expectedCounts); + } finally { + close(htables); + shutDownMiniClusters(); + } + } - ReplicationAdmin admin1 = new ReplicationAdmin(conf1); - ReplicationAdmin admin2 = new ReplicationAdmin(conf2); + @After + public void tearDown() throws IOException { + configurations = null; + utilities = null; + } - new HBaseAdmin(conf1).createTable(table); - new HBaseAdmin(conf2).createTable(table); - HTable htable1 = new HTable(conf1, tableName); - htable1.setWriteBufferSize(1024); - HTable htable2 = new HTable(conf2, tableName); - htable2.setWriteBufferSize(1024); + @SuppressWarnings("resource") + private void startMiniClusters(int numClusters) throws Exception { + Random random = new Random(); + utilities = new HBaseTestingUtility[numClusters]; + configurations = new Configuration[numClusters]; + for (int i = 0; i < numClusters; i++) { + Configuration conf = new Configuration(baseConfiguration); + conf.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/" + i + random.nextInt()); + HBaseTestingUtility utility = new HBaseTestingUtility(conf); + if (i == 0) { + utility.startMiniZKCluster(); + miniZK = utility.getZkCluster(); + } else { + utility.setZkCluster(miniZK); + } + utility.startMiniCluster(); + utilities[i] = utility; + configurations[i] = conf; + new ZooKeeperWatcher(conf, "cluster" + i, null, true); + } + } + + private void shutDownMiniClusters() throws Exception { + int numClusters = utilities.length; + for (int i = numClusters - 1; i >= 0; i--) { + if (utilities[i] != null) { + utilities[i].shutdownMiniCluster(); + } + } + miniZK.shutdown(); + } - // set M-M - admin1.addPeer("1", utility2.getClusterKey()); - admin2.addPeer("1", utility1.getClusterKey()); + private void createTableOnClusters(HTableDescriptor table) throws Exception { + int numClusters = configurations.length; + for (int i = 0; i < numClusters; i++) { + HBaseAdmin hbaseAdmin = null; + try { + hbaseAdmin = new HBaseAdmin(configurations[i]); + hbaseAdmin.createTable(table); + } finally { + close(hbaseAdmin); + } + } + } - // add rows to both clusters, - // make sure they are both replication - putAndWait(row, famName, htable1, htable2); - putAndWait(row1, famName, htable2, htable1); + private void addPeer(String id, int masterClusterNumber, + int slaveClusterNumber) throws Exception { + ReplicationAdmin replicationAdmin = null; + try { + replicationAdmin = new ReplicationAdmin( + configurations[masterClusterNumber]); + replicationAdmin.addPeer(id, + utilities[slaveClusterNumber].getClusterKey()); + } finally { + close(replicationAdmin); + } + } - // make sure "row" did not get replicated back. - assertEquals("Puts were replicated back ", 2, getCount(htable1, put)); + private void disablePeer(String id, int masterClusterNumber) throws Exception { + ReplicationAdmin replicationAdmin = null; + try { + replicationAdmin = new ReplicationAdmin( + configurations[masterClusterNumber]); + replicationAdmin.disablePeer(id); + } finally { + close(replicationAdmin); + } + } - // delete "row" and wait - deleteAndWait(row, htable1, htable2); + private void enablePeer(String id, int masterClusterNumber) throws Exception { + ReplicationAdmin replicationAdmin = null; + try { + replicationAdmin = new ReplicationAdmin( + configurations[masterClusterNumber]); + replicationAdmin.enablePeer(id); + } finally { + close(replicationAdmin); + } + } - // make the 2nd cluster replicated back - assertEquals("Puts were replicated back ", 2, getCount(htable2, put)); + private void close(Closeable... closeables) { + try { + if (closeables != null) { + for (Closeable closeable : closeables) { + closeable.close(); + } + } + } catch (Exception e) { + LOG.warn("Exception occured while closing the object:", e); + } + } - deleteAndWait(row1, htable2, htable1); + @SuppressWarnings("resource") + private HTable[] getHTablesOnClusters(byte[] tableName) throws Exception { + int numClusters = utilities.length; + HTable[] htables = new HTable[numClusters]; + for (int i = 0; i < numClusters; i++) { + HTable htable = new HTable(configurations[i], tableName); + htable.setWriteBufferSize(1024); + htables[i] = htable; + } + return htables; + } - assertEquals("Deletes were replicated back ", 2, getCount(htable1, delete)); - utility2.shutdownMiniCluster(); - utility1.shutdownMiniCluster(); + private void validateCounts(HTable[] htables, byte[] type, + int[] expectedCounts) throws IOException { + for (int i = 0; i < htables.length; i++) { + assertEquals(Bytes.toString(type) + " were replicated back ", + expectedCounts[i], getCount(htables[i], type)); + } } - private int getCount(HTable t, byte[] type) throws IOException { + private int getCount(HTable t, byte[] type) throws IOException { Get test = new Get(row); - test.setAttribute("count", new byte[]{}); + test.setAttribute("count", new byte[] {}); Result res = t.get(test); return Bytes.toInt(res.getValue(count, type)); } private void deleteAndWait(byte[] row, HTable source, HTable target) - throws Exception { + throws Exception { Delete del = new Delete(row); source.delete(del); - - Get get = new Get(row); - for (int i = 0; i < NB_RETRIES; i++) { - if (i==NB_RETRIES-1) { - fail("Waited too much time for del replication"); - } - Result res = target.get(get); - if (res.size() >= 1) { - LOG.info("Row not deleted"); - Thread.sleep(SLEEP_TIME); - } else { - break; - } - } - } - - private void check(byte[] row, byte[] fam, HTable t) throws IOException { - Get get = new Get(row); - Result res = t.get(get); - if (res.size() == 0) { - fail("Row is missing"); - } + wait(row, target, true); } private void putAndWait(byte[] row, byte[] fam, HTable source, HTable target) - throws Exception { + throws Exception { Put put = new Put(row); put.add(fam, row, row); source.put(put); + wait(row, target, false); + } + private void wait(byte[] row, HTable target, boolean isDeleted) + throws Exception { Get get = new Get(row); for (int i = 0; i < NB_RETRIES; i++) { - if (i==NB_RETRIES-1) { - fail("Waited too much time for put replication"); + if (i == NB_RETRIES - 1) { + fail("Waited too much time for replication. Row:" + Bytes.toString(row) + + ". IsDeleteReplication:" + isDeleted); } Result res = target.get(get); - if (res.size() == 0) { - LOG.info("Row not available"); + boolean sleep = isDeleted ? res.size() > 0 : res.size() == 0; + if (sleep) { + LOG.info("Waiting for more time for replication. Row:" + + Bytes.toString(row) + ". IsDeleteReplication:" + isDeleted); Thread.sleep(SLEEP_TIME); } else { - assertArrayEquals(res.value(), row); + if (!isDeleted) { + assertArrayEquals(res.value(), row); + } + LOG.info("Obtained row:" + + Bytes.toString(row) + ". IsDeleteReplication:" + isDeleted); break; } } diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java b/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java index 147a3d25cd3f..1672d978e308 100644 --- a/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java @@ -47,7 +47,7 @@ @Category(LargeTests.class) public class TestMultiSlaveReplication { - private static final Log LOG = LogFactory.getLog(TestReplication.class); + private static final Log LOG = LogFactory.getLog(TestReplicationBase.class); private static Configuration conf1; private static Configuration conf2; @@ -173,7 +173,10 @@ public void testMultiSlaveReplication() throws Exception { deleteAndWait(row2, htable1, htable2, htable3); // Even if the log was rolled in the middle of the replication // "row" is still replication. - checkRow(row, 1, htable2, htable3); + checkRow(row, 1, htable2); + // Replication thread of cluster 2 may be sleeping, and since row2 is not there in it, + // we should wait before checking. + checkWithWait(row, 1, htable3); // cleanup the rest deleteAndWait(row, htable1, htable2, htable3); @@ -183,7 +186,29 @@ public void testMultiSlaveReplication() throws Exception { utility2.shutdownMiniCluster(); utility1.shutdownMiniCluster(); } - + + private void checkWithWait(byte[] row, int count, HTable table) throws Exception { + Get get = new Get(row); + for (int i = 0; i < NB_RETRIES; i++) { + if (i == NB_RETRIES - 1) { + fail("Waited too much time while getting the row."); + } + boolean rowReplicated = false; + Result res = table.get(get); + if (res.size() >= 1) { + LOG.info("Row is replicated"); + rowReplicated = true; + assertEquals(count, res.size()); + break; + } + if (rowReplicated) { + break; + } else { + Thread.sleep(SLEEP_TIME); + } + } + } + private void checkRow(byte[] row, int count, HTable... tables) throws IOException { Get get = new Get(row); for (HTable table : tables) { diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java new file mode 100644 index 000000000000..aabaedc77b75 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java @@ -0,0 +1,154 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.replication; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.replication.ReplicationAdmin; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.junit.AfterClass; +import org.junit.BeforeClass; + +/** + * This class is only a base for other integration-level replication tests. + * Do not add tests here. + * TestReplicationSmallTests is where tests that don't require bring machines up/down should go + * All other tests should have their own classes and extend this one + */ +public class TestReplicationBase { + + private static final Log LOG = LogFactory.getLog(TestReplicationBase.class); + + protected static Configuration conf1 = HBaseConfiguration.create(); + protected static Configuration conf2; + protected static Configuration CONF_WITH_LOCALFS; + + protected static ZooKeeperWatcher zkw1; + protected static ZooKeeperWatcher zkw2; + + protected static ReplicationAdmin admin; + + protected static HTable htable1; + protected static HTable htable2; + + protected static HBaseTestingUtility utility1; + protected static HBaseTestingUtility utility2; + protected static final int NB_ROWS_IN_BATCH = 100; + protected static final int NB_ROWS_IN_BIG_BATCH = + NB_ROWS_IN_BATCH * 10; + protected static final long SLEEP_TIME = 1000; + protected static final int NB_RETRIES = 15; + protected static final int NB_RETRIES_FOR_BIG_BATCH = 30; + + protected static final byte[] tableName = Bytes.toBytes("test"); + protected static final byte[] famName = Bytes.toBytes("f"); + protected static final byte[] row = Bytes.toBytes("row"); + protected static final byte[] noRepfamName = Bytes.toBytes("norep"); + + /** + * @throws java.lang.Exception + */ + @BeforeClass + public static void setUpBeforeClass() throws Exception { + conf1.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/1"); + // smaller log roll size to trigger more events + conf1.setFloat("hbase.regionserver.logroll.multiplier", 0.0003f); + conf1.setInt("replication.source.size.capacity", 10240); + conf1.setLong("replication.source.sleepforretries", 100); + conf1.setInt("hbase.regionserver.maxlogs", 10); + conf1.setLong("hbase.master.logcleaner.ttl", 10); + conf1.setInt("zookeeper.recovery.retry", 1); + conf1.setInt("zookeeper.recovery.retry.intervalmill", 10); + conf1.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true); + conf1.setBoolean("dfs.support.append", true); + conf1.setLong(HConstants.THREAD_WAKE_FREQUENCY, 100); + conf1.setInt("replication.stats.thread.period.seconds", 5); + + utility1 = new HBaseTestingUtility(conf1); + utility1.startMiniZKCluster(); + MiniZooKeeperCluster miniZK = utility1.getZkCluster(); + // Have to reget conf1 in case zk cluster location different + // than default + conf1 = utility1.getConfiguration(); + zkw1 = new ZooKeeperWatcher(conf1, "cluster1", null, true); + admin = new ReplicationAdmin(conf1); + LOG.info("Setup first Zk"); + + // Base conf2 on conf1 so it gets the right zk cluster. + conf2 = HBaseConfiguration.create(conf1); + conf2.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/2"); + conf2.setInt("hbase.client.retries.number", 6); + conf2.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true); + conf2.setBoolean("dfs.support.append", true); + + utility2 = new HBaseTestingUtility(conf2); + utility2.setZkCluster(miniZK); + zkw2 = new ZooKeeperWatcher(conf2, "cluster2", null, true); + + admin.addPeer("2", utility2.getClusterKey()); + setIsReplication(true); + + LOG.info("Setup second Zk"); + CONF_WITH_LOCALFS = HBaseConfiguration.create(conf1); + utility1.startMiniCluster(2); + utility2.startMiniCluster(2); + + HTableDescriptor table = new HTableDescriptor(tableName); + HColumnDescriptor fam = new HColumnDescriptor(famName); + fam.setScope(HConstants.REPLICATION_SCOPE_GLOBAL); + table.addFamily(fam); + fam = new HColumnDescriptor(noRepfamName); + table.addFamily(fam); + HBaseAdmin admin1 = new HBaseAdmin(conf1); + HBaseAdmin admin2 = new HBaseAdmin(conf2); + admin1.createTable(table, HBaseTestingUtility.KEYS_FOR_HBA_CREATE_TABLE); + admin2.createTable(table); + htable1 = new HTable(conf1, tableName); + htable1.setWriteBufferSize(1024); + htable2 = new HTable(conf2, tableName); + } + + protected static void setIsReplication(boolean rep) throws Exception { + LOG.info("Set rep " + rep); + admin.setReplicating(rep); + Thread.sleep(SLEEP_TIME); + } + + /** + * @throws java.lang.Exception + */ + @AfterClass + public static void tearDownAfterClass() throws Exception { + utility2.shutdownMiniCluster(); + utility1.shutdownMiniCluster(); + } + + +} + diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java new file mode 100644 index 000000000000..b089fbe8f298 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java @@ -0,0 +1,92 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.replication; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.fail; + +@Category(LargeTests.class) +public class TestReplicationDisableInactivePeer extends TestReplicationBase { + + private static final Log LOG = LogFactory.getLog(TestReplicationDisableInactivePeer.class); + + /** + * Test disabling an inactive peer. Add a peer which is inactive, trying to + * insert, disable the peer, then activate the peer and make sure nothing is + * replicated. In Addition, enable the peer and check the updates are + * replicated. + * + * @throws Exception + */ + @Test(timeout = 600000) + public void testDisableInactivePeer() throws Exception { + + // enabling and shutdown the peer + admin.enablePeer("2"); + utility2.shutdownMiniHBaseCluster(); + + byte[] rowkey = Bytes.toBytes("disable inactive peer"); + Put put = new Put(rowkey); + put.add(famName, row, row); + htable1.put(put); + + // wait for the sleep interval of the master cluster to become long + Thread.sleep(SLEEP_TIME * NB_RETRIES); + + // disable and start the peer + admin.disablePeer("2"); + utility2.startMiniHBaseCluster(1, 2); + Get get = new Get(rowkey); + for (int i = 0; i < NB_RETRIES; i++) { + Result res = htable2.get(get); + if (res.size() >= 1) { + fail("Replication wasn't disabled"); + } else { + LOG.info("Row not replicated, let's wait a bit more..."); + Thread.sleep(SLEEP_TIME); + } + } + + // Test enable replication + admin.enablePeer("2"); + // wait since the sleep interval would be long + Thread.sleep(SLEEP_TIME * NB_RETRIES); + for (int i = 0; i < NB_RETRIES; i++) { + Result res = htable2.get(get); + if (res.size() == 0) { + LOG.info("Row not available"); + Thread.sleep(SLEEP_TIME * NB_RETRIES); + } else { + assertArrayEquals(res.value(), row); + return; + } + } + fail("Waited too much time for put replication"); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java deleted file mode 100644 index 51bd6fa54cb9..000000000000 --- a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java +++ /dev/null @@ -1,82 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.hbase.replication; - -import junit.framework.Assert; - -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; -import org.apache.zookeeper.KeeperException.SessionExpiredException; -import org.junit.*; -import org.junit.experimental.categories.Category; - -@Category(MediumTests.class) -public class TestReplicationPeer { - - private static final Log LOG = LogFactory.getLog(TestReplicationPeer.class); - private static HBaseTestingUtility utility; - private static Configuration conf; - private static ReplicationPeer rp; - - @BeforeClass - public static void setUpBeforeClass() throws Exception { - conf = HBaseConfiguration.create(); - utility = new HBaseTestingUtility(conf); - conf = utility.getConfiguration(); - utility.startMiniZKCluster(); - - rp = new ReplicationPeer(conf, "clusterKey", "clusterId"); - } - - @Test(timeout=300000) - public void testResetZooKeeperSession() throws Exception { - ZooKeeperWatcher zkw = rp.getZkw(); - zkw.getRecoverableZooKeeper().exists("/1/2", false); - - LOG.info("Expiring ReplicationPeer ZooKeeper session."); - utility.expireSession(zkw, null, false); - - try { - LOG.info("Attempting to use expired ReplicationPeer ZooKeeper session."); - // Trying to use the expired session to assert that it is indeed closed - zkw.getRecoverableZooKeeper().exists("/1/2", false); - } catch (SessionExpiredException k) { - rp.reloadZkWatcher(); - - zkw = rp.getZkw(); - - // Try to use the connection again - LOG.info("Attempting to use refreshed " - + "ReplicationPeer ZooKeeper session."); - zkw.getRecoverableZooKeeper().exists("/1/2", false); - - return; - } - - Assert.fail("ReplicationPeer ZooKeeper session was not properly expired."); - } - - - @org.junit.Rule - public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = - new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); -} - diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationQueueFailover.java b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationQueueFailover.java new file mode 100644 index 000000000000..9e7e0f555eac --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationQueueFailover.java @@ -0,0 +1,133 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.replication; + + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.UnknownScannerException; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import static org.junit.Assert.fail; + +@Category(LargeTests.class) +public class TestReplicationQueueFailover extends TestReplicationBase { + + private static final Log LOG = LogFactory.getLog(TestReplicationQueueFailover.class); + + /** + * Load up multiple tables over 2 region servers and kill a source during + * the upload. The failover happens internally. + * + * WARNING this test sometimes fails because of HBASE-3515 + * + * @throws Exception + */ + @Test(timeout=300000) + public void queueFailover() throws Exception { + // killing the RS with .META. can result into failed puts until we solve + // IO fencing + int rsToKill1 = + utility1.getHBaseCluster().getServerWithMeta() == 0 ? 1 : 0; + int rsToKill2 = + utility2.getHBaseCluster().getServerWithMeta() == 0 ? 1 : 0; + + // Takes about 20 secs to run the full loading, kill around the middle + Thread killer1 = killARegionServer(utility1, 7500, rsToKill1); + Thread killer2 = killARegionServer(utility2, 10000, rsToKill2); + + LOG.info("Start loading table"); + int initialCount = utility1.loadTable(htable1, famName); + LOG.info("Done loading table"); + killer1.join(5000); + killer2.join(5000); + LOG.info("Done waiting for threads"); + + Result[] res; + while (true) { + try { + Scan scan = new Scan(); + ResultScanner scanner = htable1.getScanner(scan); + res = scanner.next(initialCount); + scanner.close(); + break; + } catch (UnknownScannerException ex) { + LOG.info("Cluster wasn't ready yet, restarting scanner"); + } + } + // Test we actually have all the rows, we may miss some because we + // don't have IO fencing. + if (res.length != initialCount) { + LOG.warn("We lost some rows on the master cluster!"); + // We don't really expect the other cluster to have more rows + initialCount = res.length; + } + + int lastCount = 0; + + final long start = System.currentTimeMillis(); + int i = 0; + while (true) { + if (i==NB_RETRIES-1) { + fail("Waited too much time for queueFailover replication. " + + "Waited "+(System.currentTimeMillis() - start)+"ms."); + } + Scan scan2 = new Scan(); + ResultScanner scanner2 = htable2.getScanner(scan2); + Result[] res2 = scanner2.next(initialCount * 2); + scanner2.close(); + if (res2.length < initialCount) { + if (lastCount < res2.length) { + i--; // Don't increment timeout if we make progress + } else { + i++; + } + lastCount = res2.length; + LOG.info("Only got " + lastCount + " rows instead of " + + initialCount + " current i=" + i); + Thread.sleep(SLEEP_TIME*2); + } else { + break; + } + } + } + + private static Thread killARegionServer(final HBaseTestingUtility utility, + final long timeout, final int rs) { + Thread killer = new Thread() { + public void run() { + try { + Thread.sleep(timeout); + utility.expireRegionServerSession(rs); + } catch (Exception e) { + LOG.error("Couldn't kill a region server", e); + } + } + }; + killer.setDaemon(true); + killer.start(); + return killer; + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationQueueFailoverCompressed.java b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationQueueFailoverCompressed.java new file mode 100644 index 000000000000..35c0715cb1e1 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationQueueFailoverCompressed.java @@ -0,0 +1,40 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.replication; + +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.LargeTests; +import org.junit.BeforeClass; +import org.junit.experimental.categories.Category; + +/** + * Run the same test as TestReplication but with HLog compression enabled + */ +@Category(LargeTests.class) +public class TestReplicationQueueFailoverCompressed extends TestReplicationQueueFailover { + + /** + * @throws java.lang.Exception + */ + @BeforeClass + public static void setUpBeforeClass() throws Exception { + conf1.setBoolean(HConstants.ENABLE_WAL_COMPRESSION, true); + TestReplicationBase.setUpBeforeClass(); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java similarity index 65% rename from src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java rename to src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java index a9ae7cac46e0..1b47ab676b18 100644 --- a/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java @@ -1,6 +1,4 @@ -/* - * Copyright 2010 The Apache Software Foundation - * +/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information @@ -19,140 +17,43 @@ */ package org.apache.hadoop.hbase.replication; -import static org.junit.Assert.assertArrayEquals; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.fail; +import java.util.HashMap; +import java.util.List; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.client.Delete; -import org.apache.hadoop.hbase.client.Get; -import org.apache.hadoop.hbase.client.HBaseAdmin; -import org.apache.hadoop.hbase.client.HTable; -import org.apache.hadoop.hbase.client.Put; -import org.apache.hadoop.hbase.client.Result; -import org.apache.hadoop.hbase.client.ResultScanner; -import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.*; import org.apache.hadoop.hbase.client.replication.ReplicationAdmin; import org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; import org.apache.hadoop.hbase.util.JVMClusterUtil; -import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster; -import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.apache.hadoop.mapreduce.Job; -import org.junit.AfterClass; import org.junit.Before; -import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; -@Category(LargeTests.class) -public class TestReplication { - - private static final Log LOG = LogFactory.getLog(TestReplication.class); - - private static Configuration conf1; - private static Configuration conf2; - private static Configuration CONF_WITH_LOCALFS; - - private static ZooKeeperWatcher zkw1; - private static ZooKeeperWatcher zkw2; - - private static ReplicationAdmin admin; - - private static HTable htable1; - private static HTable htable2; +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.fail; +import static org.junit.Assert.assertTrue; - private static HBaseTestingUtility utility1; - private static HBaseTestingUtility utility2; - private static final int NB_ROWS_IN_BATCH = 100; - private static final int NB_ROWS_IN_BIG_BATCH = - NB_ROWS_IN_BATCH * 10; - private static final long SLEEP_TIME = 500; - private static final int NB_RETRIES = 10; +@Category(LargeTests.class) +public class TestReplicationSmallTests extends TestReplicationBase { - private static final byte[] tableName = Bytes.toBytes("test"); - private static final byte[] famName = Bytes.toBytes("f"); - private static final byte[] row = Bytes.toBytes("row"); - private static final byte[] noRepfamName = Bytes.toBytes("norep"); - - /** - * @throws java.lang.Exception - */ - @BeforeClass - public static void setUpBeforeClass() throws Exception { - conf1 = HBaseConfiguration.create(); - conf1.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/1"); - // smaller block size and capacity to trigger more operations - // and test them - conf1.setInt("hbase.regionserver.hlog.blocksize", 1024*20); - conf1.setInt("replication.source.size.capacity", 1024); - conf1.setLong("replication.source.sleepforretries", 100); - conf1.setInt("hbase.regionserver.maxlogs", 10); - conf1.setLong("hbase.master.logcleaner.ttl", 10); - conf1.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true); - conf1.setBoolean("dfs.support.append", true); - conf1.setLong(HConstants.THREAD_WAKE_FREQUENCY, 100); - - utility1 = new HBaseTestingUtility(conf1); - utility1.startMiniZKCluster(); - MiniZooKeeperCluster miniZK = utility1.getZkCluster(); - // Have to reget conf1 in case zk cluster location different - // than default - conf1 = utility1.getConfiguration(); - zkw1 = new ZooKeeperWatcher(conf1, "cluster1", null, true); - admin = new ReplicationAdmin(conf1); - LOG.info("Setup first Zk"); - - // Base conf2 on conf1 so it gets the right zk cluster. - conf2 = HBaseConfiguration.create(conf1); - conf2.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/2"); - conf2.setInt("hbase.client.retries.number", 6); - conf2.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true); - conf2.setBoolean("dfs.support.append", true); - - utility2 = new HBaseTestingUtility(conf2); - utility2.setZkCluster(miniZK); - zkw2 = new ZooKeeperWatcher(conf2, "cluster2", null, true); - - admin.addPeer("2", utility2.getClusterKey()); - setIsReplication(true); - - LOG.info("Setup second Zk"); - CONF_WITH_LOCALFS = HBaseConfiguration.create(conf1); - utility1.startMiniCluster(2); - utility2.startMiniCluster(2); - - HTableDescriptor table = new HTableDescriptor(tableName); - HColumnDescriptor fam = new HColumnDescriptor(famName); - fam.setScope(HConstants.REPLICATION_SCOPE_GLOBAL); - table.addFamily(fam); - fam = new HColumnDescriptor(noRepfamName); - table.addFamily(fam); - HBaseAdmin admin1 = new HBaseAdmin(conf1); - HBaseAdmin admin2 = new HBaseAdmin(conf2); - admin1.createTable(table); - admin2.createTable(table); - htable1 = new HTable(conf1, tableName); - htable1.setWriteBufferSize(1024); - htable2 = new HTable(conf2, tableName); - } - - private static void setIsReplication(boolean rep) throws Exception { - LOG.info("Set rep " + rep); - admin.setReplicating(rep); - Thread.sleep(SLEEP_TIME); - } + private static final Log LOG = LogFactory.getLog(TestReplicationSmallTests.class); /** * @throws java.lang.Exception */ @Before public void setUp() throws Exception { - + htable1.setAutoFlush(true); // Starting and stopping replication can make us miss new logs, // rolling like this makes sure the most recent one gets added to the queue for ( JVMClusterUtil.RegionServerThread r : @@ -175,7 +76,7 @@ public void setUp() throws Exception { Result[] res = scanner.next(NB_ROWS_IN_BIG_BATCH); scanner.close(); if (res.length != 0) { - if (res.length < lastCount) { + if (res.length < lastCount) { i--; // Don't increment timeout if we make progress } lastCount = res.length; @@ -187,15 +88,6 @@ public void setUp() throws Exception { } } - /** - * @throws java.lang.Exception - */ - @AfterClass - public static void tearDownAfterClass() throws Exception { - utility2.shutdownMiniCluster(); - utility1.shutdownMiniCluster(); - } - /** * Verify that version and column delete marker types are replicated * correctly. @@ -218,11 +110,11 @@ public void testDeleteTypes() throws Exception { put = new Put(row); put.add(famName, row, t+1, v2); htable1.put(put); - + put = new Put(row); put.add(famName, row, t+2, v3); htable1.put(put); - + Get get = new Get(row); get.setMaxVersions(); for (int i = 0; i < NB_RETRIES; i++) { @@ -368,9 +260,6 @@ public void testSmallBatch() throws Exception { break; } } - - htable1.setAutoFlush(true); - } /** @@ -442,9 +331,54 @@ public void testStartStop() throws Exception { } + /** + * Test disable/enable replication, trying to insert, make sure nothing's + * replicated, enable it, the insert should be replicated + * + * @throws Exception + */ + @Test(timeout = 300000) + public void testDisableEnable() throws Exception { + + // Test disabling replication + admin.disablePeer("2"); + + byte[] rowkey = Bytes.toBytes("disable enable"); + Put put = new Put(rowkey); + put.add(famName, row, row); + htable1.put(put); + + Get get = new Get(rowkey); + for (int i = 0; i < NB_RETRIES; i++) { + Result res = htable2.get(get); + if (res.size() >= 1) { + fail("Replication wasn't disabled"); + } else { + LOG.info("Row not replicated, let's wait a bit more..."); + Thread.sleep(SLEEP_TIME); + } + } + + // Test enable replication + admin.enablePeer("2"); + + for (int i = 0; i < NB_RETRIES; i++) { + Result res = htable2.get(get); + if (res.size() == 0) { + LOG.info("Row not available"); + Thread.sleep(SLEEP_TIME); + } else { + assertArrayEquals(res.value(), row); + return; + } + } + fail("Waited too much time for put replication"); + } + /** * Integration test for TestReplicationAdmin, removes and re-add a peer * cluster + * * @throws Exception */ @Test(timeout=300000) @@ -495,6 +429,7 @@ public void testAddAndRemoveClusters() throws Exception { } } + /** * Do a more intense version testSmallBatch, one that will trigger * hlog rolling and other non-trivial code paths @@ -517,17 +452,17 @@ public void loadTesting() throws Exception { Result[] res = scanner.next(NB_ROWS_IN_BIG_BATCH); scanner.close(); - assertEquals(NB_ROWS_IN_BATCH *10, res.length); + assertEquals(NB_ROWS_IN_BIG_BATCH, res.length); scan = new Scan(); - for (int i = 0; i < NB_RETRIES; i++) { + for (int i = 0; i < NB_RETRIES_FOR_BIG_BATCH; i++) { scanner = htable2.getScanner(scan); res = scanner.next(NB_ROWS_IN_BIG_BATCH); scanner.close(); if (res.length != NB_ROWS_IN_BIG_BATCH) { - if (i == NB_RETRIES-1) { + if (i == NB_RETRIES_FOR_BIG_BATCH-1) { int lastRow = -1; for (Result result : res) { int currentRow = Bytes.toInt(result.getRow()); @@ -594,104 +529,63 @@ public void testVerifyRepJob() throws Exception { fail("Job failed, see the log"); } assertEquals(0, job.getCounters(). - findCounter(VerifyReplication.Verifier.Counters.GOODROWS).getValue()); - assertEquals(NB_ROWS_IN_BATCH, job.getCounters(). - findCounter(VerifyReplication.Verifier.Counters.BADROWS).getValue()); + findCounter(VerifyReplication.Verifier.Counters.GOODROWS).getValue()); + assertEquals(NB_ROWS_IN_BATCH, job.getCounters(). + findCounter(VerifyReplication.Verifier.Counters.BADROWS).getValue()); } - /** - * Load up multiple tables over 2 region servers and kill a source during - * the upload. The failover happens internally. - * - * WARNING this test sometimes fails because of HBASE-3515 - * - * @throws Exception - */ - @Test(timeout=300000) - public void queueFailover() throws Exception { - utility1.createMultiRegions(htable1, famName); - - // killing the RS with .META. can result into failed puts until we solve - // IO fencing - int rsToKill1 = - utility1.getHBaseCluster().getServerWithMeta() == 0 ? 1 : 0; - int rsToKill2 = - utility2.getHBaseCluster().getServerWithMeta() == 0 ? 1 : 0; - - // Takes about 20 secs to run the full loading, kill around the middle - Thread killer1 = killARegionServer(utility1, 7500, rsToKill1); - Thread killer2 = killARegionServer(utility2, 10000, rsToKill2); - - LOG.info("Start loading table"); - int initialCount = utility1.loadTable(htable1, famName); - LOG.info("Done loading table"); - killer1.join(5000); - killer2.join(5000); - LOG.info("Done waiting for threads"); - - Result[] res; - while (true) { - try { - Scan scan = new Scan(); - ResultScanner scanner = htable1.getScanner(scan); - res = scanner.next(initialCount); - scanner.close(); - break; - } catch (UnknownScannerException ex) { - LOG.info("Cluster wasn't ready yet, restarting scanner"); - } - } - // Test we actually have all the rows, we may miss some because we - // don't have IO fencing. - if (res.length != initialCount) { - LOG.warn("We lost some rows on the master cluster!"); - // We don't really expect the other cluster to have more rows - initialCount = res.length; - } - - int lastCount = 0; - - for (int i = 0; i < NB_RETRIES; i++) { - if (i==NB_RETRIES-1) { - fail("Waited too much time for queueFailover replication"); + + /** + * Test for HBASE-8663 + * Create two new Tables with colfamilies enabled for replication then run + * ReplicationAdmin.listReplicated(). Finally verify the table:colfamilies. Note: + * TestReplicationAdmin is a better place for this testing but it would need mocks. + * @throws Exception + */ + @Test(timeout = 300000) + public void testVerifyListReplicatedTable() throws Exception { + LOG.info("testVerifyListReplicatedTable"); + + final String tName = "VerifyListReplicated_"; + final String colFam = "cf1"; + final int numOfTables = 3; + + HBaseAdmin hadmin = new HBaseAdmin(conf1); + + // Create Tables + for (int i = 0; i < numOfTables; i++) { + HTableDescriptor ht = new HTableDescriptor(tName + i); + HColumnDescriptor cfd = new HColumnDescriptor(colFam); + cfd.setScope(HConstants.REPLICATION_SCOPE_GLOBAL); + ht.addFamily(cfd); + hadmin.createTable(ht); } - Scan scan2 = new Scan(); - ResultScanner scanner2 = htable2.getScanner(scan2); - Result[] res2 = scanner2.next(initialCount * 2); - scanner2.close(); - if (res2.length < initialCount) { - if (lastCount < res2.length) { - i--; // Don't increment timeout if we make progress + + // verify the result + List> replicationColFams = admin.listReplicated(); + int[] match = new int[numOfTables]; // array of 3 with init value of zero + + for (int i = 0; i < replicationColFams.size(); i++) { + HashMap replicationEntry = replicationColFams.get(i); + String tn = replicationEntry.get(ReplicationAdmin.TNAME); + if ((tn.startsWith(tName)) && replicationEntry.get(ReplicationAdmin.CFNAME).equals(colFam)) { + int m = Integer.parseInt(tn.substring(tn.length() - 1)); // get the last digit + match[m]++; // should only increase once } - lastCount = res2.length; - LOG.info("Only got " + lastCount + " rows instead of " + - initialCount + " current i=" + i); - Thread.sleep(SLEEP_TIME*2); - } else { - break; } - } - } - - private static Thread killARegionServer(final HBaseTestingUtility utility, - final long timeout, final int rs) { - Thread killer = new Thread() { - public void run() { - try { - Thread.sleep(timeout); - utility.expireRegionServerSession(rs); - } catch (Exception e) { - LOG.error(e); - } + + // check the matching result + for (int i = 0; i < match.length; i++) { + assertTrue("listReplicated() does not match table " + i, (match[i] == 1)); } - }; - killer.setDaemon(true); - killer.start(); - return killer; - } - - @org.junit.Rule - public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = - new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); + + // drop tables + for (int i = 0; i < numOfTables; i++) { + String ht = tName + i; + hadmin.disableTable(ht); + hadmin.deleteTable(ht); + } + + hadmin.close(); + } } - diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java index fbf38f1efc0f..603b0a74d575 100644 --- a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java new file mode 100644 index 000000000000..978c436ca500 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java @@ -0,0 +1,416 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.replication; + +import static org.junit.Assert.assertEquals; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.client.Delete; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.replication.ReplicationAdmin; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.replication.regionserver.ReplicationSyncUp; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(LargeTests.class) +public class TestReplicationSyncUpTool extends TestReplicationBase { + + private static final Log LOG = LogFactory.getLog(TestReplicationSyncUpTool.class); + + private static final byte[] t1_su = Bytes.toBytes("t1_syncup"); + private static final byte[] t2_su = Bytes.toBytes("t2_syncup"); + + private static final byte[] famName = Bytes.toBytes("cf1"); + private static final byte[] qualName = Bytes.toBytes("q1"); + + private static final byte[] noRepfamName = Bytes.toBytes("norep"); + + private HTableDescriptor t1_syncupSource, t1_syncupTarget; + private HTableDescriptor t2_syncupSource, t2_syncupTarget; + + private HTable ht1Source, ht2Source, ht1TargetAtPeer1, ht2TargetAtPeer1; + + @Before + public void setUp() throws Exception { + + HColumnDescriptor fam; + + t1_syncupSource = new HTableDescriptor(t1_su); + fam = new HColumnDescriptor(famName); + fam.setScope(HConstants.REPLICATION_SCOPE_GLOBAL); + t1_syncupSource.addFamily(fam); + fam = new HColumnDescriptor(noRepfamName); + t1_syncupSource.addFamily(fam); + + t1_syncupTarget = new HTableDescriptor(t1_su); + fam = new HColumnDescriptor(famName); + t1_syncupTarget.addFamily(fam); + fam = new HColumnDescriptor(noRepfamName); + t1_syncupTarget.addFamily(fam); + + t2_syncupSource = new HTableDescriptor(t2_su); + fam = new HColumnDescriptor(famName); + fam.setScope(HConstants.REPLICATION_SCOPE_GLOBAL); + t2_syncupSource.addFamily(fam); + fam = new HColumnDescriptor(noRepfamName); + t2_syncupSource.addFamily(fam); + + t2_syncupTarget = new HTableDescriptor(t2_su); + fam = new HColumnDescriptor(famName); + t2_syncupTarget.addFamily(fam); + fam = new HColumnDescriptor(noRepfamName); + t2_syncupTarget.addFamily(fam); + + } + + /** + * Add a row to a table in each cluster, check it's replicated, delete it, + * check's gone Also check the puts and deletes are not replicated back to + * the originating cluster. + */ + @Test(timeout = 300000) + public void testSyncUpTool() throws Exception { + + /** + * Set up Replication: on Master and one Slave + * Table: t1_syncup and t2_syncup + * columnfamily: + * 'cf1' : replicated + * 'norep': not replicated + */ + setupReplication(); + + /** + * at Master: + * t1_syncup: put 100 rows into cf1, and 1 rows into norep + * t2_syncup: put 200 rows into cf1, and 1 rows into norep + * + * verify correctly replicated to slave + */ + putAndReplicateRows(); + + /** + * Verify delete works + * + * step 1: stop hbase on Slave + * + * step 2: at Master: + * t1_syncup: delete 50 rows from cf1 + * t2_syncup: delete 100 rows from cf1 + * no change on 'norep' + * + * step 3: stop hbase on master, restart hbase on Slave + * + * step 4: verify Slave still have the rows before delete + * t1_syncup: 100 rows from cf1 + * t2_syncup: 200 rows from cf1 + * + * step 5: run syncup tool on Master + * + * step 6: verify that delete show up on Slave + * t1_syncup: 50 rows from cf1 + * t2_syncup: 100 rows from cf1 + * + * verify correctly replicated to Slave + */ + mimicSyncUpAfterDelete(); + + /** + * Verify put works + * + * step 1: stop hbase on Slave + * + * step 2: at Master: + * t1_syncup: put 100 rows from cf1 + * t2_syncup: put 200 rows from cf1 + * and put another row on 'norep' + * ATTN: put to 'cf1' will overwrite existing rows, so end count will + * be 100 and 200 respectively + * put to 'norep' will add a new row. + * + * step 3: stop hbase on master, restart hbase on Slave + * + * step 4: verify Slave still has the rows before put + * t1_syncup: 50 rows from cf1 + * t2_syncup: 100 rows from cf1 + * + * step 5: run syncup tool on Master + * + * step 6: verify that put show up on Slave + * and 'norep' does not + * t1_syncup: 100 rows from cf1 + * t2_syncup: 200 rows from cf1 + * + * verify correctly replicated to Slave + */ + mimicSyncUpAfterPut(); + + } + + private void setupReplication() throws Exception { + ReplicationAdmin admin1 = new ReplicationAdmin(conf1); + ReplicationAdmin admin2 = new ReplicationAdmin(conf2); + + HBaseAdmin ha = new HBaseAdmin(conf1); + ha.createTable(t1_syncupSource); + ha.createTable(t2_syncupSource); + ha.close(); + + ha = new HBaseAdmin(conf2); + ha.createTable(t1_syncupTarget); + ha.createTable(t2_syncupTarget); + ha.close(); + + // Get HTable from Master + ht1Source = new HTable(conf1, t1_su); + ht1Source.setWriteBufferSize(1024); + ht2Source = new HTable(conf1, t2_su); + ht1Source.setWriteBufferSize(1024); + + // Get HTable from Peer1 + ht1TargetAtPeer1 = new HTable(conf2, t1_su); + ht1TargetAtPeer1.setWriteBufferSize(1024); + ht2TargetAtPeer1 = new HTable(conf2, t2_su); + ht2TargetAtPeer1.setWriteBufferSize(1024); + + /** + * set M-S : Master: utility1 Slave1: utility2 + */ + admin1.addPeer("1", utility2.getClusterKey()); + + admin1.close(); + admin2.close(); + } + + private void putAndReplicateRows() throws Exception { + LOG.debug("putAndReplicateRows"); + // add rows to Master cluster, + Put p; + + // 100 + 1 row to t1_syncup + for (int i = 0; i < NB_ROWS_IN_BATCH; i++) { + p = new Put(Bytes.toBytes("row" + i)); + p.add(famName, qualName, Bytes.toBytes("val" + i)); + ht1Source.put(p); + } + p = new Put(Bytes.toBytes("row" + 9999)); + p.add(noRepfamName, qualName, Bytes.toBytes("val" + 9999)); + ht1Source.put(p); + + // 200 + 1 row to t2_syncup + for (int i = 0; i < NB_ROWS_IN_BATCH * 2; i++) { + p = new Put(Bytes.toBytes("row" + i)); + p.add(famName, qualName, Bytes.toBytes("val" + i)); + ht2Source.put(p); + } + p = new Put(Bytes.toBytes("row" + 9999)); + p.add(noRepfamName, qualName, Bytes.toBytes("val" + 9999)); + ht2Source.put(p); + + // ensure replication completed + Thread.sleep(SLEEP_TIME); + int rowCount_ht1Source = utility1.countRows(ht1Source); + for (int i = 0; i < NB_RETRIES; i++) { + int rowCount_ht1TargetAtPeer1 = utility2.countRows(ht1TargetAtPeer1); + if (i==NB_RETRIES-1) { + assertEquals("t1_syncup has 101 rows on source, and 100 on slave1", rowCount_ht1Source - 1, + rowCount_ht1TargetAtPeer1); + } + if (rowCount_ht1Source - 1 == rowCount_ht1TargetAtPeer1) { + break; + } + Thread.sleep(SLEEP_TIME); + } + + int rowCount_ht2Source = utility1.countRows(ht2Source); + for (int i = 0; i < NB_RETRIES; i++) { + int rowCount_ht2TargetAtPeer1 = utility2.countRows(ht2TargetAtPeer1); + if (i==NB_RETRIES-1) { + assertEquals("t2_syncup has 201 rows on source, and 200 on slave1", rowCount_ht2Source - 1, + rowCount_ht2TargetAtPeer1); + } + if (rowCount_ht2Source - 1 == rowCount_ht2TargetAtPeer1) { + break; + } + Thread.sleep(SLEEP_TIME); + } + } + + private void mimicSyncUpAfterDelete() throws Exception { + LOG.debug("mimicSyncUpAfterDelete"); + utility2.shutdownMiniHBaseCluster(); + + List list = new ArrayList(); + // delete half of the rows + for (int i = 0; i < NB_ROWS_IN_BATCH / 2; i++) { + String rowKey = "row" + i; + Delete del = new Delete(rowKey.getBytes()); + list.add(del); + } + ht1Source.delete(list); + + for (int i = 0; i < NB_ROWS_IN_BATCH; i++) { + String rowKey = "row" + i; + Delete del = new Delete(rowKey.getBytes()); + list.add(del); + } + ht2Source.delete(list); + + int rowCount_ht1Source = utility1.countRows(ht1Source); + assertEquals("t1_syncup has 51 rows on source, after remove 50 of the replicated colfam", 51, + rowCount_ht1Source); + + int rowCount_ht2Source = utility1.countRows(ht2Source); + assertEquals("t2_syncup has 101 rows on source, after remove 100 of the replicated colfam", + 101, rowCount_ht2Source); + + utility1.shutdownMiniHBaseCluster(); + utility2.restartHBaseCluster(1); + + Thread.sleep(SLEEP_TIME); + + // before sync up + int rowCount_ht1TargetAtPeer1 = utility2.countRows(ht1TargetAtPeer1); + int rowCount_ht2TargetAtPeer1 = utility2.countRows(ht2TargetAtPeer1); + assertEquals("@Peer1 t1_syncup should still have 100 rows", 100, rowCount_ht1TargetAtPeer1); + assertEquals("@Peer1 t2_syncup should still have 200 rows", 200, rowCount_ht2TargetAtPeer1); + + // After sync up + for (int i = 0; i < NB_RETRIES; i++) { + syncUp(utility1); + rowCount_ht1TargetAtPeer1 = utility2.countRows(ht1TargetAtPeer1); + rowCount_ht2TargetAtPeer1 = utility2.countRows(ht2TargetAtPeer1); + if (i == NB_RETRIES - 1) { + if (rowCount_ht1TargetAtPeer1 != 50 || rowCount_ht2TargetAtPeer1 != 100) { + // syncUP still failed. Let's look at the source in case anything wrong there + utility1.restartHBaseCluster(1); + rowCount_ht1Source = utility1.countRows(ht1Source); + LOG.debug("t1_syncup should have 51 rows at source, and it is " + rowCount_ht1Source); + rowCount_ht2Source = utility1.countRows(ht2Source); + LOG.debug("t2_syncup should have 101 rows at source, and it is " + rowCount_ht2Source); + } + assertEquals("@Peer1 t1_syncup should be sync up and have 50 rows", 50, + rowCount_ht1TargetAtPeer1); + assertEquals("@Peer1 t2_syncup should be sync up and have 100 rows", 100, + rowCount_ht2TargetAtPeer1); + } + if (rowCount_ht1TargetAtPeer1 == 50 && rowCount_ht2TargetAtPeer1 == 100) { + LOG.info("SyncUpAfterDelete succeeded at retry = " + i); + break; + } else { + LOG.debug("SyncUpAfterDelete failed at retry = " + i + ", with rowCount_ht1TargetPeer1 =" + + rowCount_ht1TargetAtPeer1 + " and rowCount_ht2TargetAtPeer1 =" + + rowCount_ht2TargetAtPeer1); + } + Thread.sleep(SLEEP_TIME); + } + } + + private void mimicSyncUpAfterPut() throws Exception { + LOG.debug("mimicSyncUpAfterPut"); + utility1.restartHBaseCluster(1); + utility2.shutdownMiniHBaseCluster(); + + Put p; + // another 100 + 1 row to t1_syncup + // we should see 100 + 2 rows now + for (int i = 0; i < NB_ROWS_IN_BATCH; i++) { + p = new Put(Bytes.toBytes("row" + i)); + p.add(famName, qualName, Bytes.toBytes("val" + i)); + ht1Source.put(p); + } + p = new Put(Bytes.toBytes("row" + 9998)); + p.add(noRepfamName, qualName, Bytes.toBytes("val" + 9998)); + ht1Source.put(p); + + // another 200 + 1 row to t1_syncup + // we should see 200 + 2 rows now + for (int i = 0; i < NB_ROWS_IN_BATCH * 2; i++) { + p = new Put(Bytes.toBytes("row" + i)); + p.add(famName, qualName, Bytes.toBytes("val" + i)); + ht2Source.put(p); + } + p = new Put(Bytes.toBytes("row" + 9998)); + p.add(noRepfamName, qualName, Bytes.toBytes("val" + 9998)); + ht2Source.put(p); + + int rowCount_ht1Source = utility1.countRows(ht1Source); + assertEquals("t1_syncup has 102 rows on source", 102, rowCount_ht1Source); + int rowCount_ht2Source = utility1.countRows(ht2Source); + assertEquals("t2_syncup has 202 rows on source", 202, rowCount_ht2Source); + + utility1.shutdownMiniHBaseCluster(); + utility2.restartHBaseCluster(1); + + Thread.sleep(SLEEP_TIME); + + // before sync up + int rowCount_ht1TargetAtPeer1 = utility2.countRows(ht1TargetAtPeer1); + int rowCount_ht2TargetAtPeer1 = utility2.countRows(ht2TargetAtPeer1); + assertEquals("@Peer1 t1_syncup should be NOT sync up and have 50 rows", 50, + rowCount_ht1TargetAtPeer1); + assertEquals("@Peer1 t2_syncup should be NOT sync up and have 100 rows", 100, + rowCount_ht2TargetAtPeer1); + + // after syun up + for (int i = 0; i < NB_RETRIES; i++) { + syncUp(utility1); + rowCount_ht1TargetAtPeer1 = utility2.countRows(ht1TargetAtPeer1); + rowCount_ht2TargetAtPeer1 = utility2.countRows(ht2TargetAtPeer1); + if (i == NB_RETRIES - 1) { + if (rowCount_ht1TargetAtPeer1 != 100 || rowCount_ht2TargetAtPeer1 != 200) { + // syncUP still failed. Let's look at the source in case anything wrong there + utility1.restartHBaseCluster(1); + rowCount_ht1Source = utility1.countRows(ht1Source); + LOG.debug("t1_syncup should have 102 rows at source, and it is " + rowCount_ht1Source); + rowCount_ht2Source = utility1.countRows(ht2Source); + LOG.debug("t2_syncup should have 202 rows at source, and it is " + rowCount_ht2Source); + } + assertEquals("@Peer1 t1_syncup should be sync up and have 100 rows", 100, + rowCount_ht1TargetAtPeer1); + assertEquals("@Peer1 t2_syncup should be sync up and have 200 rows", 200, + rowCount_ht2TargetAtPeer1); + } + if (rowCount_ht1TargetAtPeer1 == 100 && rowCount_ht2TargetAtPeer1 == 200) { + LOG.info("SyncUpAfterPut succeeded at retry = " + i); + break; + } else { + LOG.debug("SyncUpAfterPut failed at retry = " + i + ", with rowCount_ht1TargetPeer1 =" + + rowCount_ht1TargetAtPeer1 + " and rowCount_ht2TargetAtPeer1 =" + + rowCount_ht2TargetAtPeer1); + } + Thread.sleep(SLEEP_TIME); + } + } + + private void syncUp(HBaseTestingUtility ut) throws Exception { + ReplicationSyncUp.setConfigure(ut.getConfiguration()); + String[] arguments = new String[] { null }; + new ReplicationSyncUp().run(arguments); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationZookeeper.java b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationZookeeper.java new file mode 100644 index 000000000000..553b5cd7d27e --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationZookeeper.java @@ -0,0 +1,118 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.replication; + +import java.io.IOException; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.Server; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; +import org.apache.zookeeper.KeeperException; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import static org.junit.Assert.assertEquals; + +@Category(MediumTests.class) +public class TestReplicationZookeeper { + + private static Configuration conf; + + private static HBaseTestingUtility utility; + + private static ZooKeeperWatcher zkw; + + private static ReplicationZookeeper repZk; + + private static String slaveClusterKey; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + utility = new HBaseTestingUtility(); + utility.startMiniZKCluster(); + conf = utility.getConfiguration(); + zkw = HBaseTestingUtility.getZooKeeperWatcher(utility); + DummyServer server = new DummyServer(); + repZk = new ReplicationZookeeper(server, new AtomicBoolean()); + slaveClusterKey = conf.get(HConstants.ZOOKEEPER_QUORUM) + ":" + + conf.get("hbase.zookeeper.property.clientPort") + ":/1"; + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + utility.shutdownMiniZKCluster(); + } + + @Test + public void testGetAddressesMissingSlave() + throws IOException, KeeperException { + repZk.addPeer("1", slaveClusterKey); + // HBASE-5586 used to get an NPE + assertEquals(0, repZk.getSlavesAddresses("1").size()); + } + + static class DummyServer implements Server { + + @Override + public Configuration getConfiguration() { + return conf; + } + + @Override + public ZooKeeperWatcher getZooKeeper() { + return zkw; + } + + @Override + public CatalogTracker getCatalogTracker() { + return null; + } + + @Override + public ServerName getServerName() { + return new ServerName("hostname.example.org", 1234, -1L); + } + + @Override + public void abort(String why, Throwable e) { + } + + @Override + public boolean isAborted() { + return false; + } + + @Override + public void stop(String why) { + } + + @Override + public boolean isStopped() { + return false; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java b/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java index 18eb530fcffb..04ff8ba2b4ec 100644 --- a/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java +++ b/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java b/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java index 07b3f3c30f0f..891aaca8a009 100644 --- a/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java +++ b/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,23 +19,39 @@ package org.apache.hadoop.hbase.replication.regionserver; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; import java.net.URLEncoder; import java.util.ArrayList; +import java.util.Collection; import java.util.List; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.atomic.AtomicBoolean; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.Server; +import org.apache.hadoop.hbase.ServerName; import org.apache.hadoop.hbase.catalog.CatalogTracker; import org.apache.hadoop.hbase.regionserver.wal.HLog; import org.apache.hadoop.hbase.regionserver.wal.HLogKey; -import org.apache.hadoop.hbase.regionserver.wal.WALEdit; import org.apache.hadoop.hbase.regionserver.wal.WALActionsListener; +import org.apache.hadoop.hbase.regionserver.wal.WALEdit; import org.apache.hadoop.hbase.replication.ReplicationSourceDummy; +import org.apache.hadoop.hbase.replication.ReplicationZookeeper; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.zookeeper.ZKUtil; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; @@ -82,7 +97,10 @@ public class TestReplicationSourceManager { private static Path oldLogDir; private static Path logDir; + + private static CountDownLatch latch; + private static List files = new ArrayList(); @BeforeClass public static void setUpBeforeClass() throws Exception { @@ -100,6 +118,9 @@ public static void setUpBeforeClass() throws Exception { ZKUtil.setData(zkw, "/hbase/replication/peers/1", Bytes.toBytes(conf.get(HConstants.ZOOKEEPER_QUORUM) + ":" + conf.get(HConstants.ZOOKEEPER_CLIENT_PORT) + ":/1")); + ZKUtil.createWithParents(zkw, "/hbase/replication/peers/1/peer-state"); + ZKUtil.setData(zkw, "/hbase/replication/peers/1/peer-state", + Bytes.toBytes(ReplicationZookeeper.PeerState.ENABLED.name())); ZKUtil.createWithParents(zkw, "/hbase/replication/state"); ZKUtil.setData(zkw, "/hbase/replication/state", Bytes.toBytes("true")); @@ -189,7 +210,7 @@ public void testLogRoll() throws Exception { hlog.rollWriter(); manager.logPositionAndCleanOldLogs(manager.getSources().get(0).getCurrentPath(), - "1", 0, false); + "1", 0, false, false); HLogKey key = new HLogKey(hri.getRegionName(), test, seq++, System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID); @@ -200,8 +221,144 @@ public void testLogRoll() throws Exception { // TODO Need a case with only 2 HLogs and we only want to delete the first one } + + @Test + public void testNodeFailoverWorkerCopyQueuesFromRSUsingMulti() throws Exception { + LOG.debug("testNodeFailoverWorkerCopyQueuesFromRSUsingMulti"); + conf.setBoolean(HConstants.ZOOKEEPER_USEMULTI, true); + final Server server = new DummyServer("hostname0.example.org"); + AtomicBoolean replicating = new AtomicBoolean(true); + ReplicationZookeeper rz = new ReplicationZookeeper(server, replicating); + // populate some znodes in the peer znode + files.add("log1"); + files.add("log2"); + for (String file : files) { + rz.addLogToList(file, "1"); + } + // create 3 DummyServers + Server s1 = new DummyServer("dummyserver1.example.org"); + Server s2 = new DummyServer("dummyserver2.example.org"); + Server s3 = new DummyServer("dummyserver3.example.org"); + + // create 3 DummyNodeFailoverWorkers + DummyNodeFailoverWorker w1 = new DummyNodeFailoverWorker( + server.getServerName().getServerName(), s1); + DummyNodeFailoverWorker w2 = new DummyNodeFailoverWorker( + server.getServerName().getServerName(), s2); + DummyNodeFailoverWorker w3 = new DummyNodeFailoverWorker( + server.getServerName().getServerName(), s3); + + latch = new CountDownLatch(3); + // start the threads + w1.start(); + w2.start(); + w3.start(); + // make sure only one is successful + int populatedMap = 0; + // wait for result now... till all the workers are done. + latch.await(); + populatedMap += w1.isLogZnodesMapPopulated() + w2.isLogZnodesMapPopulated() + + w3.isLogZnodesMapPopulated(); + assertEquals(1, populatedMap); + // close out the resources. + server.abort("", null); + } + + @Test + public void testNodeFailoverDeadServerParsing() throws Exception { + LOG.debug("testNodeFailoverDeadServerParsing"); + conf.setBoolean(HConstants.ZOOKEEPER_USEMULTI, true); + final Server server = new DummyServer("ec2-54-234-230-108.compute-1.amazonaws.com"); + AtomicBoolean replicating = new AtomicBoolean(true); + ReplicationZookeeper rz = new ReplicationZookeeper(server, replicating); + // populate some znodes in the peer znode + files.add("log1"); + files.add("log2"); + for (String file : files) { + rz.addLogToList(file, "1"); + } + // create 3 DummyServers + Server s1 = new DummyServer("ip-10-8-101-114.ec2.internal"); + Server s2 = new DummyServer("ec2-107-20-52-47.compute-1.amazonaws.com"); + Server s3 = new DummyServer("ec2-23-20-187-167.compute-1.amazonaws.com"); + + // simulate three server fail sequentially + ReplicationZookeeper rz1 = new ReplicationZookeeper(s1, new AtomicBoolean(true)); + SortedMap> testMap = + rz1.copyQueuesFromRSUsingMulti(server.getServerName().getServerName()); + ReplicationZookeeper rz2 = new ReplicationZookeeper(s2, new AtomicBoolean(true)); + testMap = rz2.copyQueuesFromRSUsingMulti(s1.getServerName().getServerName()); + ReplicationZookeeper rz3 = new ReplicationZookeeper(s3, new AtomicBoolean(true)); + testMap = rz3.copyQueuesFromRSUsingMulti(s2.getServerName().getServerName()); + + ReplicationSource s = new ReplicationSource(); + s.checkIfQueueRecovered(testMap.firstKey()); + List result = s.getDeadRegionServers(); + + // verify + assertTrue(result.contains(server.getServerName().getServerName())); + assertTrue(result.contains(s1.getServerName().getServerName())); + assertTrue(result.contains(s2.getServerName().getServerName())); + + server.abort("", null); + } + + static class DummyNodeFailoverWorker extends Thread { + private SortedMap> logZnodesMap; + Server server; + private String deadRsZnode; + ReplicationZookeeper rz; + + public DummyNodeFailoverWorker(String znode, Server s) throws Exception { + this.deadRsZnode = znode; + this.server = s; + rz = new ReplicationZookeeper(server, new AtomicBoolean(true)); + } + + @Override + public void run() { + try { + logZnodesMap = rz.copyQueuesFromRSUsingMulti(deadRsZnode); + server.abort("Done with testing", null); + } catch (Exception e) { + LOG.error("Got exception while running NodeFailoverWorker", e); + } finally { + latch.countDown(); + } + } + /** + * @return 1 when the map is not empty. + */ + private int isLogZnodesMapPopulated() { + Collection> sets = logZnodesMap.values(); + if (sets.size() > 1) { + throw new RuntimeException("unexpected size of logZnodesMap: " + sets.size()); + } + if (sets.size() == 1) { + SortedSet s = sets.iterator().next(); + for (String file : files) { + // at least one file was missing + if (!s.contains(file)) { + return 0; + } + } + return 1; // we found all the files + } + return 0; + } + } + static class DummyServer implements Server { + String hostname; + + DummyServer() { + hostname = "hostname.example.org"; + } + + DummyServer(String hostname) { + this.hostname = hostname; + } @Override public Configuration getConfiguration() { @@ -215,19 +372,19 @@ public ZooKeeperWatcher getZooKeeper() { @Override public CatalogTracker getCatalogTracker() { - return null; //To change body of implemented methods use File | Settings | File Templates. + return null; // To change body of implemented methods use File | Settings | File Templates. } @Override public ServerName getServerName() { - return new ServerName("hostname.example.org", 1234, -1L); + return new ServerName(hostname, 1234, 1L); } @Override public void abort(String why, Throwable e) { - //To change body of implemented methods use File | Settings | File Templates. + // To change body of implemented methods use File | Settings | File Templates. } - + @Override public boolean isAborted() { return false; diff --git a/src/test/java/org/apache/hadoop/hbase/rest/HBaseRESTTestingUtility.java b/src/test/java/org/apache/hadoop/hbase/rest/HBaseRESTTestingUtility.java index 6b723be2f1db..4d4c1007fa5c 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/HBaseRESTTestingUtility.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/HBaseRESTTestingUtility.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -27,6 +26,7 @@ import org.mortbay.jetty.Server; import org.mortbay.jetty.servlet.Context; import org.mortbay.jetty.servlet.ServletHolder; +import org.apache.hadoop.hbase.util.HttpServerUtil; import com.sun.jersey.spi.container.servlet.ServletContainer; @@ -68,6 +68,7 @@ public void startServletContainer(Configuration conf) throws Exception { Context context = new Context(server, "/", Context.SESSIONS); context.addServlet(sh, "/*"); context.addFilter(GzipFilter.class, "/*", 0); + HttpServerUtil.constrainHttpMethods(context); // start the server server.start(); // get the port diff --git a/src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java b/src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java index 23673c71198b..f41a41c27537 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -114,7 +113,6 @@ public class PerformanceEvaluation { protected Map commands = new TreeMap(); protected static Cluster cluster = new Cluster(); - protected static String accessToken = null; volatile Configuration conf; private boolean nomapred = false; @@ -449,8 +447,7 @@ public void setStatus(String msg) { */ private boolean checkTable() throws IOException { HTableDescriptor tableDescriptor = getTableDescriptor(); - RemoteAdmin admin = - new RemoteAdmin(new Client(cluster), conf, accessToken); + RemoteAdmin admin = new RemoteAdmin(new Client(cluster), conf); if (!admin.isTableAvailable(tableDescriptor.getName())) { admin.createTable(tableDescriptor); return true; @@ -714,8 +711,7 @@ protected int getReportingPeriod() { } void testSetup() throws IOException { - this.table = new RemoteHTable(new Client(cluster), conf, tableName, - accessToken); + this.table = new RemoteHTable(new Client(cluster), conf, tableName); } void testTakedown() throws IOException { @@ -1133,7 +1129,6 @@ protected void printUsage(final String message) { System.err.println(); System.err.println("Options:"); System.err.println(" host String. Specify Stargate endpoint."); - System.err.println(" token String. API access token."); System.err.println(" rows Integer. Rows each client runs. Default: One million"); System.err.println(" rowsPerPut Integer. Rows each Stargate (multi)Put. Default: 100"); System.err.println(" nomapred (Flag) Run multiple clients using threads " + @@ -1208,12 +1203,6 @@ public int doCommandLine(final String[] args) { continue; } - final String token = "--token="; - if (cmd.startsWith(token)) { - accessToken = cmd.substring(token.length()); - continue; - } - Class cmdClass = determineCommandClass(cmd); if (cmdClass != null) { getArgs(i + 1, args); diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestGZIPResponseWrapper.java b/src/test/java/org/apache/hadoop/hbase/rest/TestGZIPResponseWrapper.java new file mode 100644 index 000000000000..cdd69fd7283e --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestGZIPResponseWrapper.java @@ -0,0 +1,141 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.rest; + +import java.io.IOException; + +import javax.servlet.ServletOutputStream; +import javax.servlet.http.HttpServletResponse; + +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.rest.filter.GZIPResponseStream; +import org.apache.hadoop.hbase.rest.filter.GZIPResponseWrapper; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import static org.mockito.Mockito.*; +import static org.junit.Assert.*; + +@Category(MediumTests.class) +public class TestGZIPResponseWrapper { + + /** + * headers function should be called in response except header "content-length" + * + * @throws IOException + */ + @Test + public void testHeader() throws IOException { + + HttpServletResponse response = mock(HttpServletResponse.class); + + GZIPResponseWrapper test = new GZIPResponseWrapper(response); + test.setStatus(200); + verify(response).setStatus(200); + test.addHeader("header", "header value"); + verify(response).addHeader("header", "header value"); + test.addHeader("content-length", "header value2"); + verify(response, never()).addHeader("content-length", "header value"); + + test.setIntHeader("header", 5); + verify(response).setIntHeader("header", 5); + test.setIntHeader("content-length", 4); + verify(response, never()).setIntHeader("content-length", 4); + + test.setHeader("set-header", "new value"); + verify(response).setHeader("set-header", "new value"); + test.setHeader("content-length", "content length value"); + verify(response, never()).setHeader("content-length", "content length value"); + + test.sendRedirect("location"); + verify(response).sendRedirect("location"); + + test.flushBuffer(); + verify(response).flushBuffer(); + + } + + @Test + public void testResetBuffer() throws IOException { + HttpServletResponse response = mock(HttpServletResponse.class); + when(response.isCommitted()).thenReturn(false); + ServletOutputStream out = mock(ServletOutputStream.class); + when(response.getOutputStream()).thenReturn(out); + GZIPResponseWrapper test = new GZIPResponseWrapper(response); + + ServletOutputStream servletOutput = test.getOutputStream(); + assertEquals(org.apache.hadoop.hbase.rest.filter.GZIPResponseStream.class, + servletOutput.getClass()); + test.resetBuffer(); + verify(response).setHeader("Content-Encoding", null); + + when(response.isCommitted()).thenReturn(true); + servletOutput = test.getOutputStream(); + assertEquals(out.getClass(), servletOutput.getClass()); + assertNotNull(test.getWriter()); + + } + + @Test + public void testReset() throws IOException { + HttpServletResponse response = mock(HttpServletResponse.class); + when(response.isCommitted()).thenReturn(false); + ServletOutputStream out = mock(ServletOutputStream.class); + when(response.getOutputStream()).thenReturn(out); + GZIPResponseWrapper test = new GZIPResponseWrapper(response); + + ServletOutputStream servletOutput = test.getOutputStream(); + assertEquals(org.apache.hadoop.hbase.rest.filter.GZIPResponseStream.class, + servletOutput.getClass()); + test.reset(); + verify(response).setHeader("Content-Encoding", null); + + when(response.isCommitted()).thenReturn(true); + servletOutput = test.getOutputStream(); + assertEquals(out.getClass(), servletOutput.getClass()); + } + + @Test + public void testSendError() throws IOException { + HttpServletResponse response = mock(HttpServletResponse.class); + GZIPResponseWrapper test = new GZIPResponseWrapper(response); + + test.sendError(404); + verify(response).sendError(404); + + test.sendError(404, "error message"); + verify(response).sendError(404, "error message"); + + } + + @Test + public void testGZIPResponseStream() throws IOException { + HttpServletResponse httpResponce = mock(HttpServletResponse.class); + ServletOutputStream out = mock(ServletOutputStream.class); + + when(httpResponce.getOutputStream()).thenReturn(out); + GZIPResponseStream test = new GZIPResponseStream(httpResponce); + verify(httpResponce).addHeader("Content-Encoding", "gzip"); + + test.close(); + + test.resetBuffer(); + verify(httpResponce).setHeader("Content-Encoding", null); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestGzipFilter.java b/src/test/java/org/apache/hadoop/hbase/rest/TestGzipFilter.java index 570adddb51bd..d45980f3900b 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/TestGzipFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestGzipFilter.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -116,6 +115,8 @@ public void testGzipFilter() throws Exception { is.read(value, 0, VALUE_1.length); assertTrue(Bytes.equals(value, VALUE_1)); is.close(); + + testScannerResultCodes(); } @Test @@ -133,8 +134,7 @@ public void testErrorNotGzipped() throws Exception { assertTrue(contentEncoding == null || !contentEncoding.contains("gzip")); } - @Test - public void testScannerResultCodes() throws Exception { + void testScannerResultCodes() throws Exception { Header[] headers = new Header[3]; headers[0] = new Header("Content-Type", Constants.MIMETYPE_XML); headers[1] = new Header("Accept", Constants.MIMETYPE_JSON); diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestMultiRowResource.java b/src/test/java/org/apache/hadoop/hbase/rest/TestMultiRowResource.java index c5b714370d47..d485e2da04d4 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/TestMultiRowResource.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestMultiRowResource.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -117,6 +116,7 @@ public void testMultiCellGetJSON() throws IOException, JAXBException { Response response = client.get(path.toString(), Constants.MIMETYPE_JSON); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_JSON, response.getHeader("content-type")); client.delete(row_5_url); client.delete(row_6_url); @@ -143,6 +143,7 @@ public void testMultiCellGetXML() throws IOException, JAXBException { Response response = client.get(path.toString(), Constants.MIMETYPE_XML); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); client.delete(row_5_url); client.delete(row_6_url); diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestRESTMetrics.java b/src/test/java/org/apache/hadoop/hbase/rest/TestRESTMetrics.java new file mode 100644 index 000000000000..73ac925526b6 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestRESTMetrics.java @@ -0,0 +1,89 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.hbase.rest; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.rest.metrics.RESTMetrics; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import static org.junit.Assert.*; + +/** + * Test RESTMetrics class + */ +@Category(SmallTests.class) +public class TestRESTMetrics { + + @Test + public void testRESTMetrics() throws InterruptedException { + long timeout = 2000; + RESTMetrics test = new RESTMetrics(); + int incrementSucessfulGet = 20000; + int incrementSucessfulDelete = 3000000; + int incrementSucessfulPut = 3000000; + int incrementRequest = incrementSucessfulGet + incrementSucessfulDelete + incrementSucessfulPut; + + int incrementFailedGetRequests = 100; + int incrementFailedDeleteRequests = 30; + int incrementFailedPutRequests = 2; + + long start1 = System.currentTimeMillis(); + test.doUpdates(null); + + // started value + assertEquals(0, test.getRequests(), 0.01); + assertEquals(0, test.getSucessfulDeleteCount(), 0.01); + assertEquals(0, test.getSucessfulPutCount(), 0.01); + assertEquals(0, test.getSucessfulGetCount(), 0.01); + assertEquals(0, test.getFailedDeleteCount(), 0.01); + assertEquals(0, test.getFailedGetCount(), 0.01); + assertEquals(0, test.getFailedPutCount(), 0.01); + + // sleep some seconds + Thread.sleep(timeout); + test.incrementRequests(incrementRequest); + test.incrementSucessfulGetRequests(incrementSucessfulGet); + test.incrementSucessfulDeleteRequests(incrementSucessfulDelete); + test.incrementSucessfulPutRequests(incrementSucessfulPut); + test.incrementFailedGetRequests(incrementFailedGetRequests); + test.incrementFailedDeleteRequests(incrementFailedDeleteRequests); + test.incrementFailedPutRequests(incrementFailedPutRequests); + + test.doUpdates(null); + + // The maximum time for stability test + long tmax = System.currentTimeMillis() - start1; + + testData(tmax, timeout, test.getRequests(), incrementRequest); + testData(tmax, timeout, test.getSucessfulGetCount(), incrementSucessfulGet); + testData(tmax, timeout, test.getSucessfulDeleteCount(), incrementSucessfulDelete); + testData(tmax, timeout, test.getSucessfulPutCount(), incrementSucessfulPut); + testData(tmax, timeout, test.getFailedGetCount(), incrementFailedGetRequests); + testData(tmax, timeout, test.getFailedDeleteCount(), incrementFailedDeleteRequests); + testData(tmax, timeout, test.getFailedPutCount(), incrementFailedPutRequests); + + test.shutdown(); + } + + // test minimum and maximum speed + private void testData(double tmax, long tmin, float value, double requests) { + assertTrue((requests / tmax) * 1000 <= value); + assertTrue((requests / tmin) * 1000 >= value); + + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestRowResource.java b/src/test/java/org/apache/hadoop/hbase/rest/TestRowResource.java index b59436ca345c..cc14f75c0ced 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/TestRowResource.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestRowResource.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -208,6 +207,7 @@ private static void checkValueXML(String table, String row, String column, String value) throws IOException, JAXBException { Response response = getValueXML(table, row, column); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); CellSetModel cellSet = (CellSetModel) unmarshaller.unmarshal(new ByteArrayInputStream(response.getBody())); RowModel rowModel = cellSet.getRows().get(0); @@ -220,6 +220,7 @@ private static void checkValueXML(String url, String table, String row, String column, String value) throws IOException, JAXBException { Response response = getValueXML(url); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); CellSetModel cellSet = (CellSetModel) unmarshaller.unmarshal(new ByteArrayInputStream(response.getBody())); RowModel rowModel = cellSet.getRows().get(0); @@ -257,6 +258,7 @@ private static void checkValuePB(String table, String row, String column, String value) throws IOException { Response response = getValuePB(table, row, column); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF, response.getHeader("content-type")); CellSetModel cellSet = new CellSetModel(); cellSet.getObjectFromMessage(response.getBody()); RowModel rowModel = cellSet.getRows().get(0); @@ -265,11 +267,120 @@ private static void checkValuePB(String table, String row, String column, assertEquals(Bytes.toString(cell.getValue()), value); } + private static Response checkAndPutValuePB(String url, String table, + String row, String column, String valueToCheck, String valueToPut) + throws IOException { + RowModel rowModel = new RowModel(row); + rowModel.addCell(new CellModel(Bytes.toBytes(column), + Bytes.toBytes(valueToPut))); + rowModel.addCell(new CellModel(Bytes.toBytes(column), + Bytes.toBytes(valueToCheck))); + CellSetModel cellSetModel = new CellSetModel(); + cellSetModel.addRow(rowModel); + Response response = client.put(url, Constants.MIMETYPE_PROTOBUF, + cellSetModel.createProtobufOutput()); + Thread.yield(); + return response; + } + + private static Response checkAndPutValuePB(String table, String row, + String column, String valueToCheck, String valueToPut) throws IOException { + StringBuilder path = new StringBuilder(); + path.append('/'); + path.append(table); + path.append('/'); + path.append(row); + path.append("?check=put"); + return checkAndPutValuePB(path.toString(), table, row, column, + valueToCheck, valueToPut); + } + + private static Response checkAndPutValueXML(String url, String table, + String row, String column, String valueToCheck, String valueToPut) + throws IOException, JAXBException { + RowModel rowModel = new RowModel(row); + rowModel.addCell(new CellModel(Bytes.toBytes(column), + Bytes.toBytes(valueToPut))); + rowModel.addCell(new CellModel(Bytes.toBytes(column), + Bytes.toBytes(valueToCheck))); + CellSetModel cellSetModel = new CellSetModel(); + cellSetModel.addRow(rowModel); + StringWriter writer = new StringWriter(); + marshaller.marshal(cellSetModel, writer); + Response response = client.put(url, Constants.MIMETYPE_XML, + Bytes.toBytes(writer.toString())); + Thread.yield(); + return response; + } + + private static Response checkAndPutValueXML(String table, String row, + String column, String valueToCheck, String valueToPut) + throws IOException, JAXBException { + StringBuilder path = new StringBuilder(); + path.append('/'); + path.append(table); + path.append('/'); + path.append(row); + path.append("?check=put"); + return checkAndPutValueXML(path.toString(), table, row, column, + valueToCheck, valueToPut); + } + + private static Response checkAndDeleteXML(String url, String table, + String row, String column, String valueToCheck) + throws IOException, JAXBException { + RowModel rowModel = new RowModel(row); + rowModel.addCell(new CellModel(Bytes.toBytes(column), + Bytes.toBytes(valueToCheck))); + CellSetModel cellSetModel = new CellSetModel(); + cellSetModel.addRow(rowModel); + StringWriter writer = new StringWriter(); + marshaller.marshal(cellSetModel, writer); + Response response = client.put(url, Constants.MIMETYPE_XML, + Bytes.toBytes(writer.toString())); + Thread.yield(); + return response; + } + + private static Response checkAndDeleteXML(String table, String row, + String column, String valueToCheck) throws IOException, JAXBException { + StringBuilder path = new StringBuilder(); + path.append('/'); + path.append(table); + path.append('/'); + path.append(row); + path.append("?check=delete"); + return checkAndDeleteXML(path.toString(), table, row, column, valueToCheck); + } + + private static Response checkAndDeletePB(String table, String row, + String column, String value) throws IOException { + StringBuilder path = new StringBuilder(); + path.append('/'); + path.append(table); + path.append('/'); + path.append(row); + path.append("?check=delete"); + return checkAndDeleteValuePB(path.toString(), table, row, column, value); + } + + private static Response checkAndDeleteValuePB(String url, String table, + String row, String column, String valueToCheck) + throws IOException { + RowModel rowModel = new RowModel(row); + rowModel.addCell(new CellModel(Bytes.toBytes(column), Bytes + .toBytes(valueToCheck))); + CellSetModel cellSetModel = new CellSetModel(); + cellSetModel.addRow(rowModel); + Response response = client.put(url, Constants.MIMETYPE_PROTOBUF, + cellSetModel.createProtobufOutput()); + Thread.yield(); + return response; + } + @Test public void testDelete() throws IOException, JAXBException { - Response response; - - response = putValueXML(TABLE, ROW_1, COLUMN_1, VALUE_1); + Response response = putValueXML(TABLE, ROW_1, COLUMN_1, VALUE_1); assertEquals(response.getCode(), 200); response = putValueXML(TABLE, ROW_1, COLUMN_2, VALUE_2); assertEquals(response.getCode(), 200); @@ -282,6 +393,13 @@ public void testDelete() throws IOException, JAXBException { assertEquals(response.getCode(), 404); checkValueXML(TABLE, ROW_1, COLUMN_2, VALUE_2); + response = putValueXML(TABLE, ROW_1, COLUMN_1, VALUE_1); + assertEquals(response.getCode(), 200); + response = checkAndDeletePB(TABLE, ROW_1, COLUMN_1, VALUE_1); + assertEquals(response.getCode(), 200); + response = getValueXML(TABLE, ROW_1, COLUMN_1); + assertEquals(response.getCode(), 404); + response = deleteRow(TABLE, ROW_1); assertEquals(response.getCode(), 200); response = getValueXML(TABLE, ROW_1, COLUMN_1); @@ -292,16 +410,20 @@ public void testDelete() throws IOException, JAXBException { @Test public void testForbidden() throws IOException, JAXBException { - Response response; - conf.set("hbase.rest.readonly", "true"); - response = putValueXML(TABLE, ROW_1, COLUMN_1, VALUE_1); + Response response = putValueXML(TABLE, ROW_1, COLUMN_1, VALUE_1); assertEquals(response.getCode(), 403); response = putValuePB(TABLE, ROW_1, COLUMN_1, VALUE_1); assertEquals(response.getCode(), 403); + response = checkAndPutValueXML(TABLE, ROW_1, COLUMN_1, VALUE_1, VALUE_2); + assertEquals(response.getCode(), 403); + response = checkAndPutValuePB(TABLE, ROW_1, COLUMN_1, VALUE_1, VALUE_2); + assertEquals(response.getCode(), 403); response = deleteValue(TABLE, ROW_1, COLUMN_1); assertEquals(response.getCode(), 403); + response = checkAndDeletePB(TABLE, ROW_1, COLUMN_1, VALUE_1); + assertEquals(response.getCode(), 403); response = deleteRow(TABLE, ROW_1); assertEquals(response.getCode(), 403); @@ -311,6 +433,10 @@ public void testForbidden() throws IOException, JAXBException { assertEquals(response.getCode(), 200); response = putValuePB(TABLE, ROW_1, COLUMN_1, VALUE_1); assertEquals(response.getCode(), 200); + response = checkAndPutValueXML(TABLE, ROW_1, COLUMN_1, VALUE_1, VALUE_2); + assertEquals(response.getCode(), 200); + response = checkAndPutValuePB(TABLE, ROW_1, COLUMN_1, VALUE_2, VALUE_3); + assertEquals(response.getCode(), 200); response = deleteValue(TABLE, ROW_1, COLUMN_1); assertEquals(response.getCode(), 200); response = deleteRow(TABLE, ROW_1); @@ -328,6 +454,11 @@ public void testSingleCellGetPutXML() throws IOException, JAXBException { response = putValueXML(TABLE, ROW_1, COLUMN_1, VALUE_2); assertEquals(response.getCode(), 200); checkValueXML(TABLE, ROW_1, COLUMN_1, VALUE_2); + response = checkAndPutValueXML(TABLE, ROW_1, COLUMN_1, VALUE_2, VALUE_3); + assertEquals(response.getCode(), 200); + checkValueXML(TABLE, ROW_1, COLUMN_1, VALUE_3); + response = checkAndDeleteXML(TABLE, ROW_1, COLUMN_1, VALUE_3); + assertEquals(response.getCode(), 200); response = deleteRow(TABLE, ROW_1); assertEquals(response.getCode(), 200); @@ -349,6 +480,13 @@ public void testSingleCellGetPutPB() throws IOException, JAXBException { assertEquals(response.getCode(), 200); checkValuePB(TABLE, ROW_1, COLUMN_1, VALUE_2); + response = checkAndPutValuePB(TABLE, ROW_1, COLUMN_1, VALUE_2, VALUE_3); + assertEquals(response.getCode(), 200); + checkValuePB(TABLE, ROW_1, COLUMN_1, VALUE_3); + response = checkAndPutValueXML(TABLE, ROW_1, COLUMN_1, VALUE_3, VALUE_4); + assertEquals(response.getCode(), 200); + checkValuePB(TABLE, ROW_1, COLUMN_1, VALUE_4); + response = deleteRow(TABLE, ROW_1); assertEquals(response.getCode(), 200); } @@ -363,6 +501,7 @@ public void testSingleCellGetPutBinary() throws IOException { response = client.get(path, Constants.MIMETYPE_BINARY); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_BINARY, response.getHeader("content-type")); assertTrue(Bytes.equals(response.getBody(), body)); boolean foundTimestampHeader = false; for (Header header: response.getHeaders()) { @@ -386,6 +525,7 @@ public void testSingleCellGetJSON() throws IOException, JAXBException { Thread.yield(); response = client.get(path, Constants.MIMETYPE_JSON); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_JSON, response.getHeader("content-type")); response = deleteRow(TABLE, ROW_4); assertEquals(response.getCode(), 200); } @@ -526,6 +666,23 @@ public void testStartEndRowGetPutXML() throws IOException, JAXBException { } } + @Test + public void testInvalidCheckParam() throws IOException, JAXBException { + CellSetModel cellSetModel = new CellSetModel(); + RowModel rowModel = new RowModel(ROW_1); + rowModel.addCell(new CellModel(Bytes.toBytes(COLUMN_1), + Bytes.toBytes(VALUE_1))); + cellSetModel.addRow(rowModel); + StringWriter writer = new StringWriter(); + marshaller.marshal(cellSetModel, writer); + + final String path = "/" + TABLE + "/" + ROW_1 + "/" + COLUMN_1 + "?check=blah"; + + Response response = client.put(path, Constants.MIMETYPE_XML, + Bytes.toBytes(writer.toString())); + assertEquals(response.getCode(), 400); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java b/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java index 6e5849b30386..51e767ab9985 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -125,6 +124,7 @@ private static int fullTableScan(ScannerModel model) throws IOException { response = client.get(scannerURI, Constants.MIMETYPE_PROTOBUF); assertTrue(response.getCode() == 200 || response.getCode() == 204); if (response.getCode() == 200) { + assertEquals(Constants.MIMETYPE_PROTOBUF, response.getHeader("content-type")); CellSetModel cellSet = new CellSetModel(); cellSet.getObjectFromMessage(response.getBody()); Iterator rows = cellSet.getRows().iterator(); @@ -208,6 +208,7 @@ public void testSimpleScannerXML() throws IOException, JAXBException { // get a cell set response = client.get(scannerURI, Constants.MIMETYPE_XML); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); CellSetModel cellSet = (CellSetModel) unmarshaller.unmarshal(new ByteArrayInputStream(response.getBody())); // confirm batch size conformance @@ -251,6 +252,7 @@ public void testSimpleScannerPB() throws IOException { // get a cell set response = client.get(scannerURI, Constants.MIMETYPE_PROTOBUF); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF, response.getHeader("content-type")); CellSetModel cellSet = new CellSetModel(); cellSet.getObjectFromMessage(response.getBody()); // confirm batch size conformance @@ -293,6 +295,7 @@ public void testSimpleScannerBinary() throws IOException { // get a cell response = client.get(scannerURI, Constants.MIMETYPE_BINARY); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_BINARY, response.getHeader("content-type")); // verify that data was returned assertTrue(response.getBody().length > 0); // verify that the expected X-headers are present diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestScannersWithFilters.java b/src/test/java/org/apache/hadoop/hbase/rest/TestScannersWithFilters.java index 53f388c0007b..c8af1cc316a7 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/TestScannersWithFilters.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestScannersWithFilters.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -64,6 +63,7 @@ import org.apache.hadoop.hbase.util.Bytes; import static org.junit.Assert.*; + import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -226,6 +226,7 @@ private static void verifyScan(Scan s, long expectedRows, long expectedKeys) // get a cell set response = client.get(scannerURI, Constants.MIMETYPE_XML); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); CellSetModel cells = (CellSetModel) unmarshaller.unmarshal(new ByteArrayInputStream(response.getBody())); @@ -260,6 +261,7 @@ private static void verifyScanFull(Scan s, KeyValue [] kvs) // get a cell set response = client.get(scannerURI, Constants.MIMETYPE_XML); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); CellSetModel cellSet = (CellSetModel) unmarshaller.unmarshal(new ByteArrayInputStream(response.getBody())); @@ -313,6 +315,7 @@ private static void verifyScanNoEarlyOut(Scan s, long expectedRows, // get a cell set response = client.get(scannerURI, Constants.MIMETYPE_XML); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); CellSetModel cellSet = (CellSetModel) unmarshaller.unmarshal(new ByteArrayInputStream(response.getBody())); diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java b/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java index c967baba6031..e6e7ab8825fb 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -40,6 +39,7 @@ import org.apache.hadoop.hbase.util.Bytes; import static org.junit.Assert.*; + import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -110,17 +110,21 @@ public void testTableCreateAndDeleteXML() throws IOException, JAXBException { // retrieve the schema and validate it response = client.get(schemaPath, Constants.MIMETYPE_XML); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); model = fromXML(response.getBody()); TestTableSchemaModel.checkModel(model, TABLE1); - // delete the table - client.delete(schemaPath); - - // make sure HBase concurs - assertFalse(admin.tableExists(TABLE1)); + // test delete schema operation is forbidden in read-only mode + response = client.delete(schemaPath); + assertEquals(response.getCode(), 403); // return read-only setting back to default conf.set("hbase.rest.readonly", "false"); + + // delete the table and make sure HBase concurs + response = client.delete(schemaPath); + assertEquals(response.getCode(), 200); + assertFalse(admin.tableExists(TABLE1)); } @Test @@ -148,18 +152,30 @@ public void testTableCreateAndDeletePB() throws IOException, JAXBException { // retrieve the schema and validate it response = client.get(schemaPath, Constants.MIMETYPE_PROTOBUF); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF, response.getHeader("content-type")); model = new TableSchemaModel(); model.getObjectFromMessage(response.getBody()); TestTableSchemaModel.checkModel(model, TABLE2); - // delete the table - client.delete(schemaPath); + // retrieve the schema and validate it with alternate pbuf type + response = client.get(schemaPath, Constants.MIMETYPE_PROTOBUF_IETF); + assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF_IETF, response.getHeader("content-type")); + model = new TableSchemaModel(); + model.getObjectFromMessage(response.getBody()); + TestTableSchemaModel.checkModel(model, TABLE2); - // make sure HBase concurs - assertFalse(admin.tableExists(TABLE2)); + // test delete schema operation is forbidden in read-only mode + response = client.delete(schemaPath); + assertEquals(response.getCode(), 403); // return read-only setting back to default conf.set("hbase.rest.readonly", "false"); + + // delete the table and make sure HBase concurs + response = client.delete(schemaPath); + assertEquals(response.getCode(), 200); + assertFalse(admin.tableExists(TABLE2)); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestStatusResource.java b/src/test/java/org/apache/hadoop/hbase/rest/TestStatusResource.java index cffdcb67af93..e0bef84edbc3 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/TestStatusResource.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestStatusResource.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -35,6 +34,7 @@ import org.apache.hadoop.hbase.util.Bytes; import static org.junit.Assert.*; + import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -95,6 +95,7 @@ public static void tearDownAfterClass() throws Exception { public void testGetClusterStatusXML() throws IOException, JAXBException { Response response = client.get("/status/cluster", Constants.MIMETYPE_XML); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); StorageClusterStatusModel model = (StorageClusterStatusModel) context.createUnmarshaller().unmarshal( new ByteArrayInputStream(response.getBody())); @@ -103,16 +104,21 @@ public void testGetClusterStatusXML() throws IOException, JAXBException { @Test public void testGetClusterStatusPB() throws IOException { - Response response = client.get("/status/cluster", - Constants.MIMETYPE_PROTOBUF); + Response response = client.get("/status/cluster", Constants.MIMETYPE_PROTOBUF); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF, response.getHeader("content-type")); StorageClusterStatusModel model = new StorageClusterStatusModel(); model.getObjectFromMessage(response.getBody()); validate(model); + response = client.get("/status/cluster", Constants.MIMETYPE_PROTOBUF_IETF); + assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF_IETF, response.getHeader("content-type")); + model = new StorageClusterStatusModel(); + model.getObjectFromMessage(response.getBody()); + validate(model); } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); } - diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestTableResource.java b/src/test/java/org/apache/hadoop/hbase/rest/TestTableResource.java index 514db41ab907..de8ffc98af6b 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/TestTableResource.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestTableResource.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -46,6 +45,7 @@ import org.apache.hadoop.util.StringUtils; import static org.junit.Assert.*; + import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -179,12 +179,14 @@ void checkTableInfo(TableInfoModel model) { public void testTableListText() throws IOException { Response response = client.get("/", Constants.MIMETYPE_TEXT); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_TEXT, response.getHeader("content-type")); } @Test public void testTableListXML() throws IOException, JAXBException { Response response = client.get("/", Constants.MIMETYPE_XML); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); TableListModel model = (TableListModel) context.createUnmarshaller() .unmarshal(new ByteArrayInputStream(response.getBody())); @@ -195,29 +197,37 @@ public void testTableListXML() throws IOException, JAXBException { public void testTableListJSON() throws IOException { Response response = client.get("/", Constants.MIMETYPE_JSON); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_JSON, response.getHeader("content-type")); } @Test public void testTableListPB() throws IOException, JAXBException { Response response = client.get("/", Constants.MIMETYPE_PROTOBUF); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF, response.getHeader("content-type")); TableListModel model = new TableListModel(); model.getObjectFromMessage(response.getBody()); checkTableList(model); + response = client.get("/", Constants.MIMETYPE_PROTOBUF_IETF); + assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF_IETF, response.getHeader("content-type")); + model = new TableListModel(); + model.getObjectFromMessage(response.getBody()); + checkTableList(model); } @Test public void testTableInfoText() throws IOException { - Response response = client.get("/" + TABLE + "/regions", - Constants.MIMETYPE_TEXT); + Response response = client.get("/" + TABLE + "/regions", Constants.MIMETYPE_TEXT); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_TEXT, response.getHeader("content-type")); } @Test public void testTableInfoXML() throws IOException, JAXBException { - Response response = client.get("/" + TABLE + "/regions", - Constants.MIMETYPE_XML); + Response response = client.get("/" + TABLE + "/regions", Constants.MIMETYPE_XML); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); TableInfoModel model = (TableInfoModel) context.createUnmarshaller() .unmarshal(new ByteArrayInputStream(response.getBody())); @@ -226,19 +236,25 @@ public void testTableInfoXML() throws IOException, JAXBException { @Test public void testTableInfoJSON() throws IOException { - Response response = client.get("/" + TABLE + "/regions", - Constants.MIMETYPE_JSON); + Response response = client.get("/" + TABLE + "/regions", Constants.MIMETYPE_JSON); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_JSON, response.getHeader("content-type")); } @Test public void testTableInfoPB() throws IOException, JAXBException { - Response response = client.get("/" + TABLE + "/regions", - Constants.MIMETYPE_PROTOBUF); + Response response = client.get("/" + TABLE + "/regions", Constants.MIMETYPE_PROTOBUF); assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF, response.getHeader("content-type")); TableInfoModel model = new TableInfoModel(); model.getObjectFromMessage(response.getBody()); checkTableInfo(model); + response = client.get("/" + TABLE + "/regions", Constants.MIMETYPE_PROTOBUF_IETF); + assertEquals(response.getCode(), 200); + assertEquals(Constants.MIMETYPE_PROTOBUF_IETF, response.getHeader("content-type")); + model = new TableInfoModel(); + model.getObjectFromMessage(response.getBody()); + checkTableInfo(model); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/rest/TestVersionResource.java b/src/test/java/org/apache/hadoop/hbase/rest/TestVersionResource.java index a416499caa38..5f7519cf7a1a 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/TestVersionResource.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/TestVersionResource.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -26,8 +25,6 @@ import javax.xml.bind.JAXBContext; import javax.xml.bind.JAXBException; -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.rest.client.Client; @@ -38,6 +35,7 @@ import org.apache.hadoop.hbase.util.Bytes; import static org.junit.Assert.*; + import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -47,8 +45,6 @@ @Category(MediumTests.class) public class TestVersionResource { - private static final Log LOG = LogFactory.getLog(TestVersionResource.class); - private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); private static final HBaseRESTTestingUtility REST_TEST_UTIL = new HBaseRESTTestingUtility(); @@ -97,6 +93,7 @@ private static void validate(VersionModel model) { public void testGetStargateVersionText() throws IOException { Response response = client.get("/version", Constants.MIMETYPE_TEXT); assertTrue(response.getCode() == 200); + assertEquals(Constants.MIMETYPE_TEXT, response.getHeader("content-type")); String body = Bytes.toString(response.getBody()); assertTrue(body.length() > 0); assertTrue(body.contains(RESTServlet.VERSION_STRING)); @@ -114,34 +111,41 @@ public void testGetStargateVersionText() throws IOException { public void testGetStargateVersionXML() throws IOException, JAXBException { Response response = client.get("/version", Constants.MIMETYPE_XML); assertTrue(response.getCode() == 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); VersionModel model = (VersionModel) context.createUnmarshaller().unmarshal( new ByteArrayInputStream(response.getBody())); validate(model); - LOG.info("success retrieving Stargate version as XML"); } @Test public void testGetStargateVersionJSON() throws IOException { Response response = client.get("/version", Constants.MIMETYPE_JSON); assertTrue(response.getCode() == 200); + assertEquals(Constants.MIMETYPE_JSON, response.getHeader("content-type")); } @Test public void testGetStargateVersionPB() throws IOException { Response response = client.get("/version", Constants.MIMETYPE_PROTOBUF); assertTrue(response.getCode() == 200); + assertEquals(Constants.MIMETYPE_PROTOBUF, response.getHeader("content-type")); VersionModel model = new VersionModel(); model.getObjectFromMessage(response.getBody()); validate(model); - LOG.info("success retrieving Stargate version as protobuf"); + response = client.get("/version", Constants.MIMETYPE_PROTOBUF_IETF); + assertTrue(response.getCode() == 200); + assertEquals(Constants.MIMETYPE_PROTOBUF_IETF, response.getHeader("content-type")); + model = new VersionModel(); + model.getObjectFromMessage(response.getBody()); + validate(model); } @Test public void testGetStorageClusterVersionText() throws IOException { - Response response = client.get("/version/cluster", - Constants.MIMETYPE_TEXT); + Response response = client.get("/version/cluster", Constants.MIMETYPE_TEXT); assertTrue(response.getCode() == 200); + assertEquals(Constants.MIMETYPE_TEXT, response.getHeader("content-type")); } @Test @@ -149,19 +153,20 @@ public void testGetStorageClusterVersionXML() throws IOException, JAXBException { Response response = client.get("/version/cluster",Constants.MIMETYPE_XML); assertTrue(response.getCode() == 200); + assertEquals(Constants.MIMETYPE_XML, response.getHeader("content-type")); StorageClusterVersionModel clusterVersionModel = (StorageClusterVersionModel) context.createUnmarshaller().unmarshal( new ByteArrayInputStream(response.getBody())); assertNotNull(clusterVersionModel); assertNotNull(clusterVersionModel.getVersion()); - LOG.info("success retrieving storage cluster version as XML"); } @Test public void doTestGetStorageClusterVersionJSON() throws IOException { Response response = client.get("/version/cluster", Constants.MIMETYPE_JSON); assertTrue(response.getCode() == 200); + assertEquals(Constants.MIMETYPE_JSON, response.getHeader("content-type")); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteAdmin.java b/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteAdmin.java index 5a648aa45adf..77d59f1257bf 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteAdmin.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteAdmin.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,13 +19,22 @@ package org.apache.hadoop.hbase.rest.client; -import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.client.HBaseAdmin; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.util.List; + +import org.apache.hadoop.hbase.ClusterStatus; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.rest.HBaseRESTTestingUtility; -import org.apache.hadoop.hbase.rest.client.Client; +import org.apache.hadoop.hbase.rest.model.StorageClusterStatusModel; +import org.apache.hadoop.hbase.rest.model.TableModel; +import org.apache.hadoop.hbase.rest.model.VersionModel; import org.apache.hadoop.hbase.util.Bytes; - -import static org.junit.Assert.*; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -34,13 +42,15 @@ @Category(MediumTests.class) public class TestRemoteAdmin { - private static final String TABLE_1 = "TestRemoteAdmin_Table_1"; - private static final byte[] COLUMN_1 = Bytes.toBytes("a"); - static final HTableDescriptor DESC_1 = new HTableDescriptor(TABLE_1); private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); - private static final HBaseRESTTestingUtility REST_TEST_UTIL = + private static final HBaseRESTTestingUtility REST_TEST_UTIL = new HBaseRESTTestingUtility(); + private static final String TABLE_1 = "TestRemoteAdmin_Table_1"; + private static final String TABLE_2 = TABLE_1 + System.currentTimeMillis(); + private static final byte[] COLUMN_1 = Bytes.toBytes("a"); + static final HTableDescriptor DESC_1 = new HTableDescriptor(TABLE_1); + static final HTableDescriptor DESC_2 = new HTableDescriptor(TABLE_2); private static RemoteAdmin remoteAdmin; @BeforeClass @@ -70,8 +80,63 @@ public void testCreateAnDeleteTable() throws Exception { assertFalse(remoteAdmin.isTableAvailable(TABLE_1)); } + @Test + public void testGetRestVersion() throws Exception { + + VersionModel RETURNED_REST_VERSION = remoteAdmin.getRestVersion(); + System.out.print("Returned version is: " + RETURNED_REST_VERSION); + + // Assert that it contains info about rest version, OS, JVM + assertTrue("Returned REST version did not contain info about rest.", + RETURNED_REST_VERSION.toString().contains("rest")); + assertTrue("Returned REST version did not contain info about the JVM.", + RETURNED_REST_VERSION.toString().contains("JVM")); + assertTrue("Returned REST version did not contain info about OS.", + RETURNED_REST_VERSION.toString().contains("OS")); + } + + @Test + public void testClusterVersion() throws Exception { + // testing the /version/cluster endpoint + final String HBASE_VERSION = TEST_UTIL.getHBaseCluster().getClusterStatus() + .getHBaseVersion(); + assertEquals("Cluster status from REST API did not match. ", HBASE_VERSION, + remoteAdmin.getClusterVersion().getVersion()); + } + + @Test + public void testClusterStatus() throws Exception { + + ClusterStatus status = TEST_UTIL.getHBaseClusterInterface() + .getClusterStatus(); + StorageClusterStatusModel returnedStatus = remoteAdmin.getClusterStatus(); + assertEquals( + "Region count from cluster status and returned status did not match up. ", + status.getRegionsCount(), returnedStatus.getRegions()); + assertEquals( + "Dead server count from cluster status and returned status did not match up. ", + status.getDeadServers(), returnedStatus.getDeadNodes().size()); + } + + @Test + public void testListTables() throws Exception { + + remoteAdmin.createTable(DESC_2); + List tableList = remoteAdmin.getTableList().getTables(); + System.out.println("List of tables is: "); + boolean found = false; + for (TableModel tm : tableList) { + + if (tm.getName().equals(TABLE_2)) { + found = true; + break; + } + } + assertTrue("Table " + TABLE_2 + " was not found by get request to '/'", + found); + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); } - diff --git a/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteAdminRetries.java b/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteAdminRetries.java new file mode 100644 index 000000000000..cb801e3cf9da --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteAdminRetries.java @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.rest.client; + +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.util.regex.Pattern; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Tests {@link RemoteAdmin} retries. + */ +@Category(SmallTests.class) +public class TestRemoteAdminRetries { + + private static final int SLEEP_TIME = 50; + private static final int RETRIES = 3; + private static final long MAX_TIME = SLEEP_TIME * (RETRIES - 1); + + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + private RemoteAdmin remoteAdmin; + private Client client; + + @Before + public void setup() throws Exception { + client = mock(Client.class); + Response response = new Response(509); + when(client.get(anyString(), anyString())).thenReturn(response); + when(client.delete(anyString())).thenReturn(response); + when(client.put(anyString(), anyString(), any(byte[].class))).thenReturn(response); + when(client.post(anyString(), anyString(), any(byte[].class))).thenReturn(response); + Configuration configuration = TEST_UTIL.getConfiguration(); + + configuration.setInt("hbase.rest.client.max.retries", RETRIES); + configuration.setInt("hbase.rest.client.sleep", SLEEP_TIME); + + remoteAdmin = new RemoteAdmin(client, TEST_UTIL.getConfiguration(), "MyTable"); + } + + @Test + public void testFailingGetRestVersion() throws Exception { + testTimedOutGetCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteAdmin.getRestVersion(); + } + }); + } + + @Test + public void testFailingGetClusterStatus() throws Exception { + testTimedOutGetCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteAdmin.getClusterStatus(); + } + }); + } + + @Test + public void testFailingGetClusterVersion() throws Exception { + testTimedOutGetCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteAdmin.getClusterVersion(); + } + }); + } + + @Test + public void testFailingGetTableAvailable() throws Exception { + testTimedOutCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteAdmin.isTableAvailable(Bytes.toBytes("TestTable")); + } + }); + } + + @Test + public void testFailingCreateTable() throws Exception { + testTimedOutCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteAdmin.createTable(new HTableDescriptor(Bytes.toBytes("TestTable"))); + } + }); + verify(client, times(RETRIES)).put(anyString(), anyString(), any(byte[].class)); + } + + @Test + public void testFailingDeleteTable() throws Exception { + testTimedOutCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteAdmin.deleteTable("TestTable"); + } + }); + verify(client, times(RETRIES)).delete(anyString()); + } + + @Test + public void testFailingGetTableList() throws Exception { + testTimedOutGetCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteAdmin.getTableList(); + } + }); + } + + private void testTimedOutGetCall(CallExecutor callExecutor) throws Exception { + testTimedOutCall(callExecutor); + verify(client, times(RETRIES)).get(anyString(), anyString()); + } + + private void testTimedOutCall(CallExecutor callExecutor) throws Exception { + long start = System.currentTimeMillis(); + try { + callExecutor.run(); + fail("should be timeout exception!"); + } catch (IOException e) { + assertTrue(Pattern.matches(".*MyTable.*timed out", e.toString())); + } + assertTrue((System.currentTimeMillis() - start) > MAX_TIME); + } + + private static interface CallExecutor { + void run() throws Exception; + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteHTableRetries.java b/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteHTableRetries.java new file mode 100644 index 000000000000..547dfab8101a --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteHTableRetries.java @@ -0,0 +1,193 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.rest.client; + +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.util.Arrays; +import java.util.regex.Pattern; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.client.Delete; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test RemoteHTable retries. + */ +@Category(SmallTests.class) +public class TestRemoteHTableRetries { + + private static final int SLEEP_TIME = 50; + private static final int RETRIES = 3; + private static final long MAX_TIME = SLEEP_TIME * (RETRIES - 1); + + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + private static final byte[] ROW_1 = Bytes.toBytes("testrow1"); + private static final byte[] COLUMN_1 = Bytes.toBytes("a"); + private static final byte[] QUALIFIER_1 = Bytes.toBytes("1"); + private static final byte[] VALUE_1 = Bytes.toBytes("testvalue1"); + + private Client client; + private RemoteHTable remoteTable; + + @Before + public void setup() throws Exception { + client = mock(Client.class); + Response response = new Response(509); + when(client.get(anyString(), anyString())).thenReturn(response); + when(client.delete(anyString())).thenReturn(response); + when(client.put(anyString(), anyString(), any(byte[].class))).thenReturn( + response); + when(client.post(anyString(), anyString(), any(byte[].class))).thenReturn( + response); + + Configuration configuration = TEST_UTIL.getConfiguration(); + configuration.setInt("hbase.rest.client.max.retries", RETRIES); + configuration.setInt("hbase.rest.client.sleep", SLEEP_TIME); + + remoteTable = new RemoteHTable(client, TEST_UTIL.getConfiguration(), + "MyTable"); + } + + @After + public void tearDownAfterClass() throws Exception { + remoteTable.close(); + } + + @Test + public void testDelete() throws Exception { + testTimedOutCall(new CallExecutor() { + @Override + public void run() throws Exception { + Delete delete = new Delete(Bytes.toBytes("delete")); + remoteTable.delete(delete); + } + }); + verify(client, times(RETRIES)).delete(anyString()); + } + + @Test + public void testGet() throws Exception { + testTimedOutGetCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteTable.get(new Get(Bytes.toBytes("Get"))); + } + }); + } + + @Test + public void testSingleRowPut() throws Exception { + testTimedOutCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteTable.put(new Put(Bytes.toBytes("Row"))); + } + }); + verify(client, times(RETRIES)).put(anyString(), anyString(), any(byte[].class)); + } + + @Test + public void testMultiRowPut() throws Exception { + testTimedOutCall(new CallExecutor() { + @Override + public void run() throws Exception { + Put[] puts = { new Put(Bytes.toBytes("Row1")), + new Put(Bytes.toBytes("Row2")) }; + remoteTable.put(Arrays.asList(puts)); + } + }); + verify(client, times(RETRIES)).put(anyString(), anyString(), any(byte[].class)); + } + + @Test + public void testGetScanner() throws Exception { + testTimedOutCall(new CallExecutor() { + @Override + public void run() throws Exception { + remoteTable.getScanner(new Scan()); + } + }); + verify(client, times(RETRIES)).post(anyString(), anyString(), any(byte[].class)); + } + + @Test + public void testCheckAndPut() throws Exception { + testTimedOutCall(new CallExecutor() { + @Override + public void run() throws Exception { + Put put = new Put(ROW_1); + put.add(COLUMN_1, QUALIFIER_1, VALUE_1); + remoteTable.checkAndPut(ROW_1, COLUMN_1, QUALIFIER_1, VALUE_1, put ); + } + }); + verify(client, times(RETRIES)).put(anyString(), anyString(), any(byte[].class)); + } + + @Test + public void testCheckAndDelete() throws Exception { + testTimedOutCall(new CallExecutor() { + @Override + public void run() throws Exception { + Put put = new Put(ROW_1); + put.add(COLUMN_1, QUALIFIER_1, VALUE_1); + Delete delete= new Delete(ROW_1); + remoteTable.checkAndDelete(ROW_1, COLUMN_1, QUALIFIER_1, VALUE_1, delete ); + } + }); + } + + private void testTimedOutGetCall(CallExecutor callExecutor) throws Exception { + testTimedOutCall(callExecutor); + verify(client, times(RETRIES)).get(anyString(), anyString()); + } + + private void testTimedOutCall(CallExecutor callExecutor) throws Exception { + long start = System.currentTimeMillis(); + try { + callExecutor.run(); + fail("should be timeout exception!"); + } catch (IOException e) { + assertTrue(Pattern.matches(".*request timed out", e.toString())); + } + assertTrue((System.currentTimeMillis() - start) > MAX_TIME); + } + + private static interface CallExecutor { + void run() throws Exception; + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java b/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java index 7b7a67063f4b..4d1f49c1a5e2 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,13 +19,24 @@ package org.apache.hadoop.hbase.rest.client; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertFalse; + import java.io.IOException; import java.util.ArrayList; +import java.util.Collections; +import java.util.Iterator; import java.util.List; -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.hbase.*; +import org.apache.commons.httpclient.Header; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HBaseAdmin; @@ -36,20 +46,16 @@ import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.rest.HBaseRESTTestingUtility; -import org.apache.hadoop.hbase.rest.client.Client; -import org.apache.hadoop.hbase.rest.client.Cluster; -import org.apache.hadoop.hbase.rest.client.RemoteHTable; import org.apache.hadoop.hbase.util.Bytes; - -import static org.junit.Assert.*; +import org.junit.After; import org.junit.AfterClass; +import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; @Category(MediumTests.class) public class TestRemoteTable { - private static final Log LOG = LogFactory.getLog(TestRemoteTable.class); private static final String TABLE = "TestRemoteTable"; private static final byte[] ROW_1 = Bytes.toBytes("testrow1"); private static final byte[] ROW_2 = Bytes.toBytes("testrow2"); @@ -70,24 +76,29 @@ public class TestRemoteTable { private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); private static final HBaseRESTTestingUtility REST_TEST_UTIL = new HBaseRESTTestingUtility(); - private static RemoteHTable remoteTable; + private RemoteHTable remoteTable; @BeforeClass public static void setUpBeforeClass() throws Exception { TEST_UTIL.startMiniCluster(); REST_TEST_UTIL.startServletContainer(TEST_UTIL.getConfiguration()); + } + + @Before + public void before() throws Exception { HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); - LOG.info("Admin Connection=" + admin.getConnection() + ", " + - admin.getConnection().getZooKeeperWatcher()); - if (!admin.tableExists(TABLE)) { - HTableDescriptor htd = new HTableDescriptor(TABLE); - htd.addFamily(new HColumnDescriptor(COLUMN_1)); - htd.addFamily(new HColumnDescriptor(COLUMN_2)); - htd.addFamily(new HColumnDescriptor(COLUMN_3)); - admin.createTable(htd); - HTable table = new HTable(TEST_UTIL.getConfiguration(), TABLE); - LOG.info("Table connection=" + table.getConnection() + ", " + - admin.getConnection().getZooKeeperWatcher()); + if (admin.tableExists(TABLE)) { + if (admin.isTableEnabled(TABLE)) admin.disableTable(TABLE); + admin.deleteTable(TABLE); + } + HTableDescriptor htd = new HTableDescriptor(TABLE); + htd.addFamily(new HColumnDescriptor(COLUMN_1).setMaxVersions(3)); + htd.addFamily(new HColumnDescriptor(COLUMN_2).setMaxVersions(3)); + htd.addFamily(new HColumnDescriptor(COLUMN_3).setMaxVersions(3)); + admin.createTable(htd); + HTable table = null; + try { + table = new HTable(TEST_UTIL.getConfiguration(), TABLE); Put put = new Put(ROW_1); put.add(COLUMN_1, QUALIFIER_1, TS_2, VALUE_1); table.put(put); @@ -97,25 +108,36 @@ public static void setUpBeforeClass() throws Exception { put.add(COLUMN_2, QUALIFIER_2, TS_2, VALUE_2); table.put(put); table.flushCommits(); + } finally { + if (null != table) table.close(); } remoteTable = new RemoteHTable( new Client(new Cluster().add("localhost", REST_TEST_UTIL.getServletPort())), - TEST_UTIL.getConfiguration(), TABLE, null); + TEST_UTIL.getConfiguration(), TABLE); } - + + @After + public void after() throws Exception { + remoteTable.close(); + } + @AfterClass public static void tearDownAfterClass() throws Exception { - remoteTable.close(); REST_TEST_UTIL.shutdownServletContainer(); TEST_UTIL.shutdownMiniCluster(); } @Test public void testGetTableDescriptor() throws IOException { - HTableDescriptor local = new HTable(TEST_UTIL.getConfiguration(), - TABLE).getTableDescriptor(); - assertEquals(remoteTable.getTableDescriptor(), local); + HTable table = null; + try { + table = new HTable(TEST_UTIL.getConfiguration(), TABLE); + HTableDescriptor local = table.getTableDescriptor(); + assertEquals(remoteTable.getTableDescriptor(), local); + } finally { + if (null != table) table.close(); + } } @Test @@ -221,6 +243,45 @@ public void testGet() throws IOException { assertEquals(2, count); } + @Test + public void testMultiGet() throws Exception { + ArrayList gets = new ArrayList(); + gets.add(new Get(ROW_1)); + gets.add(new Get(ROW_2)); + Result[] results = remoteTable.get(gets); + assertNotNull(results); + assertEquals(2, results.length); + assertEquals(1, results[0].size()); + assertEquals(2, results[1].size()); + + //Test Versions + gets = new ArrayList(); + Get g = new Get(ROW_1); + g.setMaxVersions(3); + gets.add(g); + gets.add(new Get(ROW_2)); + results = remoteTable.get(gets); + assertNotNull(results); + assertEquals(2, results.length); + assertEquals(1, results[0].size()); + assertEquals(3, results[1].size()); + + //404 + gets = new ArrayList(); + gets.add(new Get(Bytes.toBytes("RESALLYREALLYNOTTHERE"))); + results = remoteTable.get(gets); + assertNotNull(results); + assertEquals(0, results.length); + + gets = new ArrayList(); + gets.add(new Get(Bytes.toBytes("RESALLYREALLYNOTTHERE"))); + gets.add(new Get(ROW_1)); + gets.add(new Get(ROW_2)); + results = remoteTable.get(gets); + assertNotNull(results); + assertEquals(0, results.length); + } + @Test public void testPut() throws IOException { Put put = new Put(ROW_3); @@ -262,8 +323,11 @@ public void testPut() throws IOException { value = result.getValue(COLUMN_2, QUALIFIER_2); assertNotNull(value); assertTrue(Bytes.equals(VALUE_2, value)); + + assertTrue(Bytes.equals(Bytes.toBytes("TestRemoteTable"), remoteTable.getTableName())); } + @Test public void testDelete() throws IOException { Put put = new Put(ROW_3); put.add(COLUMN_1, QUALIFIER_1, VALUE_1); @@ -284,7 +348,21 @@ public void testDelete() throws IOException { Delete delete = new Delete(ROW_3); delete.deleteColumn(COLUMN_2, QUALIFIER_2); remoteTable.delete(delete); - + + get = new Get(ROW_3); + get.addFamily(COLUMN_1); + get.addFamily(COLUMN_2); + result = remoteTable.get(get); + value1 = result.getValue(COLUMN_1, QUALIFIER_1); + value2 = result.getValue(COLUMN_2, QUALIFIER_2); + assertNotNull(value1); + assertTrue(Bytes.equals(VALUE_1, value1)); + assertNull(value2); + + delete = new Delete(ROW_3); + delete.setTimestamp(1L); + remoteTable.delete(delete); + get = new Get(ROW_3); get.addFamily(COLUMN_1); get.addFamily(COLUMN_2); @@ -308,6 +386,7 @@ public void testDelete() throws IOException { assertNull(value2); } + @Test public void testScanner() throws IOException { List puts = new ArrayList(); Put put = new Put(ROW_1); @@ -331,21 +410,123 @@ public void testScanner() throws IOException { assertEquals(1, results.length); assertTrue(Bytes.equals(ROW_1, results[0].getRow())); - results = scanner.next(3); + Result result = scanner.next(); + assertNotNull(result); + assertTrue(Bytes.equals(ROW_2, result.getRow())); + + results = scanner.next(2); assertNotNull(results); - assertEquals(3, results.length); - assertTrue(Bytes.equals(ROW_2, results[0].getRow())); - assertTrue(Bytes.equals(ROW_3, results[1].getRow())); - assertTrue(Bytes.equals(ROW_4, results[2].getRow())); + assertEquals(2, results.length); + assertTrue(Bytes.equals(ROW_3, results[0].getRow())); + assertTrue(Bytes.equals(ROW_4, results[1].getRow())); results = scanner.next(1); assertNull(results); + scanner.close(); + + scanner = remoteTable.getScanner(COLUMN_1); + results = scanner.next(4); + assertNotNull(results); + assertEquals(4, results.length); + assertTrue(Bytes.equals(ROW_1, results[0].getRow())); + assertTrue(Bytes.equals(ROW_2, results[1].getRow())); + assertTrue(Bytes.equals(ROW_3, results[2].getRow())); + assertTrue(Bytes.equals(ROW_4, results[3].getRow())); + + scanner.close(); + scanner = remoteTable.getScanner(COLUMN_1, QUALIFIER_1); + results = scanner.next(4); + assertNotNull(results); + assertEquals(4, results.length); + assertTrue(Bytes.equals(ROW_1, results[0].getRow())); + assertTrue(Bytes.equals(ROW_2, results[1].getRow())); + assertTrue(Bytes.equals(ROW_3, results[2].getRow())); + assertTrue(Bytes.equals(ROW_4, results[3].getRow())); scanner.close(); + assertTrue(remoteTable.isAutoFlush()); + } + + @Test + public void testCheckAndDelete() throws IOException { + Get get = new Get(ROW_1); + Result result = remoteTable.get(get); + byte[] value1 = result.getValue(COLUMN_1, QUALIFIER_1); + byte[] value2 = result.getValue(COLUMN_2, QUALIFIER_2); + assertNotNull(value1); + assertTrue(Bytes.equals(VALUE_1, value1)); + assertNull(value2); + assertTrue(remoteTable.exists(get)); + assertEquals(1, remoteTable.get(Collections.singletonList(get)).length); + Delete delete = new Delete(ROW_1); + + remoteTable.checkAndDelete(ROW_1, COLUMN_1, QUALIFIER_1, VALUE_1, delete); + assertFalse(remoteTable.exists(get)); + + Put put = new Put(ROW_1); + put.add(COLUMN_1, QUALIFIER_1, VALUE_1); + remoteTable.put(put); + + assertTrue(remoteTable.checkAndPut(ROW_1, COLUMN_1, QUALIFIER_1, VALUE_1, put)); + assertFalse(remoteTable.checkAndPut(ROW_1, COLUMN_1, QUALIFIER_1, VALUE_2, put)); + } + + /** + * Test RemoteHable.Scanner.iterator method + */ + @Test + public void testIteratorScaner() throws IOException { + List puts = new ArrayList(); + Put put = new Put(ROW_1); + put.add(COLUMN_1, QUALIFIER_1, VALUE_1); + puts.add(put); + put = new Put(ROW_2); + put.add(COLUMN_1, QUALIFIER_1, VALUE_1); + puts.add(put); + put = new Put(ROW_3); + put.add(COLUMN_1, QUALIFIER_1, VALUE_1); + puts.add(put); + put = new Put(ROW_4); + put.add(COLUMN_1, QUALIFIER_1, VALUE_1); + puts.add(put); + remoteTable.put(puts); + + ResultScanner scanner = remoteTable.getScanner(new Scan()); + Iterator iterator = scanner.iterator(); + assertTrue(iterator.hasNext()); + int counter = 0; + while (iterator.hasNext()) { + iterator.next(); + counter++; + } + assertEquals(4, counter); + } + + /** + * Test a some methods of class Response. + */ + @Test + public void testResponse(){ + Response response = new Response(200); + assertEquals(200, response.getCode()); + Header[] headers = new Header[2]; + headers[0] = new Header("header1", "value1"); + headers[1] = new Header("header2", "value2"); + response = new Response(200, headers); + assertEquals("value1", response.getHeader("header1")); + assertFalse(response.hasBody()); + response.setCode(404); + assertEquals(404, response.getCode()); + headers = new Header[2]; + headers[0] = new Header("header1", "value1.1"); + headers[1] = new Header("header2", "value2"); + response.setHeaders(headers); + assertEquals("value1.1", response.getHeader("header1")); + response.setBody(Bytes.toBytes("body")); + assertTrue(response.hasBody()); } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); -} - +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellModel.java index 4e7a56bc6c6c..8eb93a0639aa 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellSetModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellSetModel.java index 3d358c065f82..e1f2f8d9c3d8 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellSetModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellSetModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestColumnSchemaModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestColumnSchemaModel.java index 62014a347f23..bdeaa87a0db8 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestColumnSchemaModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestColumnSchemaModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestRowModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestRowModel.java index 80386456cedf..ca2de3ab323e 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestRowModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestRowModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestScannerModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestScannerModel.java index dc55f3accfc6..4d244c035387 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestScannerModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestScannerModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterStatusModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterStatusModel.java index 32e91d921f75..cc3fec80ec71 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterStatusModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterStatusModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -45,19 +44,25 @@ public class TestStorageClusterStatusModel extends TestCase { " name=\"test1\" maxHeapSizeMB=\"1024\" heapSizeMB=\"128\">" + "" + + " memstoreSizeMB=\"0\" readRequestsCount=\"1\"" + + " writeRequestsCount=\"2\" rootIndexSizeKB=\"1\"" + + " totalStaticIndexSizeKB=\"1\" totalStaticBloomSizeKB=\"1\"" + + " totalCompactingKVs=\"1\" currentCompactedKVs=\"1\"/>" + "" + ""+ + " memstoreSizeMB=\"0\" readRequestsCount=\"1\"" + + " writeRequestsCount=\"2\" rootIndexSizeKB=\"1\"" + + " totalStaticIndexSizeKB=\"1\" totalStaticBloomSizeKB=\"1\"" + + " totalCompactingKVs=\"1\" currentCompactedKVs=\"1\"/>"+ ""; - private static final String AS_PB = -"Ci0KBXRlc3QxEOO6i+eeJBgAIIABKIAIMhUKCS1ST09ULSwsMBABGAEgACgAMAAKOQoFdGVzdDIQ"+ -"/pKx8J4kGAAggAQogAgyIQoVLk1FVEEuLCwxMjQ2MDAwMDQzNzI0EAEYASAAKAAwABgCIAApAAAA"+ -"AAAA8D8="; - + private static final String AS_PB = + "CjsKBXRlc3QxEOO6i+eeJBgAIIABKIAIMiMKCS1ST09ULSwsMBABGAEgACgAMAA4AUACSAFQAVgB" + + "YAFoAQpHCgV0ZXN0MhD+krHwniQYACCABCiACDIvChUuTUVUQS4sLDEyNDYwMDAwNDM3MjQQARgB" + + "IAAoADAAOAFAAkgBUAFYAWABaAEYAiAAKQAAAAAAAPA/"; + private JAXBContext context; public TestStorageClusterStatusModel() throws JAXBException { @@ -71,9 +76,10 @@ private StorageClusterStatusModel buildTestModel() { model.setRequests(0); model.setAverageLoad(1.0); model.addLiveNode("test1", 1245219839331L, 128, 1024) - .addRegion(Bytes.toBytes("-ROOT-,,0"), 1, 1, 0, 0, 0); + .addRegion(Bytes.toBytes("-ROOT-,,0"), 1, 1, 0, 0, 0, 1, 2, 1, 1, 1, 1, 1); model.addLiveNode("test2", 1245239331198L, 512, 1024) - .addRegion(Bytes.toBytes(".META.,,1246000043724"),1, 1, 0, 0, 0); + .addRegion(Bytes.toBytes(".META.,,1246000043724"),1, 1, 0, 0, 0, + 1, 2, 1, 1, 1, 1, 1); return model; } @@ -119,6 +125,13 @@ private void checkModel(StorageClusterStatusModel model) { assertEquals(region.getStorefileSizeMB(), 0); assertEquals(region.getMemstoreSizeMB(), 0); assertEquals(region.getStorefileIndexSizeMB(), 0); + assertEquals(region.getReadRequestsCount(), 1); + assertEquals(region.getWriteRequestsCount(), 2); + assertEquals(region.getRootIndexSizeKB(), 1); + assertEquals(region.getTotalStaticIndexSizeKB(), 1); + assertEquals(region.getTotalStaticBloomSizeKB(), 1); + assertEquals(region.getTotalCompactingKVs(), 1); + assertEquals(region.getCurrentCompactedKVs(), 1); assertFalse(regions.hasNext()); node = nodes.next(); assertEquals(node.getName(), "test2"); @@ -133,6 +146,14 @@ private void checkModel(StorageClusterStatusModel model) { assertEquals(region.getStorefileSizeMB(), 0); assertEquals(region.getMemstoreSizeMB(), 0); assertEquals(region.getStorefileIndexSizeMB(), 0); + assertEquals(region.getReadRequestsCount(), 1); + assertEquals(region.getWriteRequestsCount(), 2); + assertEquals(region.getRootIndexSizeKB(), 1); + assertEquals(region.getTotalStaticIndexSizeKB(), 1); + assertEquals(region.getTotalStaticBloomSizeKB(), 1); + assertEquals(region.getTotalCompactingKVs(), 1); + assertEquals(region.getCurrentCompactedKVs(), 1); + assertFalse(regions.hasNext()); assertFalse(nodes.hasNext()); } diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterVersionModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterVersionModel.java index 3a014860f0a5..f15b151db629 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterVersionModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterVersionModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableInfoModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableInfoModel.java index 7fb74a7b79dc..f2455709012e 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableInfoModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableInfoModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableListModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableListModel.java index a3d1bf987798..b05a802646ee 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableListModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableListModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java index b6f0ab5d4ba6..90df8ce335c0 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableSchemaModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableSchemaModel.java index c9b298973151..851223f2d1a8 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableSchemaModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableSchemaModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/rest/model/TestVersionModel.java b/src/test/java/org/apache/hadoop/hbase/rest/model/TestVersionModel.java index 2ecb7d90d86d..dec09cf24bff 100644 --- a/src/test/java/org/apache/hadoop/hbase/rest/model/TestVersionModel.java +++ b/src/test/java/org/apache/hadoop/hbase/rest/model/TestVersionModel.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/security/TestUser.java b/src/test/java/org/apache/hadoop/hbase/security/TestUser.java index 098787c726f4..ca088047fe93 100644 --- a/src/test/java/org/apache/hadoop/hbase/security/TestUser.java +++ b/src/test/java/org/apache/hadoop/hbase/security/TestUser.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -102,6 +101,8 @@ public void testGetCurrent() throws Exception { User u = User.getCurrent(); assertNotNull(u); assertEquals(user1.getName(), u.getName()); + assertEquals(user1, u); + assertEquals(user1.hashCode(), u.hashCode()); } } diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java b/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java new file mode 100644 index 000000000000..2613f9ad1c13 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java @@ -0,0 +1,501 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.List; +import java.util.Set; +import java.util.HashSet; +import java.util.TreeSet; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.PathFilter; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.TableNotEnabledException; +import org.apache.hadoop.hbase.client.Durability; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.io.HFileLink; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.master.MasterFileSystem; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSTableDescriptors; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.FSVisitor; +import org.apache.hadoop.hbase.util.MD5Hash; +import org.junit.Assert; + +/** + * Utilities class for snapshots + */ +public class SnapshotTestingUtils { + + private static final Log LOG = LogFactory.getLog(SnapshotTestingUtils.class); + private static byte[] KEYS = Bytes.toBytes("0123456789"); + + /** + * Assert that we don't have any snapshots lists + * @throws IOException if the admin operation fails + */ + public static void assertNoSnapshots(HBaseAdmin admin) throws IOException { + assertEquals("Have some previous snapshots", 0, admin.listSnapshots().size()); + } + + /** + * Make sure that there is only one snapshot returned from the master and its + * name and table match the passed in parameters. + */ + public static List assertExistsMatchingSnapshot( + HBaseAdmin admin, String snapshotName, String tableName) + throws IOException { + // list the snapshot + List snapshots = admin.listSnapshots(); + + List returnedSnapshots = new ArrayList(); + for (SnapshotDescription sd : snapshots) { + if (snapshotName.equals(sd.getName()) && + tableName.equals(sd.getTable())) { + returnedSnapshots.add(sd); + } + } + + Assert.assertTrue("No matching snapshots found.", returnedSnapshots.size()>0); + return returnedSnapshots; + } + + /** + * Make sure that there is only one snapshot returned from the master + */ + public static void assertOneSnapshotThatMatches(HBaseAdmin admin, + HSnapshotDescription snapshot) throws IOException { + assertOneSnapshotThatMatches(admin, snapshot.getName(), + snapshot.getTable()); + } + + /** + * Make sure that there is only one snapshot returned from the master and its name and table match + * the passed in parameters. + */ + public static void assertOneSnapshotThatMatches(HBaseAdmin admin, SnapshotDescription snapshot) + throws IOException { + assertOneSnapshotThatMatches(admin, snapshot.getName(), snapshot.getTable()); + } + + /** + * Make sure that there is only one snapshot returned from the master and its + * name and table match the passed in parameters. + */ + public static List assertOneSnapshotThatMatches( + HBaseAdmin admin, String snapshotName, String tableName) + throws IOException { + // list the snapshot + List snapshots = admin.listSnapshots(); + + assertEquals("Should only have 1 snapshot", 1, snapshots.size()); + assertEquals(snapshotName, snapshots.get(0).getName()); + assertEquals(tableName, snapshots.get(0).getTable()); + + return snapshots; + } + + /** + * Make sure that there is only one snapshot returned from the master and its + * name and table match the passed in parameters. + */ + public static List assertOneSnapshotThatMatches( + HBaseAdmin admin, byte[] snapshot, byte[] tableName) throws IOException { + return assertOneSnapshotThatMatches(admin, Bytes.toString(snapshot), + Bytes.toString(tableName)); + } + + /** + * Confirm that the snapshot contains references to all the files that should + * be in the snapshot. + */ + public static void confirmSnapshotValid( + SnapshotDescription snapshotDescriptor, byte[] tableName, + byte[] testFamily, Path rootDir, HBaseAdmin admin, FileSystem fs, + boolean requireLogs, Path logsDir, Set snapshotServers) + throws IOException { + ArrayList nonEmptyTestFamilies = new ArrayList(1); + nonEmptyTestFamilies.add(testFamily); + confirmSnapshotValid(snapshotDescriptor, Bytes.toString(tableName), + nonEmptyTestFamilies, null, rootDir, admin, fs, requireLogs, + logsDir, snapshotServers); + } + + /** + * Confirm that the snapshot has no references files but only metadata. + */ + public static void confirmEmptySnapshotValid( + SnapshotDescription snapshotDescriptor, byte[] tableName, + byte[] testFamily, Path rootDir, HBaseAdmin admin, FileSystem fs, + boolean requireLogs, Path logsDir, Set snapshotServers) + throws IOException { + ArrayList emptyTestFamilies = new ArrayList(1); + emptyTestFamilies.add(testFamily); + confirmSnapshotValid(snapshotDescriptor, Bytes.toString(tableName), + null, emptyTestFamilies, rootDir, admin, fs, requireLogs, + logsDir, snapshotServers); + } + + /** + * Confirm that the snapshot contains references to all the files that should + * be in the snapshot. This method also perform some redundant check like + * the existence of the snapshotinfo or the regioninfo which are done always + * by the MasterSnapshotVerifier, at the end of the snapshot operation. + */ + public static void confirmSnapshotValid( + SnapshotDescription snapshotDescriptor, String tableName, + List nonEmptyTestFamilies, List emptyTestFamilies, + Path rootDir, HBaseAdmin admin, FileSystem fs, boolean requireLogs, + Path logsDir, Set snapshotServers) throws IOException { + // check snapshot dir + Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir( + snapshotDescriptor, rootDir); + assertTrue(fs.exists(snapshotDir)); + + // check snapshot info + Path snapshotinfo = new Path(snapshotDir, SnapshotDescriptionUtils.SNAPSHOTINFO_FILE); + assertTrue(fs.exists(snapshotinfo)); + + // check the logs dir + if (requireLogs) { + TakeSnapshotUtils.verifyAllLogsGotReferenced(fs, logsDir, + snapshotServers, snapshotDescriptor, new Path(snapshotDir, + HConstants.HREGION_LOGDIR_NAME)); + } + + // check the table info + HTableDescriptor desc = FSTableDescriptors.getTableDescriptorFromFs(fs, rootDir, tableName); + HTableDescriptor snapshotDesc = FSTableDescriptors.getTableDescriptorFromFs(fs, snapshotDir); + assertEquals(desc, snapshotDesc); + + // Extract regions and families with store files + final Set snapshotRegions = new HashSet(); + final Set snapshotFamilies = new TreeSet(Bytes.BYTES_COMPARATOR); + FSVisitor.visitTableStoreFiles(fs, snapshotDir, new FSVisitor.StoreFileVisitor() { + public void storeFile(final String region, final String family, final String hfileName) + throws IOException { + snapshotRegions.add(region); + snapshotFamilies.add(Bytes.toBytes(family)); + } + }); + + // Verify that there are store files in the specified families + if (nonEmptyTestFamilies != null) { + for (final byte[] familyName: nonEmptyTestFamilies) { + assertTrue(snapshotFamilies.contains(familyName)); + } + } + + // Verify that there are no store files in the specified families + if (emptyTestFamilies != null) { + for (final byte[] familyName: emptyTestFamilies) { + assertFalse(snapshotFamilies.contains(familyName)); + } + } + + // Avoid checking regions if the request is for an empty snapshot + if ((nonEmptyTestFamilies == null || nonEmptyTestFamilies.size() == 0) && + (emptyTestFamilies != null && emptyTestFamilies.size() > 0)) { + assertEquals(0, snapshotRegions.size()); + return; + } + + // check the region snapshot for all the regions + List regions = admin.getTableRegions(Bytes.toBytes(tableName)); + assertEquals(regions.size(), snapshotRegions.size()); + + // Verify Regions + for (HRegionInfo info : regions) { + String regionName = info.getEncodedName(); + assertTrue(snapshotRegions.contains(regionName)); + + Path regionDir = new Path(snapshotDir, regionName); + HRegionInfo snapshotRegionInfo = HRegion.loadDotRegionInfoFileContent(fs, regionDir); + assertEquals(info, snapshotRegionInfo); + } + } + + /** + * Helper method for testing async snapshot operations. Just waits for the given snapshot to + * complete on the server by repeatedly checking the master. + * @param master running the snapshot + * @param snapshot to check + * @param sleep amount to sleep between checks to see if the snapshot is done + * @throws IOException if the snapshot fails + */ + public static void waitForSnapshotToComplete(HMaster master, HSnapshotDescription snapshot, + long sleep) throws IOException { + boolean done = false; + while (!done) { + done = master.isSnapshotDone(snapshot); + try { + Thread.sleep(sleep); + } catch (InterruptedException e) { + throw new IOException(e); + } + } + } + + public static void cleanupSnapshot(HBaseAdmin admin, byte[] tableName) + throws IOException { + SnapshotTestingUtils.cleanupSnapshot(admin, Bytes.toString(tableName)); + } + + public static void cleanupSnapshot(HBaseAdmin admin, String snapshotName) + throws IOException { + // delete the taken snapshot + admin.deleteSnapshot(snapshotName); + assertNoSnapshots(admin); + } + + /** + * Expect the snapshot to throw an error when checking if the snapshot is complete + * @param master master to check + * @param snapshot the {@link HSnapshotDescription} request to pass to the master + * @param clazz expected exception from the master + */ + public static void expectSnapshotDoneException(HMaster master, HSnapshotDescription snapshot, + Class clazz) { + try { + boolean res = master.isSnapshotDone(snapshot); + Assert.fail("didn't fail to lookup a snapshot: res=" + res); + } catch (HBaseSnapshotException e) { + assertEquals("Threw wrong snapshot exception!", clazz, e.getClass()); + } catch (Throwable t) { + Assert.fail("Threw an unexpected exception:" + t); + } + } + + /** + * List all the HFiles in the given table + * + * @param fs: FileSystem where the table lives + * @param tableDir directory of the table + * @return array of the current HFiles in the table (could be a zero-length array) + * @throws IOException on unexecpted error reading the FS + */ + public static Path[] listHFiles(final FileSystem fs, final Path tableDir) + throws IOException { + final ArrayList hfiles = new ArrayList(); + FSVisitor.visitTableStoreFiles(fs, tableDir, new FSVisitor.StoreFileVisitor() { + public void storeFile(final String region, final String family, final String hfileName) + throws IOException { + hfiles.add(new Path(tableDir, new Path(region, new Path(family, hfileName)))); + } + }); + return hfiles.toArray(new Path[hfiles.size()]); + } + + /** + * Take a snapshot of the specified table and verify that the given family is + * not empty. Note that this will leave the table disabled + * in the case of an offline snapshot. + */ + public static void createSnapshotAndValidate(HBaseAdmin admin, + String tableName, String familyName, String snapshotNameString, + Path rootDir, FileSystem fs, boolean onlineSnapshot) + throws Exception { + ArrayList nonEmptyFamilyNames = new ArrayList(1); + nonEmptyFamilyNames.add(Bytes.toBytes(familyName)); + createSnapshotAndValidate(admin, tableName, nonEmptyFamilyNames, /* emptyFamilyNames= */ null, + snapshotNameString, rootDir, fs, onlineSnapshot); + } + + /** + * Take a snapshot of the specified table and verify the given families. + * Note that this will leave the table disabled in the case of an offline snapshot. + */ + public static void createSnapshotAndValidate(HBaseAdmin admin, + String tableName, List nonEmptyFamilyNames, List emptyFamilyNames, + String snapshotNameString, Path rootDir, FileSystem fs, boolean onlineSnapshot) + throws Exception { + if (!onlineSnapshot) { + try { + admin.disableTable(tableName); + } catch (TableNotEnabledException tne) { + LOG.info("In attempting to disable " + tableName + " it turns out that the this table is " + + "already disabled."); + } + } + admin.snapshot(snapshotNameString, tableName); + + List snapshots = SnapshotTestingUtils.assertExistsMatchingSnapshot(admin, + snapshotNameString, tableName); + if (snapshots == null || snapshots.size() != 1) { + Assert.fail("Incorrect number of snapshots for table " + tableName); + } + + SnapshotTestingUtils.confirmSnapshotValid(snapshots.get(0), tableName, nonEmptyFamilyNames, + emptyFamilyNames, rootDir, admin, fs, false, + new Path(rootDir, HConstants.HREGION_LOGDIR_NAME), null); + } + + /** + * Corrupt the specified snapshot by deleting some files. + * + * @param util {@link HBaseTestingUtility} + * @param snapshotName name of the snapshot to corrupt + * @return array of the corrupted HFiles + * @throws IOException on unexecpted error reading the FS + */ + public static ArrayList corruptSnapshot(final HBaseTestingUtility util, final String snapshotName) + throws IOException { + final MasterFileSystem mfs = util.getHBaseCluster().getMaster().getMasterFileSystem(); + final FileSystem fs = mfs.getFileSystem(); + + Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, + mfs.getRootDir()); + SnapshotDescription snapshotDesc = SnapshotDescriptionUtils.readSnapshotInfo(fs, snapshotDir); + final String table = snapshotDesc.getTable(); + + final ArrayList corruptedFiles = new ArrayList(); + SnapshotReferenceUtil.visitTableStoreFiles(fs, snapshotDir, new FSVisitor.StoreFileVisitor() { + public void storeFile (final String region, final String family, final String hfile) + throws IOException { + HFileLink link = HFileLink.create(util.getConfiguration(), table, region, family, hfile); + if (corruptedFiles.size() % 2 == 0) { + fs.delete(link.getAvailablePath(fs)); + corruptedFiles.add(hfile); + } + } + }); + + assertTrue(corruptedFiles.size() > 0); + return corruptedFiles; + } + + // ========================================================================== + // Table Helpers + // ========================================================================== + public static void waitForTableToBeOnline(final HBaseTestingUtility util, final byte[] tableName) + throws IOException, InterruptedException { + HRegionServer rs = util.getRSForFirstRegionInTable(tableName); + List onlineRegions = rs.getOnlineRegions(tableName); + for (HRegion region : onlineRegions) { + region.waitForFlushesAndCompactions(); + } + util.getHBaseAdmin().isTableAvailable(tableName); + } + + public static void createTable(final HBaseTestingUtility util, final byte[] tableName, + final byte[]... families) throws IOException, InterruptedException { + HTableDescriptor htd = new HTableDescriptor(tableName); + for (byte[] family: families) { + HColumnDescriptor hcd = new HColumnDescriptor(family); + htd.addFamily(hcd); + } + byte[][] splitKeys = new byte[KEYS.length-2][]; + for (int i = 0; i < splitKeys.length; ++i) { + splitKeys[i] = new byte[] { KEYS[i+1] }; + } + util.getHBaseAdmin().createTable(htd, splitKeys); + waitForTableToBeOnline(util, tableName); + assertEquals(KEYS.length-1, util.getHBaseAdmin().getTableRegions(tableName).size()); + } + + public static void loadData(final HBaseTestingUtility util, final byte[] tableName, int rows, + byte[]... families) throws IOException, InterruptedException { + loadData(util, new HTable(util.getConfiguration(), tableName), rows, families); + } + + public static void loadData(final HBaseTestingUtility util, final HTable table, int rows, + byte[]... families) throws IOException, InterruptedException { + table.setAutoFlush(false); + + // Ensure one row per region + assertTrue(rows >= KEYS.length); + for (byte k0: KEYS) { + byte[] k = new byte[] { k0 }; + byte[] value = Bytes.add(Bytes.toBytes(System.currentTimeMillis()), k); + byte[] key = Bytes.add(k, Bytes.toBytes(MD5Hash.getMD5AsHex(value))); + putData(table, families, key, value); + rows--; + } + + // Add other extra rows. more rows, more files + while (rows-- > 0) { + byte[] value = Bytes.add(Bytes.toBytes(System.currentTimeMillis()), Bytes.toBytes(rows)); + byte[] key = Bytes.toBytes(MD5Hash.getMD5AsHex(value)); + putData(table, families, key, value); + } + table.flushCommits(); + + waitForTableToBeOnline(util, table.getTableName()); + } + + private static void putData(final HTable table, final byte[][] families, + final byte[] key, final byte[] value) throws IOException { + byte[] q = Bytes.toBytes("q"); + Put put = new Put(key); + put.setDurability(Durability.SKIP_WAL); + for (byte[] family: families) { + put.add(family, q, value); + } + table.put(put); + } + + public static void deleteAllSnapshots(final HBaseAdmin admin) + throws IOException { + // Delete all the snapshots + for (SnapshotDescription snapshot: admin.listSnapshots()) { + admin.deleteSnapshot(snapshot.getName()); + } + SnapshotTestingUtils.assertNoSnapshots(admin); + } + + public static void deleteArchiveDirectory(final HBaseTestingUtility util) + throws IOException { + // Ensure the archiver to be empty + MasterFileSystem mfs = util.getMiniHBaseCluster().getMaster().getMasterFileSystem(); + Path archiveDir = new Path(mfs.getRootDir(), HConstants.HFILE_ARCHIVE_DIRECTORY); + mfs.getFileSystem().delete(archiveDir, true); + } + + public static void verifyRowCount(final HBaseTestingUtility util, final byte[] tableName, + long expectedRows) throws IOException { + HTable table = new HTable(util.getConfiguration(), tableName); + try { + assertEquals(expectedRows, util.countRows(table)); + } finally { + table.close(); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestCopyRecoveredEditsTask.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestCopyRecoveredEditsTask.java new file mode 100644 index 000000000000..b68c3b94ab1c --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestCopyRecoveredEditsTask.java @@ -0,0 +1,126 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.errorhandling.ForeignException; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +/** + * Test that we correctly copy the recovered edits from a directory + */ +@Category(SmallTests.class) +public class TestCopyRecoveredEditsTask { + + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @Test + public void testCopyFiles() throws Exception { + + SnapshotDescription snapshot = SnapshotDescription.newBuilder().setName("snapshot").build(); + ForeignExceptionDispatcher monitor = Mockito.mock(ForeignExceptionDispatcher.class); + FileSystem fs = UTIL.getTestFileSystem(); + Path root = UTIL.getDataTestDir(); + String regionName = "regionA"; + Path regionDir = new Path(root, regionName); + Path workingDir = SnapshotDescriptionUtils.getWorkingSnapshotDir(snapshot, root); + + try { + // doesn't really matter where the region's snapshot directory is, but this is pretty close + Path snapshotRegionDir = new Path(workingDir, regionName); + fs.mkdirs(snapshotRegionDir); + + // put some stuff in the recovered.edits directory + Path edits = HLog.getRegionDirRecoveredEditsDir(regionDir); + fs.mkdirs(edits); + // make a file with some data + Path file1 = new Path(edits, "0000000000000002352"); + FSDataOutputStream out = fs.create(file1); + byte[] data = new byte[] { 1, 2, 3, 4 }; + out.write(data); + out.close(); + // make an empty file + Path empty = new Path(edits, "empty"); + fs.createNewFile(empty); + + CopyRecoveredEditsTask task = new CopyRecoveredEditsTask(snapshot, monitor, fs, regionDir, + snapshotRegionDir); + CopyRecoveredEditsTask taskSpy = Mockito.spy(task); + taskSpy.call(); + + Path snapshotEdits = HLog.getRegionDirRecoveredEditsDir(snapshotRegionDir); + FileStatus[] snapshotEditFiles = FSUtils.listStatus(fs, snapshotEdits); + assertEquals("Got wrong number of files in the snapshot edits", 1, snapshotEditFiles.length); + FileStatus file = snapshotEditFiles[0]; + assertEquals("Didn't copy expected file", file1.getName(), file.getPath().getName()); + + Mockito.verify(monitor, Mockito.never()).receive(Mockito.any(ForeignException.class)); + Mockito.verify(taskSpy, Mockito.never()).snapshotFailure(Mockito.anyString(), + Mockito.any(Exception.class)); + } finally { + // cleanup the working directory + FSUtils.delete(fs, regionDir, true); + FSUtils.delete(fs, workingDir, true); + } + } + + /** + * Check that we don't get an exception if there is no recovered edits directory to copy + * @throws Exception on failure + */ + @Test + public void testNoEditsDir() throws Exception { + SnapshotDescription snapshot = SnapshotDescription.newBuilder().setName("snapshot").build(); + ForeignExceptionDispatcher monitor = Mockito.mock(ForeignExceptionDispatcher.class); + FileSystem fs = UTIL.getTestFileSystem(); + Path root = UTIL.getDataTestDir(); + String regionName = "regionA"; + Path regionDir = new Path(root, regionName); + Path workingDir = SnapshotDescriptionUtils.getWorkingSnapshotDir(snapshot, root); + try { + // doesn't really matter where the region's snapshot directory is, but this is pretty close + Path snapshotRegionDir = new Path(workingDir, regionName); + fs.mkdirs(snapshotRegionDir); + Path regionEdits = HLog.getRegionDirRecoveredEditsDir(regionDir); + assertFalse("Edits dir exists already - it shouldn't", fs.exists(regionEdits)); + + CopyRecoveredEditsTask task = new CopyRecoveredEditsTask(snapshot, monitor, fs, regionDir, + snapshotRegionDir); + task.call(); + } finally { + // cleanup the working directory + FSUtils.delete(fs, regionDir, true); + FSUtils.delete(fs, workingDir, true); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java new file mode 100644 index 000000000000..111833a44f57 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java @@ -0,0 +1,424 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.snapshot; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.HashSet; +import java.util.Set; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileUtil; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.snapshot.ExportSnapshot; +import org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil; +import org.apache.hadoop.mapreduce.Job; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test Export Snapshot Tool + */ +@Category(MediumTests.class) +public class TestExportSnapshot { + private final Log LOG = LogFactory.getLog(getClass()); + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + private final static byte[] FAMILY = Bytes.toBytes("cf"); + + private byte[] emptySnapshotName; + private byte[] snapshotName; + private byte[] tableName; + private HBaseAdmin admin; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.getConfiguration().setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + TEST_UTIL.getConfiguration().setInt("hbase.regionserver.msginterval", 100); + TEST_UTIL.getConfiguration().setInt("hbase.client.pause", 250); + TEST_UTIL.getConfiguration().setInt("hbase.client.retries.number", 6); + TEST_UTIL.getConfiguration().setBoolean("hbase.master.enabletable.roundrobin", true); + TEST_UTIL.getConfiguration().setInt("mapreduce.map.max.attempts", 10); + TEST_UTIL.getConfiguration().setInt("mapred.map.max.attempts", 10); + TEST_UTIL.startMiniCluster(3); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + /** + * Create a table and take a snapshot of the table used by the export test. + */ + @Before + public void setUp() throws Exception { + this.admin = TEST_UTIL.getHBaseAdmin(); + + long tid = System.currentTimeMillis(); + tableName = Bytes.toBytes("testtb-" + tid); + snapshotName = Bytes.toBytes("snaptb0-" + tid); + emptySnapshotName = Bytes.toBytes("emptySnaptb0-" + tid); + + // create Table + SnapshotTestingUtils.createTable(TEST_UTIL, tableName, FAMILY); + + // Take an empty snapshot + admin.snapshot(emptySnapshotName, tableName); + + // Add some rows + HTable table = new HTable(TEST_UTIL.getConfiguration(), tableName); + SnapshotTestingUtils.loadData(TEST_UTIL, tableName, 500, FAMILY); + + // take a snapshot + admin.snapshot(snapshotName, tableName); + } + + @After + public void tearDown() throws Exception { + TEST_UTIL.deleteTable(tableName); + SnapshotTestingUtils.deleteAllSnapshots(TEST_UTIL.getHBaseAdmin()); + SnapshotTestingUtils.deleteArchiveDirectory(TEST_UTIL); + admin.close(); + } + + + /** + * Verfy the result of getBalanceSplits() method. + * The result are groups of files, used as input list for the "export" mappers. + * All the groups should have similar amount of data. + * + * The input list is a pair of file path and length. + * The getBalanceSplits() function sort it by length, + * and assign to each group a file, going back and forth through the groups. + */ + @Test + public void testBalanceSplit() throws Exception { + // Create a list of files + List> files = new ArrayList>(); + for (long i = 0; i <= 20; i++) { + files.add(new Pair(new Path("file-" + i), i)); + } + + // Create 5 groups (total size 210) + // group 0: 20, 11, 10, 1 (total size: 42) + // group 1: 19, 12, 9, 2 (total size: 42) + // group 2: 18, 13, 8, 3 (total size: 42) + // group 3: 17, 12, 7, 4 (total size: 42) + // group 4: 16, 11, 6, 5 (total size: 42) + List> splits = ExportSnapshot.getBalancedSplits(files, 5); + assertEquals(5, splits.size()); + assertEquals(Arrays.asList(new Path("file-20"), new Path("file-11"), + new Path("file-10"), new Path("file-1"), new Path("file-0")), splits.get(0)); + assertEquals(Arrays.asList(new Path("file-19"), new Path("file-12"), + new Path("file-9"), new Path("file-2")), splits.get(1)); + assertEquals(Arrays.asList(new Path("file-18"), new Path("file-13"), + new Path("file-8"), new Path("file-3")), splits.get(2)); + assertEquals(Arrays.asList(new Path("file-17"), new Path("file-14"), + new Path("file-7"), new Path("file-4")), splits.get(3)); + assertEquals(Arrays.asList(new Path("file-16"), new Path("file-15"), + new Path("file-6"), new Path("file-5")), splits.get(4)); + } + + /** + * Verify if exported snapshot and copied files matches the original one. + */ + @Test + public void testExportFileSystemState() throws Exception { + testExportFileSystemState(tableName, snapshotName, snapshotName, 2); + } + + @Test + public void testExportFileSystemStateWithSkipTmp() throws Exception { + TEST_UTIL.getConfiguration().setBoolean(ExportSnapshot.CONF_SKIP_TMP, true); + testExportFileSystemState(tableName, snapshotName, snapshotName, 2); + } + + @Test + public void testEmptyExportFileSystemState() throws Exception { + testExportFileSystemState(tableName, emptySnapshotName, emptySnapshotName, 1); + } + + @Test + public void testConsecutiveExports() throws Exception { + Path copyDir = getLocalDestinationDir(); + testExportFileSystemState(tableName, snapshotName, snapshotName, 2, copyDir, false); + testExportFileSystemState(tableName, snapshotName, snapshotName, 2, copyDir, true); + removeExportDir(copyDir); + } + + @Test + public void testExportWithTargetName() throws Exception { + final byte[] targetName = Bytes.toBytes("testExportWithTargetName"); + testExportFileSystemState(tableName, snapshotName, targetName, 2); + } + + /** + * Mock a snapshot with files in the archive dir, + * two regions, and one reference file. + */ + @Test + public void testSnapshotWithRefsExportFileSystemState() throws Exception { + Configuration conf = TEST_UTIL.getConfiguration(); + + final byte[] tableWithRefsName = Bytes.toBytes("tableWithRefs"); + final String snapshotName = "tableWithRefs"; + final String TEST_FAMILY = Bytes.toString(FAMILY); + final String TEST_HFILE = "abc"; + + final SnapshotDescription sd = SnapshotDescription.newBuilder() + .setName(snapshotName).setTable(Bytes.toString(tableWithRefsName)).build(); + + FileSystem fs = TEST_UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getFileSystem(); + Path rootDir = TEST_UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getRootDir(); + Path archiveDir = new Path(rootDir, HConstants.HFILE_ARCHIVE_DIRECTORY); + + HTableDescriptor htd = new HTableDescriptor(tableWithRefsName); + htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); + + // First region, simple with one plain hfile. + HRegion r0 = HRegion.createHRegion(new HRegionInfo(htd.getName()), archiveDir, + conf, htd, null, true, true); + Path storeFile = new Path(new Path(r0.getRegionDir(), TEST_FAMILY), TEST_HFILE); + FSDataOutputStream out = fs.create(storeFile); + out.write(Bytes.toBytes("Test Data")); + out.close(); + r0.close(); + + // Second region, used to test the split case. + // This region contains a reference to the hfile in the first region. + HRegion r1 = HRegion.createHRegion(new HRegionInfo(htd.getName()), archiveDir, + conf, htd, null, true, true); + out = fs.create(new Path(new Path(r1.getRegionDir(), TEST_FAMILY), + storeFile.getName() + '.' + r0.getRegionInfo().getEncodedName())); + out.write(Bytes.toBytes("Test Data")); + out.close(); + r1.close(); + + Path tableDir = HTableDescriptor.getTableDir(archiveDir, tableWithRefsName); + Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir); + FileUtil.copy(fs, tableDir, fs, snapshotDir, false, conf); + SnapshotDescriptionUtils.writeSnapshotInfo(sd, snapshotDir, fs); + + byte[] name = Bytes.toBytes(snapshotName); + testExportFileSystemState(tableWithRefsName, name, name, 2); + } + + private void testExportFileSystemState(final byte[] tableName, final byte[] snapshotName, + final byte[] targetName, int filesExpected) throws Exception { + Path copyDir = getHdfsDestinationDir(); + testExportFileSystemState(tableName, snapshotName, targetName, filesExpected, copyDir, false); + removeExportDir(copyDir); + } + + /** + * Test ExportSnapshot + */ + private void testExportFileSystemState(final byte[] tableName, final byte[] snapshotName, + final byte[] targetName, int filesExpected, Path copyDir, boolean overwrite) + throws Exception { + URI hdfsUri = FileSystem.get(TEST_UTIL.getConfiguration()).getUri(); + FileSystem fs = FileSystem.get(copyDir.toUri(), new Configuration()); + copyDir = copyDir.makeQualified(fs); + + List opts = new ArrayList(); + opts.add("-snapshot"); + opts.add(Bytes.toString(snapshotName)); + opts.add("-copy-to"); + opts.add(copyDir.toString()); + if (targetName != snapshotName) { + opts.add("-target"); + opts.add(Bytes.toString(targetName)); + } + if (overwrite) opts.add("-overwrite"); + + // Export Snapshot + int res = ExportSnapshot.innerMain(TEST_UTIL.getConfiguration(), + opts.toArray(new String[opts.size()])); + assertEquals(0, res); + + // Verify File-System state + FileStatus[] rootFiles = fs.listStatus(copyDir); + assertEquals(filesExpected, rootFiles.length); + for (FileStatus fileStatus: rootFiles) { + String name = fileStatus.getPath().getName(); + assertTrue(fileStatus.isDir()); + assertTrue(name.equals(HConstants.SNAPSHOT_DIR_NAME) || name.equals(".archive")); + } + + // compare the snapshot metadata and verify the hfiles + final FileSystem hdfs = FileSystem.get(hdfsUri, TEST_UTIL.getConfiguration()); + final Path snapshotDir = new Path(HConstants.SNAPSHOT_DIR_NAME, Bytes.toString(snapshotName)); + final Path targetDir = new Path(HConstants.SNAPSHOT_DIR_NAME, Bytes.toString(targetName)); + verifySnapshot(hdfs, new Path(TEST_UTIL.getDefaultRootDirPath(), snapshotDir), + fs, new Path(copyDir, targetDir)); + verifyArchive(fs, copyDir, tableName, Bytes.toString(targetName)); + FSUtils.logFileSystemState(hdfs, snapshotDir, LOG); + } + + /** + * Check that ExportSnapshot will return a failure if something fails. + */ + @Test + public void testExportFailure() throws Exception { + assertEquals(1, runExportAndInjectFailures(snapshotName, false)); + } + + /* + * Execute the ExportSnapshot job injecting failures + */ + private int runExportAndInjectFailures(final byte[] snapshotName, boolean retry) + throws Exception { + Path copyDir = TEST_UTIL.getDataTestDir("export-" + System.currentTimeMillis()); + URI hdfsUri = FileSystem.get(TEST_UTIL.getConfiguration()).getUri(); + FileSystem fs = FileSystem.get(copyDir.toUri(), new Configuration()); + copyDir = copyDir.makeQualified(fs); + + Configuration conf = new Configuration(TEST_UTIL.getConfiguration()); + conf.setBoolean(ExportSnapshot.CONF_TEST_FAILURE, true); + conf.setBoolean(ExportSnapshot.CONF_TEST_RETRY, retry); + + // Export Snapshot + int res = ExportSnapshot.innerMain(conf, new String[] { + "-snapshot", Bytes.toString(snapshotName), + "-copy-to", copyDir.toString() + }); + return res; + } + + /* + * verify if the snapshot folder on file-system 1 match the one on file-system 2 + */ + private void verifySnapshot(final FileSystem fs1, final Path root1, + final FileSystem fs2, final Path root2) throws IOException { + Set s = new HashSet(); + assertEquals(listFiles(fs1, root1, root1), listFiles(fs2, root2, root2)); + } + + /* + * Verify if the files exists + */ + private void verifyArchive(final FileSystem fs, final Path rootDir, + final byte[] tableName, final String snapshotName) throws IOException { + final Path exportedSnapshot = new Path(rootDir, + new Path(HConstants.SNAPSHOT_DIR_NAME, snapshotName)); + final Path exportedArchive = new Path(rootDir, HConstants.HFILE_ARCHIVE_DIRECTORY); + LOG.debug(listFiles(fs, exportedArchive, exportedArchive)); + SnapshotReferenceUtil.visitReferencedFiles(fs, exportedSnapshot, + new SnapshotReferenceUtil.FileVisitor() { + public void storeFile (final String region, final String family, final String hfile) + throws IOException { + verifyNonEmptyFile(new Path(exportedArchive, + new Path(Bytes.toString(tableName), new Path(region, new Path(family, hfile))))); + } + + public void recoveredEdits (final String region, final String logfile) + throws IOException { + verifyNonEmptyFile(new Path(exportedSnapshot, + new Path(Bytes.toString(tableName), new Path(region, logfile)))); + } + + public void logFile (final String server, final String logfile) + throws IOException { + verifyNonEmptyFile(new Path(exportedSnapshot, new Path(server, logfile))); + } + + private void verifyNonEmptyFile(final Path path) throws IOException { + assertTrue(path + " should exist", fs.exists(path)); + assertTrue(path + " should not be empty", fs.getFileStatus(path).getLen() > 0); + } + }); + + // Verify Snapshot description + SnapshotDescription desc = SnapshotDescriptionUtils.readSnapshotInfo(fs, exportedSnapshot); + assertTrue(desc.getName().equals(snapshotName)); + assertTrue(desc.getTable().equals(Bytes.toString(tableName))); + } + + private Set listFiles(final FileSystem fs, final Path root, final Path dir) + throws IOException { + Set files = new HashSet(); + int rootPrefix = root.toString().length(); + FileStatus[] list = FSUtils.listStatus(fs, dir); + if (list != null) { + for (FileStatus fstat: list) { + LOG.debug(fstat.getPath()); + if (fstat.isDir()) { + files.addAll(listFiles(fs, root, fstat.getPath())); + } else { + files.add(fstat.getPath().toString().substring(rootPrefix)); + } + } + } + return files; + } + + private Path getHdfsDestinationDir() { + Path rootDir = TEST_UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getRootDir(); + Path path = new Path(new Path(rootDir, "export-test"), "export-" + System.currentTimeMillis()); + LOG.info("HDFS export destination path: " + path); + return path; + } + + private Path getLocalDestinationDir() { + Path path = TEST_UTIL.getDataTestDir("local-export-" + System.currentTimeMillis()); + LOG.info("Local export destination path: " + path); + return path; + } + + private void removeExportDir(final Path path) throws IOException { + FileSystem fs = FileSystem.get(path.toUri(), new Configuration()); + FSUtils.logFileSystemState(fs, path, LOG); + fs.delete(path, true); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java new file mode 100644 index 000000000000..c7a265f18a54 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java @@ -0,0 +1,474 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.IOException; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.CountDownLatch; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.TableNotFoundException; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSTableDescriptors; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.JVMClusterUtil.RegionServerThread; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test creating/using/deleting snapshots from the client + *

+ * This is an end-to-end test for the snapshot utility + * + * TODO This is essentially a clone of TestSnapshotFromClient. This is worth refactoring this + * because there will be a few more flavors of snapshots that need to run these tests. + */ +@Category(LargeTests.class) +public class TestFlushSnapshotFromClient { + private static final Log LOG = LogFactory.getLog(TestFlushSnapshotFromClient.class); + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static final int NUM_RS = 2; + private static final String STRING_TABLE_NAME = "test"; + private static final byte[] TEST_FAM = Bytes.toBytes("fam"); + private static final byte[] TABLE_NAME = Bytes.toBytes(STRING_TABLE_NAME); + + /** + * Setup the config for the cluster + * @throws Exception on failure + */ + @BeforeClass + public static void setupCluster() throws Exception { + setupConf(UTIL.getConfiguration()); + UTIL.startMiniCluster(NUM_RS); + } + + private static void setupConf(Configuration conf) { + // disable the ui + conf.setInt("hbase.regionsever.info.port", -1); + // change the flush size to a small amount, regulating number of store files + conf.setInt("hbase.hregion.memstore.flush.size", 25000); + // so make sure we get a compaction when doing a load, but keep around some + // files in the store + conf.setInt("hbase.hstore.compaction.min", 10); + conf.setInt("hbase.hstore.compactionThreshold", 10); + // block writes if we get to 12 store files + conf.setInt("hbase.hstore.blockingStoreFiles", 12); + // drop the number of attempts for the hbase admin + conf.setInt("hbase.client.retries.number", 1); + // Enable snapshot + conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + // prevent aggressive region split + conf.set(HConstants.HBASE_REGION_SPLIT_POLICY_KEY, + ConstantSizeRegionSplitPolicy.class.getName()); + } + + @Before + public void setup() throws Exception { + UTIL.createTable(TABLE_NAME, TEST_FAM); + } + + @After + public void tearDown() throws Exception { + UTIL.deleteTable(TABLE_NAME); + SnapshotTestingUtils.deleteAllSnapshots(UTIL.getHBaseAdmin()); + SnapshotTestingUtils.deleteArchiveDirectory(UTIL); + } + + @AfterClass + public static void cleanupTest() throws Exception { + try { + UTIL.shutdownMiniCluster(); + } catch (Exception e) { + LOG.warn("failure shutting down cluster", e); + } + } + + /** + * Test snapshotting a table that is online without flushing + * @throws Exception + */ + @Test + public void testSkipFlushTableSnapshot() throws Exception { + HBaseAdmin admin = UTIL.getHBaseAdmin(); + // make sure we don't fail on listing snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + + // put some stuff in the table + HTable table = new HTable(UTIL.getConfiguration(), TABLE_NAME); + UTIL.loadTable(table, TEST_FAM); + + // get the name of all the regionservers hosting the snapshotted table + Set snapshotServers = new HashSet(); + List servers = UTIL.getMiniHBaseCluster().getLiveRegionServerThreads(); + for (RegionServerThread server : servers) { + if (server.getRegionServer().getOnlineRegions(TABLE_NAME).size() > 0) { + snapshotServers.add(server.getRegionServer().getServerName().toString()); + } + } + + LOG.debug("FS state before snapshot:"); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + + // take a snapshot of the enabled table + String snapshotString = "skipFlushTableSnapshot"; + byte[] snapshot = Bytes.toBytes(snapshotString); + admin.snapshot(snapshotString, STRING_TABLE_NAME, SnapshotDescription.Type.SKIPFLUSH); + LOG.debug("Snapshot completed."); + + // make sure we have the snapshot + List snapshots = SnapshotTestingUtils.assertOneSnapshotThatMatches(admin, + snapshot, TABLE_NAME); + + // make sure its a valid snapshot + FileSystem fs = UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getFileSystem(); + Path rootDir = UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getRootDir(); + LOG.debug("FS state after snapshot:"); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + + SnapshotTestingUtils.confirmSnapshotValid(snapshots.get(0), TABLE_NAME, TEST_FAM, rootDir, + admin, fs, false, new Path(rootDir, HConstants.HREGION_LOGDIR_NAME), snapshotServers); + + admin.deleteSnapshot(snapshot); + snapshots = admin.listSnapshots(); + SnapshotTestingUtils.assertNoSnapshots(admin); + } + /** + * Test simple flush snapshotting a table that is online + * @throws Exception + */ + @Test + public void testFlushTableSnapshot() throws Exception { + HBaseAdmin admin = UTIL.getHBaseAdmin(); + // make sure we don't fail on listing snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + + // put some stuff in the table + HTable table = new HTable(UTIL.getConfiguration(), TABLE_NAME); + UTIL.loadTable(table, TEST_FAM); + + // get the name of all the regionservers hosting the snapshotted table + Set snapshotServers = new HashSet(); + List servers = UTIL.getMiniHBaseCluster().getLiveRegionServerThreads(); + for (RegionServerThread server : servers) { + if (server.getRegionServer().getOnlineRegions(TABLE_NAME).size() > 0) { + snapshotServers.add(server.getRegionServer().getServerName().toString()); + } + } + + LOG.debug("FS state before snapshot:"); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + + // take a snapshot of the enabled table + String snapshotString = "offlineTableSnapshot"; + byte[] snapshot = Bytes.toBytes(snapshotString); + admin.snapshot(snapshotString, STRING_TABLE_NAME, SnapshotDescription.Type.FLUSH); + LOG.debug("Snapshot completed."); + + // make sure we have the snapshot + List snapshots = SnapshotTestingUtils.assertOneSnapshotThatMatches(admin, + snapshot, TABLE_NAME); + + // make sure its a valid snapshot + FileSystem fs = UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getFileSystem(); + Path rootDir = UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getRootDir(); + LOG.debug("FS state after snapshot:"); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + + SnapshotTestingUtils.confirmSnapshotValid(snapshots.get(0), TABLE_NAME, TEST_FAM, rootDir, + admin, fs, false, new Path(rootDir, HConstants.HREGION_LOGDIR_NAME), snapshotServers); + + admin.deleteSnapshot(snapshot); + snapshots = admin.listSnapshots(); + SnapshotTestingUtils.assertNoSnapshots(admin); + } + + @Test + public void testSnapshotFailsOnNonExistantTable() throws Exception { + HBaseAdmin admin = UTIL.getHBaseAdmin(); + // make sure we don't fail on listing snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + String tableName = "_not_a_table"; + + // make sure the table doesn't exist + boolean fail = false; + do { + try { + admin.getTableDescriptor(Bytes.toBytes(tableName)); + fail = true; + LOG.error("Table:" + tableName + " already exists, checking a new name"); + tableName = tableName+"!"; + } catch (TableNotFoundException e) { + fail = false; + } + } while (fail); + + // snapshot the non-existant table + try { + admin.snapshot("fail", tableName, SnapshotDescription.Type.FLUSH); + fail("Snapshot succeeded even though there is not table."); + } catch (SnapshotCreationException e) { + LOG.info("Correctly failed to snapshot a non-existant table:" + e.getMessage()); + } + } + + @Test(timeout = 60000) + public void testAsyncFlushSnapshot() throws Exception { + HBaseAdmin admin = UTIL.getHBaseAdmin(); + SnapshotDescription snapshot = SnapshotDescription.newBuilder().setName("asyncSnapshot") + .setTable(STRING_TABLE_NAME).setType(SnapshotDescription.Type.FLUSH).build(); + + // take the snapshot async + admin.takeSnapshotAsync(snapshot); + + // constantly loop, looking for the snapshot to complete + HMaster master = UTIL.getMiniHBaseCluster().getMaster(); + SnapshotTestingUtils.waitForSnapshotToComplete(master, new HSnapshotDescription(snapshot), 200); + LOG.info(" === Async Snapshot Completed ==="); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + // make sure we get the snapshot + SnapshotTestingUtils.assertOneSnapshotThatMatches(admin, snapshot); + + // test that we can delete the snapshot + admin.deleteSnapshot(snapshot.getName()); + LOG.info(" === Async Snapshot Deleted ==="); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + // make sure we don't have any snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + LOG.info(" === Async Snapshot Test Completed ==="); + + } + + /** + * Basic end-to-end test of simple-flush-based snapshots + */ + @Test + public void testFlushCreateListDestroy() throws Exception { + LOG.debug("------- Starting Snapshot test -------------"); + HBaseAdmin admin = UTIL.getHBaseAdmin(); + // make sure we don't fail on listing snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + // load the table so we have some data + UTIL.loadTable(new HTable(UTIL.getConfiguration(), TABLE_NAME), TEST_FAM); + // and wait until everything stabilizes + waitForTableToBeOnline(TABLE_NAME); + + String snapshotName = "flushSnapshotCreateListDestroy"; + // test creating the snapshot + admin.snapshot(snapshotName, STRING_TABLE_NAME, SnapshotDescription.Type.FLUSH); + logFSTree(new Path(UTIL.getConfiguration().get(HConstants.HBASE_DIR))); + + // make sure we only have 1 matching snapshot + List snapshots = SnapshotTestingUtils.assertOneSnapshotThatMatches(admin, + snapshotName, STRING_TABLE_NAME); + + // check the directory structure + FileSystem fs = UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getFileSystem(); + Path rootDir = UTIL.getHBaseCluster().getMaster().getMasterFileSystem().getRootDir(); + Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshots.get(0), rootDir); + assertTrue(fs.exists(snapshotDir)); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), snapshotDir, LOG); + Path snapshotinfo = new Path(snapshotDir, SnapshotDescriptionUtils.SNAPSHOTINFO_FILE); + assertTrue(fs.exists(snapshotinfo)); + + // check the table info + HTableDescriptor desc = FSTableDescriptors.getTableDescriptor(fs, rootDir, TABLE_NAME); + HTableDescriptor snapshotDesc = FSTableDescriptors.getTableDescriptor(fs, + SnapshotDescriptionUtils.getSnapshotsDir(rootDir), Bytes.toBytes(snapshotName)); + assertEquals(desc, snapshotDesc); + + // check the region snapshot for all the regions + List regions = admin.getTableRegions(TABLE_NAME); + for (HRegionInfo info : regions) { + String regionName = info.getEncodedName(); + Path regionDir = new Path(snapshotDir, regionName); + HRegionInfo snapshotRegionInfo = HRegion.loadDotRegionInfoFileContent(fs, regionDir); + assertEquals(info, snapshotRegionInfo); + // check to make sure we have the family + Path familyDir = new Path(regionDir, Bytes.toString(TEST_FAM)); + assertTrue(fs.exists(familyDir)); + // make sure we have some file references + assertTrue(fs.listStatus(familyDir).length > 0); + } + + // test that we can delete the snapshot + admin.deleteSnapshot(snapshotName); + FSUtils.logFileSystemState(UTIL.getTestFileSystem(), + FSUtils.getRootDir(UTIL.getConfiguration()), LOG); + + // make sure we don't have any snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + LOG.debug("------- Flush-Snapshot Create List Destroy-------------"); + } + + /** + * Demonstrate that we reject snapshot requests if there is a snapshot already running on the + * same table currently running and that concurrent snapshots on different tables can both + * succeed concurretly. + */ + @Test(timeout=60000) + public void testConcurrentSnapshottingAttempts() throws IOException, InterruptedException { + final String STRING_TABLE2_NAME = STRING_TABLE_NAME + "2"; + final byte[] TABLE2_NAME = Bytes.toBytes(STRING_TABLE2_NAME); + + int ssNum = 20; + HBaseAdmin admin = UTIL.getHBaseAdmin(); + // make sure we don't fail on listing snapshots + SnapshotTestingUtils.assertNoSnapshots(admin); + // create second testing table + UTIL.createTable(TABLE2_NAME, TEST_FAM); + // load the table so we have some data + UTIL.loadTable(new HTable(UTIL.getConfiguration(), TABLE_NAME), TEST_FAM); + UTIL.loadTable(new HTable(UTIL.getConfiguration(), TABLE2_NAME), TEST_FAM); + // and wait until everything stabilizes + waitForTableToBeOnline(TABLE_NAME); + waitForTableToBeOnline(TABLE2_NAME); + + final CountDownLatch toBeSubmitted = new CountDownLatch(ssNum); + // We'll have one of these per thread + class SSRunnable implements Runnable { + SnapshotDescription ss; + SSRunnable(SnapshotDescription ss) { + this.ss = ss; + } + + @Override + public void run() { + try { + HBaseAdmin admin = UTIL.getHBaseAdmin(); + LOG.info("Submitting snapshot request: " + SnapshotDescriptionUtils.toString(ss)); + admin.takeSnapshotAsync(ss); + } catch (Exception e) { + LOG.info("Exception during snapshot request: " + SnapshotDescriptionUtils.toString(ss) + + ". This is ok, we expect some", e); + } + LOG.info("Submitted snapshot request: " + SnapshotDescriptionUtils.toString(ss)); + toBeSubmitted.countDown(); + } + }; + + // build descriptions + SnapshotDescription[] descs = new SnapshotDescription[ssNum]; + for (int i = 0; i < ssNum; i++) { + SnapshotDescription.Builder builder = SnapshotDescription.newBuilder(); + builder.setTable((i % 2) == 0 ? STRING_TABLE_NAME : STRING_TABLE2_NAME); + builder.setName("ss"+i); + builder.setType(SnapshotDescription.Type.FLUSH); + descs[i] = builder.build(); + } + + // kick each off its own thread + for (int i=0 ; i < ssNum; i++) { + new Thread(new SSRunnable(descs[i])).start(); + } + + // wait until all have been submitted + toBeSubmitted.await(); + + // loop until all are done. + while (true) { + int doneCount = 0; + for (SnapshotDescription ss : descs) { + try { + if (admin.isSnapshotFinished(ss)) { + doneCount++; + } + } catch (Exception e) { + LOG.warn("Got an exception when checking for snapshot " + ss.getName(), e); + doneCount++; + } + } + if (doneCount == descs.length) { + break; + } + Thread.sleep(100); + } + + // dump for debugging + logFSTree(new Path(UTIL.getConfiguration().get(HConstants.HBASE_DIR))); + + List taken = admin.listSnapshots(); + int takenSize = taken.size(); + LOG.info("Taken " + takenSize + " snapshots: " + taken); + assertTrue("We expect at least 1 request to be rejected because of we concurrently" + + " issued many requests", takenSize < ssNum && takenSize > 0); + + // Verify that there's at least one snapshot per table + int t1SnapshotsCount = 0; + int t2SnapshotsCount = 0; + for (SnapshotDescription ss : taken) { + if (ss.getTable().equals(STRING_TABLE_NAME)) { + t1SnapshotsCount++; + } else if (ss.getTable().equals(STRING_TABLE2_NAME)) { + t2SnapshotsCount++; + } + } + assertTrue("We expect at least 1 snapshot of table1 ", t1SnapshotsCount > 0); + assertTrue("We expect at least 1 snapshot of table2 ", t2SnapshotsCount > 0); + + // delete snapshots so subsequent tests are clean. + for (SnapshotDescription ss : taken) { + admin.deleteSnapshot(ss.getName()); + } + UTIL.deleteTable(TABLE2_NAME); + } + + private void logFSTree(Path root) throws IOException { + FSUtils.logFileSystemState(UTIL.getDFSCluster().getFileSystem(), root, LOG); + } + + private void waitForTableToBeOnline(final byte[] tableName) throws IOException { + HRegionServer rs = UTIL.getRSForFirstRegionInTable(tableName); + List onlineRegions = rs.getOnlineRegions(tableName); + for (HRegion region : onlineRegions) { + region.waitForFlushesAndCompactions(); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestReferenceRegionHFilesTask.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestReferenceRegionHFilesTask.java new file mode 100644 index 000000000000..73c2aba863b3 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestReferenceRegionHFilesTask.java @@ -0,0 +1,92 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.snapshot.ReferenceRegionHFilesTask; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +@Category(SmallTests.class) +public class TestReferenceRegionHFilesTask { + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @Test + public void testRun() throws IOException { + FileSystem fs = UTIL.getTestFileSystem(); + // setup the region internals + Path testdir = UTIL.getDataTestDir(); + Path regionDir = new Path(testdir, "region"); + Path family1 = new Path(regionDir, "fam1"); + // make an empty family + Path family2 = new Path(regionDir, "fam2"); + fs.mkdirs(family2); + + // add some files to family 1 + Path file1 = new Path(family1, "05f99689ae254693836613d1884c6b63"); + fs.createNewFile(file1); + Path file2 = new Path(family1, "7ac9898bf41d445aa0003e3d699d5d26"); + fs.createNewFile(file2); + + // create the snapshot directory + Path snapshotRegionDir = new Path(testdir, HConstants.SNAPSHOT_DIR_NAME); + fs.mkdirs(snapshotRegionDir); + + SnapshotDescription snapshot = SnapshotDescription.newBuilder().setName("name") + .setTable("table").build(); + ForeignExceptionDispatcher monitor = Mockito.mock(ForeignExceptionDispatcher.class); + ReferenceRegionHFilesTask task = new ReferenceRegionHFilesTask(snapshot, monitor, regionDir, + fs, snapshotRegionDir); + ReferenceRegionHFilesTask taskSpy = Mockito.spy(task); + task.call(); + + // make sure we never get an error + Mockito.verify(taskSpy, Mockito.never()).snapshotFailure(Mockito.anyString(), + Mockito.any(Exception.class)); + + // verify that all the hfiles get referenced + List hfiles = new ArrayList(2); + FileStatus[] regions = FSUtils.listStatus(fs, snapshotRegionDir); + for (FileStatus region : regions) { + FileStatus[] fams = FSUtils.listStatus(fs, region.getPath()); + for (FileStatus fam : fams) { + FileStatus[] files = FSUtils.listStatus(fs, fam.getPath()); + for (FileStatus file : files) { + hfiles.add(file.getPath().getName()); + } + } + } + assertTrue("Didn't reference :" + file1, hfiles.contains(file1.getName())); + assertTrue("Didn't reference :" + file1, hfiles.contains(file2.getName())); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java new file mode 100644 index 000000000000..7eadad9aee92 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java @@ -0,0 +1,200 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.master.MasterFileSystem; +import org.apache.hadoop.hbase.master.snapshot.SnapshotManager; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test clone/restore snapshots from the client + * + * TODO This is essentially a clone of TestRestoreSnapshotFromClient. This is worth refactoring + * this because there will be a few more flavors of snapshots that need to run these tests. + */ +@Category(LargeTests.class) +public class TestRestoreFlushSnapshotFromClient { + final Log LOG = LogFactory.getLog(getClass()); + + private final static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + private final byte[] FAMILY = Bytes.toBytes("cf"); + + private byte[] snapshotName0; + private byte[] snapshotName1; + private byte[] snapshotName2; + private int snapshot0Rows; + private int snapshot1Rows; + private byte[] tableName; + private HBaseAdmin admin; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + UTIL.getConfiguration().setBoolean("hbase.online.schema.update.enable", true); + UTIL.getConfiguration().setInt("hbase.regionserver.msginterval", 100); + UTIL.getConfiguration().setInt("hbase.client.pause", 250); + UTIL.getConfiguration().setInt("hbase.client.retries.number", 6); + UTIL.getConfiguration().setBoolean( + "hbase.master.enabletable.roundrobin", true); + + // Enable snapshot + UTIL.getConfiguration().setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true); + + UTIL.startMiniCluster(3); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + UTIL.shutdownMiniCluster(); + } + + /** + * Initialize the tests with a table filled with some data + * and two snapshots (snapshotName0, snapshotName1) of different states. + * The tableName, snapshotNames and the number of rows in the snapshot are initialized. + */ + @Before + public void setup() throws Exception { + this.admin = UTIL.getHBaseAdmin(); + + long tid = System.currentTimeMillis(); + tableName = Bytes.toBytes("testtb-" + tid); + snapshotName0 = Bytes.toBytes("snaptb0-" + tid); + snapshotName1 = Bytes.toBytes("snaptb1-" + tid); + snapshotName2 = Bytes.toBytes("snaptb2-" + tid); + + // create Table and disable it + SnapshotTestingUtils.createTable(UTIL, tableName, FAMILY); + HTable table = new HTable(UTIL.getConfiguration(), tableName); + SnapshotTestingUtils.loadData(UTIL, table, 500, FAMILY); + snapshot0Rows = UTIL.countRows(table); + LOG.info("=== before snapshot with 500 rows"); + logFSTree(); + + // take a snapshot + admin.snapshot(Bytes.toString(snapshotName0), Bytes.toString(tableName), + SnapshotDescription.Type.FLUSH); + + LOG.info("=== after snapshot with 500 rows"); + logFSTree(); + + // insert more data + SnapshotTestingUtils.loadData(UTIL, table, 500, FAMILY); + snapshot1Rows = UTIL.countRows(table); + LOG.info("=== before snapshot with 1000 rows"); + logFSTree(); + + // take a snapshot of the updated table + admin.snapshot(Bytes.toString(snapshotName1), Bytes.toString(tableName), + SnapshotDescription.Type.FLUSH); + LOG.info("=== after snapshot with 1000 rows"); + logFSTree(); + table.close(); + } + + @After + public void tearDown() throws Exception { + SnapshotTestingUtils.deleteAllSnapshots(UTIL.getHBaseAdmin()); + SnapshotTestingUtils.deleteArchiveDirectory(UTIL); + } + + @Test + public void testTakeFlushSnapshot() throws IOException { + // taking happens in setup. + } + + @Test + public void testRestoreSnapshot() throws IOException { + SnapshotTestingUtils.verifyRowCount(UTIL, tableName, snapshot1Rows); + + // Restore from snapshot-0 + admin.disableTable(tableName); + admin.restoreSnapshot(snapshotName0); + logFSTree(); + admin.enableTable(tableName); + LOG.info("=== after restore with 500 row snapshot"); + logFSTree(); + SnapshotTestingUtils.verifyRowCount(UTIL, tableName, snapshot0Rows); + + // Restore from snapshot-1 + admin.disableTable(tableName); + admin.restoreSnapshot(snapshotName1); + admin.enableTable(tableName); + SnapshotTestingUtils.verifyRowCount(UTIL, tableName, snapshot1Rows); + } + + @Test(expected=SnapshotDoesNotExistException.class) + public void testCloneNonExistentSnapshot() throws IOException, InterruptedException { + String snapshotName = "random-snapshot-" + System.currentTimeMillis(); + String tableName = "random-table-" + System.currentTimeMillis(); + admin.cloneSnapshot(snapshotName, tableName); + } + + @Test + public void testCloneSnapshot() throws IOException, InterruptedException { + byte[] clonedTableName = Bytes.toBytes("clonedtb-" + System.currentTimeMillis()); + testCloneSnapshot(clonedTableName, snapshotName0, snapshot0Rows); + testCloneSnapshot(clonedTableName, snapshotName1, snapshot1Rows); + } + + private void testCloneSnapshot(final byte[] tableName, final byte[] snapshotName, + int snapshotRows) throws IOException, InterruptedException { + // create a new table from snapshot + admin.cloneSnapshot(snapshotName, tableName); + SnapshotTestingUtils.verifyRowCount(UTIL, tableName, snapshotRows); + + UTIL.deleteTable(tableName); + } + + @Test + public void testRestoreSnapshotOfCloned() throws IOException, InterruptedException { + byte[] clonedTableName = Bytes.toBytes("clonedtb-" + System.currentTimeMillis()); + admin.cloneSnapshot(snapshotName0, clonedTableName); + SnapshotTestingUtils.verifyRowCount(UTIL, clonedTableName, snapshot0Rows); + admin.snapshot(Bytes.toString(snapshotName2), Bytes.toString(clonedTableName), SnapshotDescription.Type.FLUSH); + UTIL.deleteTable(clonedTableName); + + admin.cloneSnapshot(snapshotName2, clonedTableName); + SnapshotTestingUtils.verifyRowCount(UTIL, clonedTableName, snapshot0Rows); + UTIL.deleteTable(clonedTableName); + } + + // ========================================================================== + // Helpers + // ========================================================================== + private void logFSTree() throws IOException { + MasterFileSystem mfs = UTIL.getMiniHBaseCluster().getMaster().getMasterFileSystem(); + FSUtils.logFileSystemState(mfs.getFileSystem(), mfs.getRootDir(), LOG); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java new file mode 100644 index 000000000000..b995a56163fc --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java @@ -0,0 +1,202 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.IOException; +import java.util.Arrays; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.FileUtil; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.client.HConnection; +import org.apache.hadoop.hbase.client.HConnectionTestingUtility; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.apache.hadoop.hbase.io.HFileLink; +import org.apache.hadoop.hbase.monitoring.MonitoredTask; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.StoreFile; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSTableDescriptors; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.MD5Hash; +import org.junit.*; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +/** + * Test the restore/clone operation from a file-system point of view. + */ +@Category(SmallTests.class) +public class TestRestoreSnapshotHelper { + final Log LOG = LogFactory.getLog(getClass()); + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private final static String TEST_FAMILY = "cf"; + private final static String TEST_HFILE = "abc"; + + private Configuration conf; + private Path archiveDir; + private FileSystem fs; + private Path rootDir; + + @Before + public void setup() throws Exception { + rootDir = TEST_UTIL.getDataTestDir("testRestore"); + archiveDir = new Path(rootDir, HConstants.HFILE_ARCHIVE_DIRECTORY); + fs = TEST_UTIL.getTestFileSystem(); + conf = TEST_UTIL.getConfiguration(); + FSUtils.setRootDir(conf, rootDir); + } + + @After + public void tearDown() throws Exception { + fs.delete(TEST_UTIL.getDataTestDir(), true); + } + + @Test + public void testRestore() throws IOException { + HTableDescriptor htd = createTableDescriptor("testtb"); + + Path snapshotDir = new Path(rootDir, "snapshot"); + createSnapshot(rootDir, snapshotDir, htd); + + // Test clone a snapshot + HTableDescriptor htdClone = createTableDescriptor("testtb-clone"); + testRestore(snapshotDir, htd.getNameAsString(), htdClone); + verifyRestore(rootDir, htd, htdClone); + + // Test clone a clone ("link to link") + Path cloneDir = HTableDescriptor.getTableDir(rootDir, htdClone.getName()); + HTableDescriptor htdClone2 = createTableDescriptor("testtb-clone2"); + testRestore(cloneDir, htdClone.getNameAsString(), htdClone2); + verifyRestore(rootDir, htd, htdClone2); + } + + private void verifyRestore(final Path rootDir, final HTableDescriptor sourceHtd, + final HTableDescriptor htdClone) throws IOException { + String[] files = getHFiles(HTableDescriptor.getTableDir(rootDir, htdClone.getName())); + assertEquals(2, files.length); + assertTrue(files[0] + " should be a HFileLink", HFileLink.isHFileLink(files[0])); + assertTrue(files[1] + " should be a Referene", StoreFile.isReference(files[1])); + assertEquals(sourceHtd.getNameAsString(), HFileLink.getReferencedTableName(files[0])); + assertEquals(TEST_HFILE, HFileLink.getReferencedHFileName(files[0])); + Path refPath = getReferredToFile(files[1]); + assertTrue(refPath.getName() + " should be a HFileLink", HFileLink.isHFileLink(refPath.getName())); + assertEquals(files[0], refPath.getName()); + } + + /** + * Execute the restore operation + * @param snapshotDir The snapshot directory to use as "restore source" + * @param sourceTableName The name of the snapshotted table + * @param htdClone The HTableDescriptor of the table to restore/clone. + */ + public void testRestore(final Path snapshotDir, final String sourceTableName, + final HTableDescriptor htdClone) throws IOException { + LOG.debug("pre-restore table=" + htdClone.getNameAsString() + " snapshot=" + snapshotDir); + FSUtils.logFileSystemState(fs, rootDir, LOG); + + FSTableDescriptors.createTableDescriptor(htdClone, conf); + RestoreSnapshotHelper helper = getRestoreHelper(rootDir, snapshotDir, sourceTableName, htdClone); + helper.restoreHdfsRegions(); + + LOG.debug("post-restore table=" + htdClone.getNameAsString() + " snapshot=" + snapshotDir); + FSUtils.logFileSystemState(fs, rootDir, LOG); + } + + /** + * Initialize the restore helper, based on the snapshot and table information provided. + */ + private RestoreSnapshotHelper getRestoreHelper(final Path rootDir, final Path snapshotDir, + final String sourceTableName, final HTableDescriptor htdClone) throws IOException { + CatalogTracker catalogTracker = Mockito.mock(CatalogTracker.class); + HTableDescriptor tableDescriptor = Mockito.mock(HTableDescriptor.class); + ForeignExceptionDispatcher monitor = Mockito.mock(ForeignExceptionDispatcher.class); + MonitoredTask status = Mockito.mock(MonitoredTask.class); + + SnapshotDescription sd = SnapshotDescription.newBuilder() + .setName("snapshot").setTable(sourceTableName).build(); + + return new RestoreSnapshotHelper(conf, fs, sd, snapshotDir, + htdClone, HTableDescriptor.getTableDir(rootDir, htdClone.getName()), monitor, status); + } + + private void createSnapshot(final Path rootDir, final Path snapshotDir, final HTableDescriptor htd) + throws IOException { + // First region, simple with one plain hfile. + HRegion r0 = HRegion.createHRegion(new HRegionInfo(htd.getName()), archiveDir, + conf, htd, null, true, true); + Path storeFile = new Path(new Path(r0.getRegionDir(), TEST_FAMILY), TEST_HFILE); + fs.createNewFile(storeFile); + r0.close(); + + // Second region, used to test the split case. + // This region contains a reference to the hfile in the first region. + HRegion r1 = HRegion.createHRegion(new HRegionInfo(htd.getName()), archiveDir, + conf, htd, null, true, true); + fs.createNewFile(new Path(new Path(r1.getRegionDir(), TEST_FAMILY), + storeFile.getName() + '.' + r0.getRegionInfo().getEncodedName())); + r1.close(); + + Path tableDir = HTableDescriptor.getTableDir(archiveDir, htd.getName()); + FileUtil.copy(fs, tableDir, fs, snapshotDir, false, conf); + } + + private HTableDescriptor createTableDescriptor(final String tableName) { + HTableDescriptor htd = new HTableDescriptor(tableName); + htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); + return htd; + } + + private Path getReferredToFile(final String referenceName) { + Path fakeBasePath = new Path(new Path("table", "region"), "cf"); + return StoreFile.getReferredToFile(new Path(fakeBasePath, referenceName)); + } + + private String[] getHFiles(final Path tableDir) throws IOException { + List files = new ArrayList(); + for (Path regionDir: FSUtils.getRegionDirs(fs, tableDir)) { + for (Path familyDir: FSUtils.getFamilyDirs(fs, regionDir)) { + for (FileStatus file: FSUtils.listStatus(fs, familyDir)) { + files.add(file.getPath().getName()); + } + } + } + Collections.sort(files); + return files.toArray(new String[files.size()]); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotDescriptionUtils.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotDescriptionUtils.java new file mode 100644 index 000000000000..0a32ca3db46c --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotDescriptionUtils.java @@ -0,0 +1,105 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.fail; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription.Type; +import org.apache.hadoop.hbase.util.EnvironmentEdge; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManagerTestHelper; +import org.junit.After; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test that the {@link SnapshotDescription} helper is helping correctly. + */ +@Category(MediumTests.class) +public class TestSnapshotDescriptionUtils { + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static FileSystem fs; + private static Path root; + + @BeforeClass + public static void setupFS() throws Exception { + fs = UTIL.getTestFileSystem(); + root = new Path(UTIL.getDataTestDir(), "hbase"); + } + + @After + public void cleanupFS() throws Exception { + if (fs.exists(root)) { + if (!fs.delete(root, true)) { + throw new IOException("Failed to delete root test dir: " + root); + } + if (!fs.mkdirs(root)) { + throw new IOException("Failed to create root test dir: " + root); + } + } + EnvironmentEdgeManagerTestHelper.reset(); + } + + private static final Log LOG = LogFactory.getLog(TestSnapshotDescriptionUtils.class); + + @Test + public void testValidateMissingTableName() { + Configuration conf = new Configuration(false); + try { + SnapshotDescriptionUtils.validate(SnapshotDescription.newBuilder().setName("fail").build(), + conf); + fail("Snapshot was considered valid without a table name"); + } catch (IllegalArgumentException e) { + LOG.debug("Correctly failed when snapshot doesn't have a tablename"); + } + } + + /** + * Test that we throw an exception if there is no working snapshot directory when we attempt to + * 'complete' the snapshot + * @throws Exception on failure + */ + @Test + public void testCompleteSnapshotWithNoSnapshotDirectoryFailure() throws Exception { + Path snapshotDir = new Path(root, HConstants.SNAPSHOT_DIR_NAME); + Path tmpDir = new Path(snapshotDir, ".tmp"); + Path workingDir = new Path(tmpDir, "not_a_snapshot"); + assertFalse("Already have working snapshot dir: " + workingDir + + " but shouldn't. Test file leak?", fs.exists(workingDir)); + SnapshotDescription snapshot = SnapshotDescription.newBuilder().setName("snapshot").build(); + try { + SnapshotDescriptionUtils.completeSnapshot(snapshot, root, workingDir, fs); + fail("Shouldn't successfully complete move of a non-existent directory."); + } catch (IOException e) { + LOG.info("Correctly failed to move non-existant directory: " + e.getMessage()); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotLogSplitter.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotLogSplitter.java new file mode 100644 index 000000000000..66b941a1c91b --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotLogSplitter.java @@ -0,0 +1,176 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.Map; +import java.util.TreeMap; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.regionserver.wal.HLogKey; +import org.apache.hadoop.hbase.regionserver.wal.WALEdit; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.*; +import org.junit.experimental.categories.Category; + +/** + * Test snapshot log splitter + */ +@Category(SmallTests.class) +public class TestSnapshotLogSplitter { + final Log LOG = LogFactory.getLog(getClass()); + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + private byte[] TEST_QUALIFIER = Bytes.toBytes("q"); + private byte[] TEST_FAMILY = Bytes.toBytes("f"); + + private Configuration conf; + private FileSystem fs; + private Path logFile; + + @Before + public void setup() throws Exception { + conf = TEST_UTIL.getConfiguration(); + fs = FileSystem.get(conf); + logFile = new Path(TEST_UTIL.getDataTestDir(), "test.log"); + writeTestLog(logFile); + } + + @After + public void tearDown() throws Exception { + fs.delete(logFile, false); + } + + @Test + public void testSplitLogs() throws IOException { + Map regionsMap = new TreeMap(Bytes.BYTES_COMPARATOR); + splitTestLogs(getTableName(5), regionsMap); + } + + @Test + public void testSplitLogsOnDifferentTable() throws IOException { + byte[] tableName = getTableName(1); + Map regionsMap = new TreeMap(Bytes.BYTES_COMPARATOR); + for (int j = 0; j < 10; ++j) { + byte[] regionName = getRegionName(tableName, j); + byte[] newRegionName = getNewRegionName(tableName, j); + regionsMap.put(regionName, newRegionName); + } + splitTestLogs(tableName, regionsMap); + } + + /* + * Split and verify test logs for the specified table + */ + private void splitTestLogs(final byte[] tableName, final Map regionsMap) + throws IOException { + Path tableDir = new Path(TEST_UTIL.getDataTestDir(), Bytes.toString(tableName)); + SnapshotLogSplitter logSplitter = new SnapshotLogSplitter(conf, fs, tableDir, + tableName, regionsMap); + try { + logSplitter.splitLog(logFile); + } finally { + logSplitter.close(); + } + verifyRecoverEdits(tableDir, tableName, regionsMap); + } + + /* + * Verify that every logs in the table directory has just the specified table and regions. + */ + private void verifyRecoverEdits(final Path tableDir, final byte[] tableName, + final Map regionsMap) throws IOException { + for (FileStatus regionStatus: FSUtils.listStatus(fs, tableDir)) { + assertTrue(regionStatus.getPath().getName().startsWith(Bytes.toString(tableName))); + Path regionEdits = HLog.getRegionDirRecoveredEditsDir(regionStatus.getPath()); + byte[] regionName = Bytes.toBytes(regionStatus.getPath().getName()); + assertFalse(regionsMap.containsKey(regionName)); + for (FileStatus logStatus: FSUtils.listStatus(fs, regionEdits)) { + HLog.Reader reader = HLog.getReader(fs, logStatus.getPath(), conf); + try { + HLog.Entry entry; + while ((entry = reader.next()) != null) { + HLogKey key = entry.getKey(); + assertArrayEquals(tableName, key.getTablename()); + assertArrayEquals(regionName, key.getEncodedRegionName()); + } + } finally { + reader.close(); + } + } + } + } + + /* + * Write some entries in the log file. + * 7 different tables with name "testtb-%d" + * 10 region per table with name "tableName-region-%d" + * 50 entry with row key "row-%d" + */ + private void writeTestLog(final Path logFile) throws IOException { + fs.mkdirs(logFile.getParent()); + HLog.Writer writer = HLog.createWriter(fs, logFile, conf); + try { + for (int i = 0; i < 7; ++i) { + byte[] tableName = getTableName(i); + for (int j = 0; j < 10; ++j) { + byte[] regionName = getRegionName(tableName, j); + for (int k = 0; k < 50; ++k) { + byte[] rowkey = Bytes.toBytes("row-" + k); + HLogKey key = new HLogKey(regionName, tableName, (long)k, + System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID); + WALEdit edit = new WALEdit(); + edit.add(new KeyValue(rowkey, TEST_FAMILY, TEST_QUALIFIER, rowkey)); + writer.append(new HLog.Entry(key, edit)); + } + } + } + } finally { + writer.close(); + } + } + + private byte[] getTableName(int tableId) { + return Bytes.toBytes("testtb-" + tableId); + } + + private byte[] getRegionName(final byte[] tableName, int regionId) { + return Bytes.toBytes(Bytes.toString(tableName) + "-region-" + regionId); + } + + private byte[] getNewRegionName(final byte[] tableName, int regionId) { + return Bytes.toBytes(Bytes.toString(tableName) + "-new-region-" + regionId); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotTask.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotTask.java new file mode 100644 index 000000000000..36b70508df74 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotTask.java @@ -0,0 +1,58 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; + +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.errorhandling.ForeignException; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.snapshot.SnapshotTask; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +@Category(SmallTests.class) +public class TestSnapshotTask { + + /** + * Check that errors from running the task get propagated back to the error listener. + */ + @Test + public void testErrorPropagation() throws Exception { + ForeignExceptionDispatcher error = mock(ForeignExceptionDispatcher.class); + SnapshotDescription snapshot = SnapshotDescription.newBuilder().setName("snapshot") + .setTable("table").build(); + final Exception thrown = new Exception("Failed!"); + SnapshotTask fail = new SnapshotTask(snapshot, error) { + @Override + public Void call() { + snapshotFailure("Injected failure", thrown); + return null; + } + }; + fail.call(); + + verify(error, Mockito.times(1)).receive(any(ForeignException.class)); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/snapshot/TestWALReferenceTask.java b/src/test/java/org/apache/hadoop/hbase/snapshot/TestWALReferenceTask.java new file mode 100644 index 000000000000..a813ba78f9b6 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/snapshot/TestWALReferenceTask.java @@ -0,0 +1,103 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.snapshot; + +import java.io.IOException; +import java.util.HashSet; +import java.util.Set; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher; +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription; +import org.apache.hadoop.hbase.snapshot.ReferenceServerWALsTask; +import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils; +import org.apache.hadoop.hbase.snapshot.TakeSnapshotUtils; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +/** + * Test that the WAL reference task works as expected + */ +@Category(SmallTests.class) +public class TestWALReferenceTask { + + private static final Log LOG = LogFactory.getLog(TestWALReferenceTask.class); + private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @Test + public void testRun() throws IOException { + Configuration conf = UTIL.getConfiguration(); + FileSystem fs = UTIL.getTestFileSystem(); + // setup the log dir + Path testDir = UTIL.getDataTestDir(); + Set servers = new HashSet(); + Path logDir = new Path(testDir, ".logs"); + Path server1Dir = new Path(logDir, "Server1"); + servers.add(server1Dir.getName()); + Path server2Dir = new Path(logDir, "me.hbase.com,56073,1348618509968"); + servers.add(server2Dir.getName()); + // logs under server 1 + Path log1_1 = new Path(server1Dir, "me.hbase.com%2C56073%2C1348618509968.1348618520536"); + Path log1_2 = new Path(server1Dir, "me.hbase.com%2C56073%2C1348618509968.1234567890123"); + // logs under server 2 + Path log2_1 = new Path(server2Dir, "me.hbase.com%2C56074%2C1348618509998.1348618515589"); + Path log2_2 = new Path(server2Dir, "me.hbase.com%2C56073%2C1348618509968.1234567890123"); + + // create all the log files + fs.createNewFile(log1_1); + fs.createNewFile(log1_2); + fs.createNewFile(log2_1); + fs.createNewFile(log2_2); + + FSUtils.logFileSystemState(fs, testDir, LOG); + FSUtils.setRootDir(conf, testDir); + SnapshotDescription snapshot = SnapshotDescription.newBuilder() + .setName("testWALReferenceSnapshot").build(); + ForeignExceptionDispatcher listener = Mockito.mock(ForeignExceptionDispatcher.class); + + // reference all the files in the first server directory + ReferenceServerWALsTask task = new ReferenceServerWALsTask(snapshot, listener, server1Dir, + conf, fs); + task.call(); + + // reference all the files in the first server directory + task = new ReferenceServerWALsTask(snapshot, listener, server2Dir, conf, fs); + task.call(); + + // verify that we got everything + FSUtils.logFileSystemState(fs, testDir, LOG); + Path workingDir = SnapshotDescriptionUtils.getWorkingSnapshotDir(snapshot, testDir); + Path snapshotLogDir = new Path(workingDir, HConstants.HREGION_LOGDIR_NAME); + + // make sure we reference the all the wal files + TakeSnapshotUtils.verifyAllLogsGotReferenced(fs, logDir, servers, snapshot, snapshotLogDir); + + // make sure we never got an error + Mockito.verify(listener, Mockito.atLeastOnce()).rethrowException(); + Mockito.verifyNoMoreInteractions(listener); + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java b/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java new file mode 100644 index 000000000000..8a397f5cbc56 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java @@ -0,0 +1,1094 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.test; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import java.io.StringWriter; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Random; +import java.util.UUID; + +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.GnuParser; +import org.apache.commons.cli.HelpFormatter; +import org.apache.commons.cli.Options; +import org.apache.commons.cli.ParseException; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.conf.Configured; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.IntegrationTestingUtility; +import org.apache.hadoop.hbase.IntegrationTests; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.ScannerCallable; +import org.apache.hadoop.hbase.io.ImmutableBytesWritable; +import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; +import org.apache.hadoop.hbase.mapreduce.TableMapper; +import org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.io.BytesWritable; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.io.Writable; +import org.apache.hadoop.mapreduce.Counter; +import org.apache.hadoop.mapreduce.Counters; +import org.apache.hadoop.mapreduce.InputFormat; +import org.apache.hadoop.mapreduce.InputSplit; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapreduce.JobContext; +import org.apache.hadoop.mapreduce.Mapper; +import org.apache.hadoop.mapreduce.RecordReader; +import org.apache.hadoop.mapreduce.Reducer; +import org.apache.hadoop.mapreduce.TaskAttemptContext; +import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; +import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat; +import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; +import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat; +import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat; +import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * This is an integration test borrowed from goraci, written by Keith Turner, + * which is in turn inspired by the Accumulo test called continous ingest (ci). + * The original source code can be found here: + * https://github.com/keith-turner/goraci + * https://github.com/enis/goraci/ + * + * Apache Accumulo [0] has a simple test suite that verifies that data is not + * lost at scale. This test suite is called continuous ingest. This test runs + * many ingest clients that continually create linked lists containing 25 + * million nodes. At some point the clients are stopped and a map reduce job is + * run to ensure no linked list has a hole. A hole indicates data was lost.·· + * + * The nodes in the linked list are random. This causes each linked list to + * spread across the table. Therefore if one part of a table loses data, then it + * will be detected by references in another part of the table. + * + * THE ANATOMY OF THE TEST + * + * Below is rough sketch of how data is written. For specific details look at + * the Generator code. + * + * 1 Write out 1 million nodes· 2 Flush the client· 3 Write out 1 million that + * reference previous million· 4 If this is the 25th set of 1 million nodes, + * then update 1st set of million to point to last· 5 goto 1 + * + * The key is that nodes only reference flushed nodes. Therefore a node should + * never reference a missing node, even if the ingest client is killed at any + * point in time. + * + * When running this test suite w/ Accumulo there is a script running in + * parallel called the Aggitator that randomly and continuously kills server + * processes.·· The outcome was that many data loss bugs were found in Accumulo + * by doing this.· This test suite can also help find bugs that impact uptime + * and stability when· run for days or weeks.·· + * + * This test suite consists the following· - a few Java programs· - a little + * helper script to run the java programs - a maven script to build it.·· + * + * When generating data, its best to have each map task generate a multiple of + * 25 million. The reason for this is that circular linked list are generated + * every 25M. Not generating a multiple in 25M will result in some nodes in the + * linked list not having references. The loss of an unreferenced node can not + * be detected. + * + * + * Below is a description of the Java programs + * + * Generator - A map only job that generates data. As stated previously,· + * its best to generate data in multiples of 25M. + * + * Verify - A map reduce job that looks for holes. Look at the counts after running. REFERENCED and + * UNREFERENCED are· ok, any UNDEFINED counts are bad. Do not run at the· same + * time as the Generator. + * + * Walker - A standalong program that start following a linked list· and emits timing info.·· + * + * Print - A standalone program that prints nodes in the linked list + * + * Delete - A standalone program that deletes a single node + * + * This class can be run as a unit test, as an integration test, or from the command line + */ +@Category(IntegrationTests.class) +public class IntegrationTestBigLinkedList extends Configured implements Tool { + private static final byte[] NO_KEY = new byte[1]; + + private static final String TABLE_NAME_KEY = "IntegrationTestBigLinkedList.table"; + + private static final String DEFAULT_TABLE_NAME = "IntegrationTestBigLinkedList"; + + private static byte[] FAMILY_NAME = Bytes.toBytes("meta"); + + //link to the id of the prev node in the linked list + private static final byte[] COLUMN_PREV = Bytes.toBytes("prev"); + + //identifier of the mapred task that generated this row + private static final byte[] COLUMN_CLIENT = Bytes.toBytes("client"); + + //the id of the row within the same client. + private static final byte[] COLUMN_COUNT = Bytes.toBytes("count"); + + /** How many rows to write per map task. This has to be a multiple of 25M */ + private static final String GENERATOR_NUM_ROWS_PER_MAP_KEY + = "IntegrationTestBigLinkedList.generator.num_rows"; + + private static final String GENERATOR_NUM_MAPPERS_KEY + = "IntegrationTestBigLinkedList.generator.map.tasks"; + + private static final String GENERATOR_WIDTH_KEY + = "IntegrationTestBigLinkedList.generator.width"; + + private static final String GENERATOR_WRAP_KEY + = "IntegrationTestBigLinkedList.generator.wrap"; + + protected int NUM_SLAVES_BASE = 3; // number of slaves for the cluster + + private static final int WIDTH_DEFAULT = 1000000; + private static final int WRAP_DEFAULT = 25; + + private static final int ROWKEY_LENGTH = 16; + + static class CINode { + byte[] key; + byte[] prev; + + String client; + long count; + } + + /** + * A Map only job that generates random linked list and stores them. + */ + static class Generator extends Configured implements Tool { + + private static final Log LOG = LogFactory.getLog(Generator.class); + + public static enum Counts { + UNREFERENCED, UNDEFINED, REFERENCED, CORRUPT + } + + static class GeneratorInputFormat extends InputFormat { + static class GeneratorInputSplit extends InputSplit implements Writable { + @Override + public long getLength() throws IOException, InterruptedException { + return 1; + } + @Override + public String[] getLocations() throws IOException, InterruptedException { + return new String[0]; + } + @Override + public void readFields(DataInput arg0) throws IOException { + } + @Override + public void write(DataOutput arg0) throws IOException { + } + } + + static class GeneratorRecordReader extends RecordReader { + private long count; + private long numNodes; + private Random rand; + + @Override + public void close() throws IOException { + } + + @Override + public BytesWritable getCurrentKey() throws IOException, InterruptedException { + byte[] bytes = new byte[ROWKEY_LENGTH]; + rand.nextBytes(bytes); + return new BytesWritable(bytes); + } + + @Override + public NullWritable getCurrentValue() throws IOException, InterruptedException { + return NullWritable.get(); + } + + @Override + public float getProgress() throws IOException, InterruptedException { + return (float)(count / (double)numNodes); + } + + @Override + public void initialize(InputSplit arg0, TaskAttemptContext context) + throws IOException, InterruptedException { + numNodes = context.getConfiguration().getLong(GENERATOR_NUM_ROWS_PER_MAP_KEY, 25000000); + rand = new Random(); + } + + @Override + public boolean nextKeyValue() throws IOException, InterruptedException { + return count++ < numNodes; + } + + } + + @Override + public RecordReader createRecordReader( + InputSplit split, TaskAttemptContext context) throws IOException, InterruptedException { + GeneratorRecordReader rr = new GeneratorRecordReader(); + rr.initialize(split, context); + return rr; + } + + @Override + public List getSplits(JobContext job) throws IOException, InterruptedException { + int numMappers = job.getConfiguration().getInt(GENERATOR_NUM_MAPPERS_KEY, 1); + + ArrayList splits = new ArrayList(numMappers); + + for (int i = 0; i < numMappers; i++) { + splits.add(new GeneratorInputSplit()); + } + + return splits; + } + } + + /** Ensure output files from prev-job go to map inputs for current job */ + static class OneFilePerMapperSFIF extends SequenceFileInputFormat { + @Override + protected boolean isSplitable(JobContext context, Path filename) { + return false; + } + } + + /** + * Some ASCII art time: + * [ . . . ] represents one batch of random longs of length WIDTH + * + * _________________________ + * | ______ | + * | | || + * __+_________________+_____ || + * v v v ||| + * first = [ . . . . . . . . . . . ] ||| + * ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ||| + * | | | | | | | | | | | ||| + * prev = [ . . . . . . . . . . . ] ||| + * ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ||| + * | | | | | | | | | | | ||| + * current = [ . . . . . . . . . . . ] ||| + * ||| + * ... ||| + * ||| + * last = [ . . . . . . . . . . . ] ||| + * | | | | | | | | | | |-----||| + * | |--------|| + * |___________________________| + */ + static class GeneratorMapper + extends Mapper { + Random rand = new Random(); + + byte[][] first = null; + byte[][] prev = null; + byte[][] current = null; + byte[] id; + long count = 0; + int i; + HTable table; + long numNodes; + long wrap; + int width; + + protected void setup(Context context) throws IOException, InterruptedException { + id = Bytes.toBytes(UUID.randomUUID().toString()); + Configuration conf = context.getConfiguration(); + table = new HTable(conf, getTableName(conf)); + table.setAutoFlush(false); + table.setWriteBufferSize(4 * 1024 * 1024); + this.width = context.getConfiguration().getInt(GENERATOR_WIDTH_KEY, WIDTH_DEFAULT); + current = new byte[this.width][]; + int wrapMultiplier = context.getConfiguration().getInt(GENERATOR_WRAP_KEY, WRAP_DEFAULT); + this.wrap = (long)wrapMultiplier * width; + this.numNodes = context.getConfiguration().getLong( + GENERATOR_NUM_ROWS_PER_MAP_KEY, (long)WIDTH_DEFAULT * WRAP_DEFAULT); + if (this.numNodes < this.wrap) { + this.wrap = this.numNodes; + } + }; + + protected void cleanup(Context context) throws IOException ,InterruptedException { + table.close(); + }; + + @Override + protected void map(BytesWritable key, NullWritable value, Context output) throws IOException { + current[i] = new byte[key.getLength()]; + System.arraycopy(key.getBytes(), 0, current[i], 0, key.getLength()); + if (++i == current.length) { + persist(output, count, prev, current, id); + i = 0; + + if (first == null) + first = current; + prev = current; + current = new byte[this.width][]; + + count += current.length; + output.setStatus("Count " + count); + + if (count % wrap == 0) { + // this block of code turns the 1 million linked list of length 25 into one giant + //circular linked list of 25 million + circularLeftShift(first); + + persist(output, -1, prev, first, null); + + first = null; + prev = null; + } + } + } + + private static void circularLeftShift(T[] first) { + T ez = first[0]; + for (int i = 0; i < first.length - 1; i++) + first[i] = first[i + 1]; + first[first.length - 1] = ez; + } + + private void persist(Context output, long count, byte[][] prev, byte[][] current, byte[] id) + throws IOException { + for (int i = 0; i < current.length; i++) { + Put put = new Put(current[i]); + put.add(FAMILY_NAME, COLUMN_PREV, prev == null ? NO_KEY : prev[i]); + + if (count >= 0) { + put.add(FAMILY_NAME, COLUMN_COUNT, Bytes.toBytes(count + i)); + } + if (id != null) { + put.add(FAMILY_NAME, COLUMN_CLIENT, id); + } + table.put(put); + + if (i % 1000 == 0) { + // Tickle progress every so often else maprunner will think us hung + output.progress(); + } + } + + table.flushCommits(); + } + } + + @Override + public int run(String[] args) throws Exception { + if (args.length < 3) { + System.out.println("Usage : " + Generator.class.getSimpleName() + + " [ ]"); + System.out.println(" where should be a multiple of " + + " width*wrap multiplier, 25M by default"); + return 0; + } + + int numMappers = Integer.parseInt(args[0]); + long numNodes = Long.parseLong(args[1]); + Path tmpOutput = new Path(args[2]); + Integer width = (args.length < 4) ? null : Integer.parseInt(args[3]); + Integer wrapMuplitplier = (args.length < 5) ? null : Integer.parseInt(args[4]); + return run(numMappers, numNodes, tmpOutput, width, wrapMuplitplier); + } + + protected void createSchema() throws IOException { + HBaseAdmin admin = new HBaseAdmin(getConf()); + byte[] tableName = getTableName(getConf()); + if (!admin.tableExists(tableName)) { + HTableDescriptor htd = new HTableDescriptor(getTableName(getConf())); + htd.addFamily(new HColumnDescriptor(FAMILY_NAME)); + admin.createTable(htd); + } + admin.close(); + } + + public int runRandomInputGenerator(int numMappers, long numNodes, Path tmpOutput, + Integer width, Integer wrapMuplitplier) throws Exception { + LOG.info("Running RandomInputGenerator with numMappers=" + numMappers + + ", numNodes=" + numNodes); + Job job = new Job(getConf()); + + job.setJobName("Random Input Generator"); + job.setNumReduceTasks(0); + job.setJarByClass(getClass()); + + job.setInputFormatClass(GeneratorInputFormat.class); + job.setOutputKeyClass(BytesWritable.class); + job.setOutputValueClass(NullWritable.class); + + setJobConf(job, numMappers, numNodes, width, wrapMuplitplier); + + job.setMapperClass(Mapper.class); //identity mapper + + FileOutputFormat.setOutputPath(job, tmpOutput); + job.setOutputFormatClass(SequenceFileOutputFormat.class); + + boolean success = job.waitForCompletion(true); + + return success ? 0 : 1; + } + + public int runGenerator(int numMappers, long numNodes, Path tmpOutput, + Integer width, Integer wrapMuplitplier) throws Exception { + LOG.info("Running Generator with numMappers=" + numMappers +", numNodes=" + numNodes); + createSchema(); + + Job job = new Job(getConf()); + + job.setJobName("Link Generator"); + job.setNumReduceTasks(0); + job.setJarByClass(getClass()); + + FileInputFormat.setInputPaths(job, tmpOutput); + job.setInputFormatClass(OneFilePerMapperSFIF.class); + job.setOutputKeyClass(NullWritable.class); + job.setOutputValueClass(NullWritable.class); + + setJobConf(job, numMappers, numNodes, width, wrapMuplitplier); + + job.setMapperClass(GeneratorMapper.class); + + job.setOutputFormatClass(NullOutputFormat.class); + + job.getConfiguration().setBoolean("mapred.map.tasks.speculative.execution", false); + TableMapReduceUtil.addDependencyJars(job); + TableMapReduceUtil.initCredentials(job); + + boolean success = job.waitForCompletion(true); + + return success ? 0 : 1; + } + + public int run(int numMappers, long numNodes, Path tmpOutput, + Integer width, Integer wrapMuplitplier) throws Exception { + int ret = runRandomInputGenerator(numMappers, numNodes, tmpOutput, width, wrapMuplitplier); + if (ret > 0) { + return ret; + } + return runGenerator(numMappers, numNodes, tmpOutput, width, wrapMuplitplier); + } + } + + /** + * A Map Reduce job that verifies that the linked lists generated by + * {@link Generator} do not have any holes. + */ + static class Verify extends Configured implements Tool { + + private static final Log LOG = LogFactory.getLog(Verify.class); + private static final BytesWritable DEF = new BytesWritable(NO_KEY); + + private Job job; + + public static class VerifyMapper extends TableMapper { + private BytesWritable row = new BytesWritable(); + private BytesWritable ref = new BytesWritable(); + + @Override + protected void map(ImmutableBytesWritable key, Result value, Context context) + throws IOException ,InterruptedException { + byte[] rowKey = key.get(); + row.set(rowKey, 0, rowKey.length); + context.write(row, DEF); + byte[] prev = value.getValue(FAMILY_NAME, COLUMN_PREV); + if (prev != null && prev.length > 0) { + ref.set(prev, 0, prev.length); + context.write(ref, row); + } else { + LOG.warn(String.format("Prev is not set for: %s", Bytes.toStringBinary(rowKey))); + } + } + } + + public static enum Counts { + UNREFERENCED, UNDEFINED, REFERENCED, CORRUPT, EXTRAREFERENCES + } + + public static class VerifyReducer extends Reducer { + private ArrayList refs = new ArrayList(); + + public void reduce(BytesWritable key, Iterable values, Context context) + throws IOException, InterruptedException { + + int defCount = 0; + + refs.clear(); + for (BytesWritable type : values) { + if (type.getLength() == DEF.getLength()) { + defCount++; + } else { + byte[] bytes = new byte[type.getLength()]; + System.arraycopy(type.getBytes(), 0, bytes, 0, type.getLength()); + refs.add(bytes); + } + } + + // TODO check for more than one def, should not happen + + StringBuilder refsSb = null; + String keyString = null; + if (defCount == 0 || refs.size() != 1) { + refsSb = new StringBuilder(); + String comma = ""; + for (byte[] ref : refs) { + refsSb.append(comma); + comma = ","; + refsSb.append(Bytes.toStringBinary(ref)); + } + byte[] bytes = new byte[key.getLength()]; + keyString = Bytes.toStringBinary(key.getBytes(), 0, key.getLength()); + } + + if (defCount == 0 && refs.size() > 0) { + // this is bad, found a node that is referenced but not defined. It must have been + // lost, emit some info about this node for debugging purposes. + context.write(new Text(keyString), new Text(refsSb.toString())); + context.getCounter(Counts.UNDEFINED).increment(1); + } else if (defCount > 0 && refs.size() == 0) { + // node is defined but not referenced + context.write(new Text(keyString), new Text("none")); + context.getCounter(Counts.UNREFERENCED).increment(1); + } else { + if (refs.size() > 1) { + context.write(new Text(keyString), new Text(refsSb.toString())); + context.getCounter(Counts.EXTRAREFERENCES).increment(refs.size() - 1); + } + // node is defined and referenced + context.getCounter(Counts.REFERENCED).increment(1); + } + + } + } + + @Override + public int run(String[] args) throws Exception { + + if (args.length != 2) { + System.out.println("Usage : " + Verify.class.getSimpleName() + " "); + return 0; + } + + String outputDir = args[0]; + int numReducers = Integer.parseInt(args[1]); + + return run(outputDir, numReducers); + } + + public int run(String outputDir, int numReducers) throws Exception { + return run(new Path(outputDir), numReducers); + } + + public int run(Path outputDir, int numReducers) throws Exception { + LOG.info("Running Verify with outputDir=" + outputDir +", numReducers=" + numReducers); + + job = new Job(getConf()); + + job.setJobName("Link Verifier"); + job.setNumReduceTasks(numReducers); + job.setJarByClass(getClass()); + + setJobScannerConf(job); + + Scan scan = new Scan(); + scan.addColumn(FAMILY_NAME, COLUMN_PREV); + scan.setCaching(10000); + scan.setCacheBlocks(false); + + TableMapReduceUtil.initTableMapperJob(getTableName(getConf()), scan, + VerifyMapper.class, BytesWritable.class, BytesWritable.class, job); + + job.getConfiguration().setBoolean("mapred.map.tasks.speculative.execution", false); + + job.setReducerClass(VerifyReducer.class); + job.setOutputFormatClass(TextOutputFormat.class); + TextOutputFormat.setOutputPath(job, outputDir); + + boolean success = job.waitForCompletion(true); + + return success ? 0 : 1; + } + + public boolean verify(long expectedReferenced) throws Exception { + if (job == null) { + throw new IllegalStateException("You should call run() first"); + } + + Counters counters = job.getCounters(); + + Counter referenced = counters.findCounter(Counts.REFERENCED); + Counter unreferenced = counters.findCounter(Counts.UNREFERENCED); + Counter undefined = counters.findCounter(Counts.UNDEFINED); + Counter multiref = counters.findCounter(Counts.EXTRAREFERENCES); + + boolean success = true; + //assert + if (expectedReferenced != referenced.getValue()) { + LOG.error("Expected referenced count does not match with actual referenced count. " + + "expected referenced=" + expectedReferenced + " ,actual=" + referenced.getValue()); + success = false; + } + + if (unreferenced.getValue() > 0) { + boolean couldBeMultiRef = (multiref.getValue() == unreferenced.getValue()); + LOG.error("Unreferenced nodes were not expected. Unreferenced count=" + unreferenced.getValue() + + (couldBeMultiRef ? "; could be due to duplicate random numbers" : "")); + success = false; + } + + if (undefined.getValue() > 0) { + LOG.error("Found an undefined node. Undefined count=" + undefined.getValue()); + success = false; + } + + return success; + } + } + + /** + * Executes Generate and Verify in a loop. Data is not cleaned between runs, so each iteration + * adds more data. + */ + private static class Loop extends Configured implements Tool { + + private static final Log LOG = LogFactory.getLog(Loop.class); + + protected void runGenerator(int numMappers, long numNodes, + String outputDir, Integer width, Integer wrapMuplitplier) throws Exception { + Path outputPath = new Path(outputDir); + UUID uuid = UUID.randomUUID(); //create a random UUID. + Path generatorOutput = new Path(outputPath, uuid.toString()); + + Generator generator = new Generator(); + generator.setConf(getConf()); + int retCode = generator.run(numMappers, numNodes, generatorOutput, width, wrapMuplitplier); + if (retCode > 0) { + throw new RuntimeException("Generator failed with return code: " + retCode); + } + } + + protected void runVerify(String outputDir, int numReducers, long expectedNumNodes) throws Exception { + Path outputPath = new Path(outputDir); + UUID uuid = UUID.randomUUID(); //create a random UUID. + Path iterationOutput = new Path(outputPath, uuid.toString()); + + Verify verify = new Verify(); + verify.setConf(getConf()); + int retCode = verify.run(iterationOutput, numReducers); + if (retCode > 0) { + throw new RuntimeException("Verify.run failed with return code: " + retCode); + } + + boolean verifySuccess = verify.verify(expectedNumNodes); + if (!verifySuccess) { + throw new RuntimeException("Verify.verify failed"); + } + + LOG.info("Verify finished with succees. Total nodes=" + expectedNumNodes); + } + + @Override + public int run(String[] args) throws Exception { + if (args.length < 5) { + System.err.println("Usage: Loop [ ]"); + return 1; + } + LOG.info("Running Loop with args:" + Arrays.deepToString(args)); + + int numIterations = Integer.parseInt(args[0]); + int numMappers = Integer.parseInt(args[1]); + long numNodes = Long.parseLong(args[2]); + String outputDir = args[3]; + int numReducers = Integer.parseInt(args[4]); + Integer width = (args.length < 6) ? null : Integer.parseInt(args[5]); + Integer wrapMuplitplier = (args.length < 7) ? null : Integer.parseInt(args[6]); + + long expectedNumNodes = 0; + + if (numIterations < 0) { + numIterations = Integer.MAX_VALUE; //run indefinitely (kind of) + } + + for (int i = 0; i < numIterations; i++) { + LOG.info("Starting iteration = " + i); + runGenerator(numMappers, numNodes, outputDir, width, wrapMuplitplier); + expectedNumNodes += numMappers * numNodes; + + runVerify(outputDir, numReducers, expectedNumNodes); + } + + return 0; + } + } + + /** + * A stand alone program that prints out portions of a list created by {@link Generator} + */ + private static class Print extends Configured implements Tool { + public int run(String[] args) throws Exception { + Options options = new Options(); + options.addOption("s", "start", true, "start key"); + options.addOption("e", "end", true, "end key"); + options.addOption("l", "limit", true, "number to print"); + + GnuParser parser = new GnuParser(); + CommandLine cmd = null; + try { + cmd = parser.parse(options, args); + if (cmd.getArgs().length != 0) { + throw new ParseException("Command takes no arguments"); + } + } catch (ParseException e) { + System.err.println("Failed to parse command line " + e.getMessage()); + System.err.println(); + HelpFormatter formatter = new HelpFormatter(); + formatter.printHelp(getClass().getSimpleName(), options); + System.exit(-1); + } + + HTable table = new HTable(getConf(), getTableName(getConf())); + + Scan scan = new Scan(); + scan.setBatch(10000); + + if (cmd.hasOption("s")) + scan.setStartRow(Bytes.toBytesBinary(cmd.getOptionValue("s"))); + + if (cmd.hasOption("e")) + scan.setStopRow(Bytes.toBytesBinary(cmd.getOptionValue("e"))); + + int limit = 0; + if (cmd.hasOption("l")) + limit = Integer.parseInt(cmd.getOptionValue("l")); + else + limit = 100; + + ResultScanner scanner = table.getScanner(scan); + + CINode node = new CINode(); + Result result = scanner.next(); + int count = 0; + while (result != null && count++ < limit) { + node = getCINode(result, node); + System.out.printf("%s:%s:%012d:%s\n", Bytes.toStringBinary(node.key), + Bytes.toStringBinary(node.prev), node.count, node.client); + result = scanner.next(); + } + scanner.close(); + table.close(); + + return 0; + } + } + + /** + * A stand alone program that deletes a single node. + */ + private static class Delete extends Configured implements Tool { + public int run(String[] args) throws Exception { + if (args.length != 1) { + System.out.println("Usage : " + Delete.class.getSimpleName() + " "); + return 0; + } + byte[] val = Bytes.toBytesBinary(args[0]); + + org.apache.hadoop.hbase.client.Delete delete + = new org.apache.hadoop.hbase.client.Delete(val); + + HTable table = new HTable(getConf(), getTableName(getConf())); + + table.delete(delete); + table.flushCommits(); + table.close(); + + System.out.println("Delete successful"); + return 0; + } + } + + /** + * A stand alone program that follows a linked list created by {@link Generator} and prints timing info. + */ + private static class Walker extends Configured implements Tool { + public int run(String[] args) throws IOException { + Options options = new Options(); + options.addOption("n", "num", true, "number of queries"); + options.addOption("s", "start", true, "key to start at, binary string"); + options.addOption("l", "logevery", true, "log every N queries"); + + GnuParser parser = new GnuParser(); + CommandLine cmd = null; + try { + cmd = parser.parse(options, args); + if (cmd.getArgs().length != 0) { + throw new ParseException("Command takes no arguments"); + } + } catch (ParseException e) { + System.err.println("Failed to parse command line " + e.getMessage()); + System.err.println(); + HelpFormatter formatter = new HelpFormatter(); + formatter.printHelp(getClass().getSimpleName(), options); + System.exit(-1); + } + + long maxQueries = Long.MAX_VALUE; + if (cmd.hasOption('n')) { + maxQueries = Long.parseLong(cmd.getOptionValue("n")); + } + Random rand = new Random(); + boolean isSpecificStart = cmd.hasOption('s'); + byte[] startKey = isSpecificStart ? Bytes.toBytesBinary(cmd.getOptionValue('s')) : null; + int logEvery = cmd.hasOption('l') ? Integer.parseInt(cmd.getOptionValue('l')) : 1; + + HTable table = new HTable(getConf(), getTableName(getConf())); + long numQueries = 0; + // If isSpecificStart is set, only walk one list from that particular node. + // Note that in case of circular (or P-shaped) list it will walk forever, as is + // the case in normal run without startKey. + while (numQueries < maxQueries && (numQueries == 0 || !isSpecificStart)) { + if (!isSpecificStart) { + startKey = new byte[ROWKEY_LENGTH]; + rand.nextBytes(startKey); + } + CINode node = findStartNode(table, startKey); + if (node == null && isSpecificStart) { + System.err.printf("Start node not found: %s \n", Bytes.toStringBinary(startKey)); + } + numQueries++; + while (node != null && node.prev.length != NO_KEY.length && numQueries < maxQueries) { + byte[] prev = node.prev; + long t1 = System.currentTimeMillis(); + node = getNode(prev, table, node); + long t2 = System.currentTimeMillis(); + if (numQueries % logEvery == 0) { + System.out.printf("CQ %d: %d %s \n", numQueries, t2 - t1, Bytes.toStringBinary(prev)); + } + numQueries++; + if (node == null) { + System.err.printf("UNDEFINED NODE %s \n", Bytes.toStringBinary(prev)); + } else if (node.prev.length == NO_KEY.length) { + System.err.printf("TERMINATING NODE %s \n", Bytes.toStringBinary(node.key)); + } + } + } + + table.close(); + return 0; + } + + private static CINode findStartNode(HTable table, byte[] startKey) throws IOException { + Scan scan = new Scan(); + scan.setStartRow(startKey); + scan.setBatch(1); + scan.addColumn(FAMILY_NAME, COLUMN_PREV); + + long t1 = System.currentTimeMillis(); + ResultScanner scanner = table.getScanner(scan); + Result result = scanner.next(); + long t2 = System.currentTimeMillis(); + scanner.close(); + + if ( result != null) { + CINode node = getCINode(result, new CINode()); + System.out.printf("FSR %d %s\n", t2 - t1, Bytes.toStringBinary(node.key)); + return node; + } + + System.out.println("FSR " + (t2 - t1)); + + return null; + } + + private CINode getNode(byte[] row, HTable table, CINode node) throws IOException { + Get get = new Get(row); + get.addColumn(FAMILY_NAME, COLUMN_PREV); + Result result = table.get(get); + return getCINode(result, node); + } + } + + private static byte[] getTableName(Configuration conf) { + return Bytes.toBytes(conf.get(TABLE_NAME_KEY, DEFAULT_TABLE_NAME)); + } + + private static CINode getCINode(Result result, CINode node) { + node.key = new byte[result.getRow().length]; + System.arraycopy(result.getRow(), 0, node.key, 0, node.key.length); + if (result.containsColumn(FAMILY_NAME, COLUMN_PREV)) { + byte[] value = result.getValue(FAMILY_NAME, COLUMN_PREV); + node.prev = new byte[value.length]; + System.arraycopy(value, 0, node.prev, 0, node.prev.length); + } else { + node.prev = NO_KEY; + } + if (result.containsColumn(FAMILY_NAME, COLUMN_COUNT)) { + node.count = Bytes.toLong(result.getValue(FAMILY_NAME, COLUMN_COUNT)); + } else { + node.count = -1; + } + if (result.containsColumn(FAMILY_NAME, COLUMN_CLIENT)) { + node.client = Bytes.toString(result.getValue(FAMILY_NAME, COLUMN_CLIENT)); + } else { + node.client = ""; + } + return node; + } + + private IntegrationTestingUtility util; + + @Before + public void setUp() throws Exception { + util = getTestingUtil(); + util.initializeCluster(3); + this.setConf(util.getConfiguration()); + } + + @After + public void tearDown() throws Exception { + util.restoreCluster(); + } + + @Test + public void testContinuousIngest() throws IOException, Exception { + //Loop + int ret = ToolRunner.run(getTestingUtil().getConfiguration(), new Loop(), + new String[] {"1", "1", "2000000", + getTestDir("IntegrationTestBigLinkedList", "testContinuousIngest").toString(), "1"}); + org.junit.Assert.assertEquals(0, ret); + } + + public Path getTestDir(String testName, String subdir) throws IOException { + //HBaseTestingUtility.getDataTestDirOnTestFs() has not been backported. + FileSystem fs = FileSystem.get(getConf()); + Path base = new Path(fs.getWorkingDirectory(), "test-data"); + String randomStr = UUID.randomUUID().toString(); + Path testDir = new Path(base, randomStr); + fs.deleteOnExit(testDir); + + return new Path(new Path(testDir, testName), subdir); + } + + private IntegrationTestingUtility getTestingUtil() { + if (this.util == null) { + if (getConf() == null) { + this.util = new IntegrationTestingUtility(); + } else { + this.util = new IntegrationTestingUtility(getConf()); + } + } + return util; + } + + private int printUsage() { + System.err.println("Usage: " + this.getClass().getSimpleName() + " COMMAND [COMMAND options]"); + System.err.println(" where COMMAND is one of:"); + System.err.println(""); + System.err.println(" Generator A map only job that generates data."); + System.err.println(" Verify A map reduce job that looks for holes"); + System.err.println(" Look at the counts after running"); + System.err.println(" REFERENCED and UNREFERENCED are ok"); + System.err.println(" any UNDEFINED counts are bad. Do not"); + System.err.println(" run at the same time as the Generator."); + System.err.println(" Walker A standalong program that starts "); + System.err.println(" following a linked list and emits"); + System.err.println(" timing info."); + System.err.println(" Print A standalone program that prints nodes"); + System.err.println(" in the linked list."); + System.err.println(" Delete A standalone program that deletes a·"); + System.err.println(" single node."); + System.err.println(" Loop A program to Loop through Generator and"); + System.err.println(" Verify steps"); + System.err.println("\t "); + return 1; + } + + @Override + public int run(String[] args) throws Exception { + //get the class, run with the conf + if (args.length < 1) { + return printUsage(); + } + Tool tool = null; + if (args[0].equals("Generator")) { + tool = new Generator(); + } else if (args[0].equals("Verify")) { + tool = new Verify(); + } else if (args[0].equals("Loop")) { + tool = new Loop(); + } else if (args[0].equals("Walker")) { + tool = new Walker(); + } else if (args[0].equals("Print")) { + tool = new Print(); + } else if (args[0].equals("Delete")) { + tool = new Delete(); + } else { + return printUsage(); + } + + args = Arrays.copyOfRange(args, 1, args.length); + return ToolRunner.run(getConf(), tool, args); + } + + public static void main(String[] args) throws Exception { + int ret = ToolRunner.run(HBaseConfiguration.create(), new IntegrationTestBigLinkedList(), args); + System.exit(ret); + } + + private static void setJobConf(Job job, int numMappers, long numNodes, + Integer width, Integer wrapMuplitplier) { + job.getConfiguration().setInt(GENERATOR_NUM_MAPPERS_KEY, numMappers); + job.getConfiguration().setLong(GENERATOR_NUM_ROWS_PER_MAP_KEY, numNodes); + if (width != null) { + job.getConfiguration().setInt(GENERATOR_WIDTH_KEY, width.intValue()); + } + if (wrapMuplitplier != null) { + job.getConfiguration().setInt(GENERATOR_WRAP_KEY, wrapMuplitplier.intValue()); + } + } + + private static void setJobScannerConf(Job job) { + // Make sure scanners log something useful to make debugging possible. + job.getConfiguration().setBoolean(ScannerCallable.LOG_SCANNER_ACTIVITY, true); + job.getConfiguration().setInt(TableRecordReaderImpl.LOG_PER_ROW_COUNT, 100000); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java b/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java new file mode 100644 index 000000000000..3bf7f24ed4d8 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java @@ -0,0 +1,460 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.Random; +import java.util.UUID; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.conf.Configured; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.IntegrationTestingUtility; +import org.apache.hadoop.hbase.IntegrationTests; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.io.ImmutableBytesWritable; +import org.apache.hadoop.hbase.mapreduce.NMapInputFormat; +import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; +import org.apache.hadoop.hbase.mapreduce.TableMapper; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.io.BytesWritable; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.mapreduce.Counter; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapreduce.Mapper; +import org.apache.hadoop.mapreduce.Reducer; +import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import com.google.common.collect.Lists; + +/** + * A large test which loads a lot of data that has internal references, and + * verifies the data. + * + * In load step, 200 map tasks are launched, which in turn write loadmapper.num_to_write + * (default 100K) rows to an hbase table. Rows are written in blocks, for a total of + * 100 blocks. Each row in a block, contains loadmapper.backrefs (default 50) references + * to random rows in the prev block. + * + * Verify step is scans the table, and verifies that for every referenced row, the row is + * actually there (no data loss). Failed rows are output from reduce to be saved in the + * job output dir in hdfs and inspected later. + * + * This class can be run as a unit test, as an integration test, or from the command line + * + * Originally taken from Apache Bigtop. + */ +@Category(IntegrationTests.class) +public class IntegrationTestLoadAndVerify extends Configured implements Tool { + private static final String TEST_NAME = "IntegrationTestLoadAndVerify"; + private static final byte[] TEST_FAMILY = Bytes.toBytes("f1"); + private static final byte[] TEST_QUALIFIER = Bytes.toBytes("q1"); + + private static final String NUM_TO_WRITE_KEY = + "loadmapper.num_to_write"; + private static final long NUM_TO_WRITE_DEFAULT = 100*1000; + + private static final String TABLE_NAME_KEY = "loadmapper.table"; + private static final String TABLE_NAME_DEFAULT = "table"; + + private static final String NUM_BACKREFS_KEY = "loadmapper.backrefs"; + private static final int NUM_BACKREFS_DEFAULT = 50; + + private static final String NUM_MAP_TASKS_KEY = "loadmapper.map.tasks"; + private static final String NUM_REDUCE_TASKS_KEY = "verify.reduce.tasks"; + private static final int NUM_MAP_TASKS_DEFAULT = 200; + private static final int NUM_REDUCE_TASKS_DEFAULT = 35; + + private static final int SCANNER_CACHING = 500; + + private IntegrationTestingUtility util; + + private enum Counters { + ROWS_WRITTEN, + REFERENCES_WRITTEN, + REFERENCES_CHECKED; + } + + @Before + public void setUp() throws Exception { + util = getTestingUtil(); + util.initializeCluster(3); + this.setConf(util.getConfiguration()); + getConf().setLong(NUM_TO_WRITE_KEY, NUM_TO_WRITE_DEFAULT / 100); + getConf().setInt(NUM_MAP_TASKS_KEY, NUM_MAP_TASKS_DEFAULT / 100); + getConf().setInt(NUM_REDUCE_TASKS_KEY, NUM_REDUCE_TASKS_DEFAULT / 10); + } + + @After + public void tearDown() throws Exception { + util.restoreCluster(); + } + + /** + * Converts a "long" value between endian systems. + * Borrowed from Apache Commons IO + * @param value value to convert + * @return the converted value + */ + public static long swapLong(long value) + { + return + ( ( ( value >> 0 ) & 0xff ) << 56 ) + + ( ( ( value >> 8 ) & 0xff ) << 48 ) + + ( ( ( value >> 16 ) & 0xff ) << 40 ) + + ( ( ( value >> 24 ) & 0xff ) << 32 ) + + ( ( ( value >> 32 ) & 0xff ) << 24 ) + + ( ( ( value >> 40 ) & 0xff ) << 16 ) + + ( ( ( value >> 48 ) & 0xff ) << 8 ) + + ( ( ( value >> 56 ) & 0xff ) << 0 ); + } + + public static class LoadMapper + extends Mapper + { + private long recordsToWrite; + private HTable table; + private Configuration conf; + private int numBackReferencesPerRow; + private String shortTaskId; + + private Random rand = new Random(); + + private Counter rowsWritten, refsWritten; + + @Override + public void setup(Context context) throws IOException { + conf = context.getConfiguration(); + recordsToWrite = conf.getLong(NUM_TO_WRITE_KEY, NUM_TO_WRITE_DEFAULT); + String tableName = conf.get(TABLE_NAME_KEY, TABLE_NAME_DEFAULT); + numBackReferencesPerRow = conf.getInt(NUM_BACKREFS_KEY, NUM_BACKREFS_DEFAULT); + table = new HTable(conf, tableName); + table.setWriteBufferSize(4*1024*1024); + table.setAutoFlush(false); + + String taskId = conf.get("mapred.task.id"); + Matcher matcher = Pattern.compile(".+_m_(\\d+_\\d+)").matcher(taskId); + if (!matcher.matches()) { + throw new RuntimeException("Strange task ID: " + taskId); + } + shortTaskId = matcher.group(1); + + rowsWritten = context.getCounter(Counters.ROWS_WRITTEN); + refsWritten = context.getCounter(Counters.REFERENCES_WRITTEN); + } + + @Override + public void cleanup(Context context) throws IOException { + table.flushCommits(); + table.close(); + } + + @Override + protected void map(NullWritable key, NullWritable value, + Context context) throws IOException, InterruptedException { + + String suffix = "/" + shortTaskId; + byte[] row = Bytes.add(new byte[8], Bytes.toBytes(suffix)); + + int BLOCK_SIZE = (int)(recordsToWrite / 100); + + for (long i = 0; i < recordsToWrite;) { + long blockStart = i; + for (long idxInBlock = 0; + idxInBlock < BLOCK_SIZE && i < recordsToWrite; + idxInBlock++, i++) { + + long byteSwapped = swapLong(i); + Bytes.putLong(row, 0, byteSwapped); + + Put p = new Put(row); + p.add(TEST_FAMILY, TEST_QUALIFIER, HConstants.EMPTY_BYTE_ARRAY); + if (blockStart > 0) { + for (int j = 0; j < numBackReferencesPerRow; j++) { + long referredRow = blockStart - BLOCK_SIZE + rand.nextInt(BLOCK_SIZE); + Bytes.putLong(row, 0, swapLong(referredRow)); + p.add(TEST_FAMILY, row, HConstants.EMPTY_BYTE_ARRAY); + } + refsWritten.increment(1); + } + rowsWritten.increment(1); + table.put(p); + + if (i % 100 == 0) { + context.setStatus("Written " + i + "/" + recordsToWrite + " records"); + context.progress(); + } + } + // End of block, flush all of them before we start writing anything + // pointing to these! + table.flushCommits(); + } + } + } + + public static class VerifyMapper extends TableMapper { + static final BytesWritable EMPTY = new BytesWritable(HConstants.EMPTY_BYTE_ARRAY); + + @Override + protected void map(ImmutableBytesWritable key, Result value, Context context) + throws IOException, InterruptedException { + BytesWritable bwKey = new BytesWritable(key.get()); + BytesWritable bwVal = new BytesWritable(); + for (KeyValue kv : value.list()) { + if (Bytes.compareTo(TEST_QUALIFIER, 0, TEST_QUALIFIER.length, + kv.getBuffer(), kv.getQualifierOffset(), kv.getQualifierLength()) == 0) { + context.write(bwKey, EMPTY); + } else { + bwVal.set(kv.getBuffer(), kv.getQualifierOffset(), kv.getQualifierLength()); + context.write(bwVal, bwKey); + } + } + } + } + + public static class VerifyReducer extends Reducer { + private Counter refsChecked; + private Counter rowsWritten; + + @Override + public void setup(Context context) throws IOException { + refsChecked = context.getCounter(Counters.REFERENCES_CHECKED); + rowsWritten = context.getCounter(Counters.ROWS_WRITTEN); + } + + @Override + protected void reduce(BytesWritable referredRow, Iterable referrers, + VerifyReducer.Context ctx) throws IOException, InterruptedException { + boolean gotOriginalRow = false; + int refCount = 0; + + for (BytesWritable ref : referrers) { + if (ref.getLength() == 0) { + assert !gotOriginalRow; + gotOriginalRow = true; + } else { + refCount++; + } + } + refsChecked.increment(refCount); + + if (!gotOriginalRow) { + String parsedRow = makeRowReadable(referredRow.getBytes(), referredRow.getLength()); + String binRow = Bytes.toStringBinary(referredRow.getBytes(), 0, referredRow.getLength()); + ctx.write(new Text(binRow), new Text(parsedRow)); + rowsWritten.increment(1); + } + } + + private String makeRowReadable(byte[] bytes, int length) { + long rowIdx = swapLong(Bytes.toLong(bytes, 0)); + String suffix = Bytes.toString(bytes, 8, length - 8); + + return "Row #" + rowIdx + " suffix " + suffix; + } + } + + private void doLoad(Configuration conf, HTableDescriptor htd) throws Exception { + Path outputDir = getTestDir(TEST_NAME, "load-output"); + + NMapInputFormat.setNumMapTasks(conf, conf.getInt(NUM_MAP_TASKS_KEY, NUM_MAP_TASKS_DEFAULT)); + conf.set(TABLE_NAME_KEY, htd.getNameAsString()); + + Job job = new Job(conf); + job.setJobName(TEST_NAME + " Load for " + htd.getNameAsString()); + job.setJarByClass(this.getClass()); + job.setMapperClass(LoadMapper.class); + job.setInputFormatClass(NMapInputFormat.class); + job.setNumReduceTasks(0); + FileOutputFormat.setOutputPath(job, outputDir); + + TableMapReduceUtil.addDependencyJars(job); + TableMapReduceUtil.addDependencyJars( + job.getConfiguration(), HTable.class, Lists.class); + TableMapReduceUtil.initCredentials(job); + assertTrue(job.waitForCompletion(true)); + } + + private void doVerify(Configuration conf, HTableDescriptor htd) throws Exception { + Path outputDir = getTestDir(TEST_NAME, "verify-output"); + + Job job = new Job(conf); + job.setJarByClass(this.getClass()); + job.setJobName(TEST_NAME + " Verification for " + htd.getNameAsString()); + + Scan scan = new Scan(); + + TableMapReduceUtil.initTableMapperJob( + htd.getNameAsString(), scan, VerifyMapper.class, + BytesWritable.class, BytesWritable.class, job); + int scannerCaching = conf.getInt("verify.scannercaching", SCANNER_CACHING); + TableMapReduceUtil.setScannerCaching(job, scannerCaching); + + job.setReducerClass(VerifyReducer.class); + job.setNumReduceTasks(conf.getInt(NUM_REDUCE_TASKS_KEY, NUM_REDUCE_TASKS_DEFAULT)); + FileOutputFormat.setOutputPath(job, outputDir); + assertTrue(job.waitForCompletion(true)); + + long numOutputRecords = job.getCounters().findCounter(Counters.ROWS_WRITTEN).getValue(); + assertEquals(0, numOutputRecords); + } + + public Path getTestDir(String testName, String subdir) throws IOException { + //HBaseTestingUtility.getDataTestDirOnTestFs() has not been backported. + FileSystem fs = FileSystem.get(getConf()); + Path base = new Path(fs.getWorkingDirectory(), "test-data"); + String randomStr = UUID.randomUUID().toString(); + Path testDir = new Path(base, randomStr); + fs.deleteOnExit(testDir); + + return new Path(new Path(testDir, testName), subdir); + } + + @Test + public void testLoadAndVerify() throws Exception { + HTableDescriptor htd = new HTableDescriptor(TEST_NAME); + htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); + + HBaseAdmin admin = getTestingUtil().getHBaseAdmin(); + int numPreCreate = 40; + admin.createTable(htd, Bytes.toBytes(0L), Bytes.toBytes(-1L), numPreCreate); + + doLoad(getConf(), htd); + doVerify(getConf(), htd); + + // Only disable and drop if we succeeded to verify - otherwise it's useful + // to leave it around for post-mortem + deleteTable(admin, htd); + } + + private void deleteTable(HBaseAdmin admin, HTableDescriptor htd) + throws IOException, InterruptedException { + // Use disableTestAsync because disable can take a long time to complete + System.out.print("Disabling table " + htd.getNameAsString() +" "); + admin.disableTableAsync(htd.getName()); + + long start = System.currentTimeMillis(); + // NOTE tables can be both admin.isTableEnabled=false and + // isTableDisabled=false, when disabling must use isTableDisabled! + while (!admin.isTableDisabled(htd.getName())) { + System.out.print("."); + Thread.sleep(1000); + } + long delta = System.currentTimeMillis() - start; + System.out.println(" " + delta +" ms"); + System.out.println("Deleting table " + htd.getNameAsString() +" "); + admin.deleteTable(htd.getName()); + } + + public void usage() { + System.err.println(this.getClass().getSimpleName() + " [-Doptions] "); + System.err.println(" Loads a table with row dependencies and verifies the dependency chains"); + System.err.println("Options"); + System.err.println(" -Dloadmapper.table= Table to write/verify (default autogen)"); + System.err.println(" -Dloadmapper.backrefs= Number of backreferences per row (default 50)"); + System.err.println(" -Dloadmapper.num_to_write= Number of rows per mapper (default 100,000 per mapper)"); + System.err.println(" -Dloadmapper.deleteAfter= Delete after a successful verify (default true)"); + System.err.println(" -Dloadmapper.numPresplits= Number of presplit regions to start with (default 40)"); + System.err.println(" -Dloadmapper.map.tasks= Number of map tasks for load (default 200)"); + System.err.println(" -Dverify.reduce.tasks= Number of reduce tasks for verify (default 35)"); + System.err.println(" -Dverify.scannercaching= Number hbase scanner caching rows to read (default 50)"); + } + + public int run(String argv[]) throws Exception { + if (argv.length < 1 || argv.length > 1) { + usage(); + return 1; + } + + IntegrationTestingUtility.setUseDistributedCluster(getConf()); + boolean doLoad = false; + boolean doVerify = false; + boolean doDelete = getConf().getBoolean("loadmapper.deleteAfter",true); + int numPresplits = getConf().getInt("loadmapper.numPresplits", 40); + + if (argv[0].equals("load")) { + doLoad = true; + } else if (argv[0].equals("verify")) { + doVerify= true; + } else if (argv[0].equals("loadAndVerify")) { + doLoad=true; + doVerify= true; + } else { + System.err.println("Invalid argument " + argv[0]); + usage(); + return 1; + } + + // create HTableDescriptor for specified table + String table = getConf().get(TABLE_NAME_KEY, TEST_NAME); + HTableDescriptor htd = new HTableDescriptor(table); + htd.addFamily(new HColumnDescriptor(TEST_FAMILY)); + + HBaseAdmin admin = new HBaseAdmin(getConf()); + if (doLoad) { + admin.createTable(htd, Bytes.toBytes(0L), Bytes.toBytes(-1L), numPresplits); + doLoad(getConf(), htd); + } + if (doVerify) { + doVerify(getConf(), htd); + if (doDelete) { + deleteTable(admin, htd); + } + } + return 0; + } + + private IntegrationTestingUtility getTestingUtil() { + if (this.util == null) { + if (getConf() == null) { + this.util = new IntegrationTestingUtility(); + } else { + this.util = new IntegrationTestingUtility(getConf()); + } + } + return util; + } + + public static void main(String argv[]) throws Exception { + Configuration conf = HBaseConfiguration.create(); + int ret = ToolRunner.run(conf, new IntegrationTestLoadAndVerify(), argv); + System.exit(ret); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/thrift/TestCallQueue.java b/src/test/java/org/apache/hadoop/hbase/thrift/TestCallQueue.java index 671991e45935..1965b0508bae 100644 --- a/src/test/java/org/apache/hadoop/hbase/thrift/TestCallQueue.java +++ b/src/test/java/org/apache/hadoop/hbase/thrift/TestCallQueue.java @@ -76,7 +76,7 @@ public TestCallQueue(int elementsAdded, int elementsRemoved) { " elementsRemoved:" + elementsRemoved); } - @Test(timeout=3000) + @Test(timeout=60000) public void testPutTake() throws Exception { ThriftMetrics metrics = createMetrics(); CallQueue callQueue = new CallQueue( @@ -90,7 +90,7 @@ public void testPutTake() throws Exception { verifyMetrics(metrics, "timeInQueue_num_ops", elementsRemoved); } - @Test(timeout=3000) + @Test(timeout=60000) public void testOfferPoll() throws Exception { ThriftMetrics metrics = createMetrics(); CallQueue callQueue = new CallQueue( diff --git a/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java b/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java index 8357e29641a9..e3477a3a0a43 100644 --- a/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java +++ b/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -28,18 +27,23 @@ import java.util.ArrayList; import java.util.List; import java.util.Map; -import java.util.Collection; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.filter.ParseFilter; -import org.junit.experimental.categories.Category; -import org.junit.Test; -import org.junit.BeforeClass; import org.apache.hadoop.hbase.thrift.generated.BatchMutation; import org.apache.hadoop.hbase.thrift.generated.ColumnDescriptor; import org.apache.hadoop.hbase.thrift.generated.Hbase; import org.apache.hadoop.hbase.thrift.generated.Mutation; import org.apache.hadoop.hbase.thrift.generated.TCell; +import org.apache.hadoop.hbase.thrift.generated.TRegionInfo; import org.apache.hadoop.hbase.thrift.generated.TRowResult; +import org.apache.hadoop.hbase.thrift.generated.TIncrement; +import org.apache.hadoop.hbase.thrift.ThriftServerRunner.HBaseHandler; +import org.apache.hadoop.hbase.thrift.generated.TScan; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.metrics.ContextFactory; import org.apache.hadoop.metrics.MetricsContext; @@ -47,9 +51,9 @@ import org.apache.hadoop.metrics.spi.NoEmitMetricsContext; import org.apache.hadoop.metrics.spi.OutputRecord; import org.junit.AfterClass; -import org.apache.hadoop.hbase.MediumTests; -import org.apache.hadoop.hbase.HBaseTestingUtility; -import org.apache.hadoop.conf.Configuration; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; /** * Unit testing for ThriftServerRunner.HBaseHandler, a part of the @@ -58,16 +62,21 @@ @Category(MediumTests.class) public class TestThriftServer { private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); + private static final Log LOG = LogFactory.getLog(TestThriftServer.class); protected static final int MAXVERSIONS = 3; private static ByteBuffer asByteBuffer(String i) { return ByteBuffer.wrap(Bytes.toBytes(i)); } + private static ByteBuffer asByteBuffer(long l) { + return ByteBuffer.wrap(Bytes.toBytes(l)); + } // Static names for tables, columns, rows, and values private static ByteBuffer tableAname = asByteBuffer("tableA"); private static ByteBuffer tableBname = asByteBuffer("tableB"); private static ByteBuffer columnAname = asByteBuffer("columnA:"); + private static ByteBuffer columnAAname = asByteBuffer("columnA:A"); private static ByteBuffer columnBname = asByteBuffer("columnB:"); private static ByteBuffer rowAname = asByteBuffer("rowA"); private static ByteBuffer rowBname = asByteBuffer("rowB"); @@ -75,9 +84,11 @@ private static ByteBuffer asByteBuffer(String i) { private static ByteBuffer valueBname = asByteBuffer("valueB"); private static ByteBuffer valueCname = asByteBuffer("valueC"); private static ByteBuffer valueDname = asByteBuffer("valueD"); + private static ByteBuffer valueEname = asByteBuffer(100l); @BeforeClass public static void beforeClass() throws Exception { + UTIL.getConfiguration().setBoolean(ThriftServerRunner.COALESCE_INC_KEY, true); UTIL.startMiniCluster(); } @@ -116,6 +127,9 @@ public void testAll() throws Exception { public void doTestTableCreateDrop() throws Exception { ThriftServerRunner.HBaseHandler handler = new ThriftServerRunner.HBaseHandler(UTIL.getConfiguration()); + doTestTableCreateDrop(handler); + } + public static void doTestTableCreateDrop(Hbase.Iface handler) throws Exception { createTestTables(handler); dropTestTables(handler); } @@ -155,13 +169,13 @@ private static void setupMetricsContext() throws IOException { } private static void verifyMetrics(ThriftMetrics metrics, String name, int expectValue) - throws Exception { - MetricsContext context = MetricsUtil.getContext( - ThriftMetrics.CONTEXT_NAME); - metrics.doUpdates(context); - OutputRecord record = context.getAllRecords().get( - ThriftMetrics.CONTEXT_NAME).iterator().next(); - assertEquals(expectValue, record.getMetric(name).intValue()); + throws Exception { + MetricsContext context = MetricsUtil.getContext( + ThriftMetrics.CONTEXT_NAME); + metrics.doUpdates(context); + OutputRecord record = context.getAllRecords().get( + ThriftMetrics.CONTEXT_NAME).iterator().next(); + assertEquals(expectValue, record.getMetric(name).intValue()); } public static void createTestTables(Hbase.Iface handler) throws Exception { @@ -189,6 +203,44 @@ public static void dropTestTables(Hbase.Iface handler) throws Exception { handler.deleteTable(tableAname); } + public void doTestIncrements() throws Exception { + ThriftServerRunner.HBaseHandler handler = + new ThriftServerRunner.HBaseHandler(UTIL.getConfiguration()); + createTestTables(handler); + doTestIncrements(handler); + dropTestTables(handler); + } + + public static void doTestIncrements(HBaseHandler handler) throws Exception { + List mutations = new ArrayList(1); + mutations.add(new Mutation(false, columnAAname, valueEname, true)); + mutations.add(new Mutation(false, columnAname, valueEname, true)); + handler.mutateRow(tableAname, rowAname, mutations, null); + handler.mutateRow(tableAname, rowBname, mutations, null); + + List increments = new ArrayList(); + increments.add(new TIncrement(tableAname, rowBname, columnAAname, 7)); + increments.add(new TIncrement(tableAname, rowBname, columnAAname, 7)); + increments.add(new TIncrement(tableAname, rowBname, columnAAname, 7)); + + int numIncrements = 60000; + for (int i = 0; i < numIncrements; i++) { + handler.increment(new TIncrement(tableAname, rowAname, columnAname, 2)); + handler.incrementRows(increments); + } + + Thread.sleep(1000); + long lv = handler.get(tableAname, rowAname, columnAname, null).get(0).value.getLong(); + assertEquals((100 + (2 * numIncrements)), lv ); + + + lv = handler.get(tableAname, rowBname, columnAAname, null).get(0).value.getLong(); + assertEquals((100 + (3 * 7 * numIncrements)), lv); + + assertTrue(handler.coalescer.getSuccessfulCoalescings() > 0); + + } + /** * Tests adding a series of Mutations and BatchMutations, including a * delete mutation. Also tests data retrieval, and getting back multiple @@ -197,9 +249,13 @@ public static void dropTestTables(Hbase.Iface handler) throws Exception { * @throws Exception */ public void doTestTableMutations() throws Exception { - // Setup ThriftServerRunner.HBaseHandler handler = new ThriftServerRunner.HBaseHandler(UTIL.getConfiguration()); + doTestTableMutations(handler); + } + + public static void doTestTableMutations(Hbase.Iface handler) throws Exception { + // Setup handler.createTable(tableAname, getColumnDescriptors()); // Apply a few Mutations to rowA @@ -237,7 +293,7 @@ public void doTestTableMutations() throws Exception { TRowResult rowResult2 = handler.getRow(tableAname, rowBname, null).get(0); assertEquals(rowBname, rowResult2.row); assertEquals(valueCname, rowResult2.columns.get(columnAname).value); - assertEquals(valueDname, rowResult2.columns.get(columnBname).value); + assertEquals(valueDname, rowResult2.columns.get(columnBname).value); // Apply some deletes handler.deleteAll(tableAname, rowAname, columnBname, null); @@ -255,7 +311,7 @@ public void doTestTableMutations() throws Exception { handler.mutateRow(tableAname, rowAname, mutations, null); TRowResult rowResult3 = handler.getRow(tableAname, rowAname, null).get(0); assertEquals(rowAname, rowResult3.row); - assertEquals(0, rowResult3.columns.get(columnAname).value.array().length); + assertEquals(0, rowResult3.columns.get(columnAname).value.remaining()); // Teardown handler.disableTable(tableAname); @@ -406,6 +462,46 @@ public void doTestTableScanners() throws Exception { assertEquals(rowResult4a.columns.size(), 1); assertEquals(rowResult4a.columns.get(columnBname).value, valueBname); + // Test scanner using a TScan object once with sortColumns False and once with sortColumns true + TScan scanNoSortColumns = new TScan(); + scanNoSortColumns.setStartRow(rowAname); + scanNoSortColumns.setStopRow(rowBname); + + int scanner5 = handler.scannerOpenWithScan(tableAname , scanNoSortColumns, null); + TRowResult rowResult5 = handler.scannerGet(scanner5).get(0); + assertEquals(rowResult5.columns.size(), 1); + assertEquals(rowResult5.columns.get(columnBname).value, valueCname); + + TScan scanSortColumns = new TScan(); + scanSortColumns.setStartRow(rowAname); + scanSortColumns.setStopRow(rowBname); + scanSortColumns = scanSortColumns.setSortColumns(true); + + int scanner6 = handler.scannerOpenWithScan(tableAname ,scanSortColumns, null); + TRowResult rowResult6 = handler.scannerGet(scanner6).get(0); + assertEquals(rowResult6.sortedColumns.size(), 1); + assertEquals(rowResult6.sortedColumns.get(0).getCell().value, valueCname); + + List rowBmutations = new ArrayList(); + for (int i = 0; i < 20; i++) { + rowBmutations.add(new Mutation(false, asByteBuffer("columnA:" + i), valueCname, true)); + } + ByteBuffer rowC = asByteBuffer("rowC"); + handler.mutateRow(tableAname, rowC, rowBmutations, null); + + TScan scanSortMultiColumns = new TScan(); + scanSortMultiColumns.setStartRow(rowC); + scanSortMultiColumns = scanSortMultiColumns.setSortColumns(true); + int scanner7 = handler.scannerOpenWithScan(tableAname, scanSortMultiColumns, null); + TRowResult rowResult7 = handler.scannerGet(scanner7).get(0); + + ByteBuffer smallerColumn = asByteBuffer("columnA:"); + for (int i = 0; i < 20; i++) { + ByteBuffer currentColumn = rowResult7.sortedColumns.get(i).columnName; + assertTrue(Bytes.compareTo(smallerColumn.array(), currentColumn.array()) < 0); + smallerColumn = currentColumn; + } + // Teardown handler.disableTable(tableAname); handler.deleteTable(tableAname); @@ -420,26 +516,35 @@ public void doTestTableScanners() throws Exception { public void doTestGetTableRegions() throws Exception { ThriftServerRunner.HBaseHandler handler = new ThriftServerRunner.HBaseHandler(UTIL.getConfiguration()); + doTestGetTableRegions(handler); + } + + public static void doTestGetTableRegions(Hbase.Iface handler) + throws Exception { + assertEquals(handler.getTableNames().size(), 0); handler.createTable(tableAname, getColumnDescriptors()); - int regionCount = handler.getTableRegions(tableAname).size(); + assertEquals(handler.getTableNames().size(), 1); + List regions = handler.getTableRegions(tableAname); + int regionCount = regions.size(); assertEquals("empty table should have only 1 region, " + "but found " + regionCount, regionCount, 1); + LOG.info("Region found:" + regions.get(0)); handler.disableTable(tableAname); handler.deleteTable(tableAname); regionCount = handler.getTableRegions(tableAname).size(); assertEquals("non-existing table should have 0 region, " + "but found " + regionCount, regionCount, 0); } - + public void doTestFilterRegistration() throws Exception { Configuration conf = UTIL.getConfiguration(); - + conf.set("hbase.thrift.filters", "MyFilter:filterclass"); ThriftServerRunner.registerFilters(conf); - + Map registeredFilters = ParseFilter.getAllFilters(); - + assertEquals("filterclass", registeredFilters.get("MyFilter")); } @@ -482,7 +587,7 @@ private List getColumnList(boolean includeA, boolean includeB) { * @return a List of Mutations for a row, with columnA having valueA * and columnB having valueB */ - private List getMutations() { + private static List getMutations() { List mutations = new ArrayList(); mutations.add(new Mutation(false, columnAname, valueAname, true)); mutations.add(new Mutation(false, columnBname, valueBname, true)); @@ -497,7 +602,7 @@ private List getMutations() { * (rowB, columnA): place valueC * (rowB, columnB): place valueD */ - private List getBatchMutations() { + private static List getBatchMutations() { List batchMutations = new ArrayList(); // Mutations to rowA. You can't mix delete and put anymore. diff --git a/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java b/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java index 0e158fb5eca8..29857dddecea 100644 --- a/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java +++ b/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java @@ -59,7 +59,7 @@ @RunWith(Parameterized.class) public class TestThriftServerCmdLine { - public static final Log LOG = + public static final Log LOG = LogFactory.getLog(TestThriftServerCmdLine.class); private final ImplType implType; @@ -72,7 +72,7 @@ public class TestThriftServerCmdLine { private Thread cmdLineThread; private volatile Exception cmdLineException; - + private Exception clientSideException; private ThriftServer thriftServer; @@ -139,7 +139,7 @@ public void run() { cmdLineThread.start(); } - @Test(timeout=30 * 1000) + @Test(timeout=120 * 1000) public void testRunThriftServer() throws Exception { List args = new ArrayList(); if (implType != null) { @@ -164,7 +164,11 @@ public void testRunThriftServer() throws Exception { thriftServer = new ThriftServer(TEST_UTIL.getConfiguration()); startCmdLineThread(args.toArray(new String[0])); - Threads.sleepWithoutInterrupt(2000); + // wait up to 20s for the server to start + for (int i = 0; i < 200 + && (thriftServer.serverRunner == null || thriftServer.serverRunner.tserver == null); i++) { + Thread.sleep(100); + } Class expectedClass = implType != null ? implType.serverClass : TBoundedThreadPoolServer.class; @@ -194,22 +198,20 @@ private void talkToThriftServer() throws Exception { } sock.open(); - TProtocol prot; - if (specifyCompact) { - prot = new TCompactProtocol(transport); - } else { - prot = new TBinaryProtocol(transport); - } - Hbase.Client client = new Hbase.Client(prot); - List tableNames = client.getTableNames(); - if (tableNames.isEmpty()) { - TestThriftServer.createTestTables(client); - assertEquals(2, client.getTableNames().size()); - } else { - assertEquals(2, tableNames.size()); - assertEquals(2, client.getColumnDescriptors(tableNames.get(0)).size()); + try { + TProtocol prot; + if (specifyCompact) { + prot = new TCompactProtocol(transport); + } else { + prot = new TBinaryProtocol(transport); + } + Hbase.Client client = new Hbase.Client(prot); + TestThriftServer.doTestTableCreateDrop(client); + TestThriftServer.doTestGetTableRegions(client); + TestThriftServer.doTestTableMutations(client); + } finally { + sock.close(); } - sock.close(); } private void stopCmdLineThread() throws Exception { diff --git a/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java b/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java index b848ac732926..64a654b978bb 100644 --- a/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java +++ b/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java @@ -1,5 +1,4 @@ /* - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,19 +18,14 @@ */ package org.apache.hadoop.hbase.thrift2; -import static org.junit.Assert.assertArrayEquals; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertTrue; -import static org.junit.Assert.fail; - import java.io.IOException; import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.Collections; import java.util.Comparator; import java.util.List; +import java.util.Map; +import java.util.HashMap; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -41,6 +35,12 @@ import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.Delete; +import org.apache.hadoop.hbase.client.Durability; +import org.apache.hadoop.hbase.filter.ParseFilter; import org.apache.hadoop.hbase.thrift.ThriftMetrics; import org.apache.hadoop.hbase.thrift2.generated.TColumn; import org.apache.hadoop.hbase.thrift2.generated.TColumnIncrement; @@ -55,6 +55,9 @@ import org.apache.hadoop.hbase.thrift2.generated.TPut; import org.apache.hadoop.hbase.thrift2.generated.TResult; import org.apache.hadoop.hbase.thrift2.generated.TScan; +import org.apache.hadoop.hbase.thrift2.generated.TMutation; +import org.apache.hadoop.hbase.thrift2.generated.TRowMutations; +import org.apache.hadoop.hbase.thrift2.generated.TDurability; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.metrics.ContextFactory; import org.apache.hadoop.metrics.MetricsContext; @@ -68,8 +71,16 @@ import org.junit.Test; import org.junit.experimental.categories.Category; +import static org.apache.hadoop.hbase.thrift2.ThriftUtilities.getFromThrift; +import static org.apache.hadoop.hbase.thrift2.ThriftUtilities.putFromThrift; +import static org.apache.hadoop.hbase.thrift2.ThriftUtilities.scanFromThrift; +import static org.apache.hadoop.hbase.thrift2.ThriftUtilities.deleteFromThrift; +import static org.junit.Assert.*; +import static java.nio.ByteBuffer.wrap; + /** - * Unit testing for ThriftServer.HBaseHandler, a part of the org.apache.hadoop.hbase.thrift2 package. + * Unit testing for ThriftServer.HBaseHandler, a part of the org.apache.hadoop.hbase.thrift2 + * package. */ @Category(MediumTests.class) public class TestThriftHBaseServiceHandler { @@ -91,7 +102,8 @@ public class TestThriftHBaseServiceHandler { .setMaxVersions(2) }; - public void assertTColumnValuesEqual(List columnValuesA, List columnValuesB) { + public void assertTColumnValuesEqual(List columnValuesA, + List columnValuesB) { assertEquals(columnValuesA.size(), columnValuesB.size()); Comparator comparator = new Comparator() { @Override @@ -141,17 +153,15 @@ private ThriftHBaseServiceHandler createHandler() { public void testExists() throws TIOError, TException { ThriftHBaseServiceHandler handler = createHandler(); byte[] rowName = "testExists".getBytes(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); - TGet get = new TGet(ByteBuffer.wrap(rowName)); + TGet get = new TGet(wrap(rowName)); assertFalse(handler.exists(table, get)); List columnValues = new ArrayList(); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), ByteBuffer - .wrap(valueAname))); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyBname), ByteBuffer.wrap(qualifierBname), ByteBuffer - .wrap(valueBname))); - TPut put = new TPut(ByteBuffer.wrap(rowName), columnValues); + columnValues.add(new TColumnValue(wrap(familyAname), wrap(qualifierAname), wrap(valueAname))); + columnValues.add(new TColumnValue(wrap(familyBname), wrap(qualifierBname), wrap(valueBname))); + TPut put = new TPut(wrap(rowName), columnValues); put.setColumnValues(columnValues); handler.put(table, put); @@ -163,20 +173,18 @@ public void testExists() throws TIOError, TException { public void testPutGet() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); byte[] rowName = "testPutGet".getBytes(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); List columnValues = new ArrayList(); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), ByteBuffer - .wrap(valueAname))); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyBname), ByteBuffer.wrap(qualifierBname), ByteBuffer - .wrap(valueBname))); - TPut put = new TPut(ByteBuffer.wrap(rowName), columnValues); + columnValues.add(new TColumnValue(wrap(familyAname), wrap(qualifierAname), wrap(valueAname))); + columnValues.add(new TColumnValue(wrap(familyBname), wrap(qualifierBname), wrap(valueBname))); + TPut put = new TPut(wrap(rowName), columnValues); put.setColumnValues(columnValues); handler.put(table, put); - TGet get = new TGet(ByteBuffer.wrap(rowName)); + TGet get = new TGet(wrap(rowName)); TResult result = handler.get(table, get); assertArrayEquals(rowName, result.getRow()); @@ -187,24 +195,22 @@ public void testPutGet() throws Exception { @Test public void testPutGetMultiple() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); byte[] rowName1 = "testPutGetMultiple1".getBytes(); byte[] rowName2 = "testPutGetMultiple2".getBytes(); List columnValues = new ArrayList(); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), ByteBuffer - .wrap(valueAname))); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyBname), ByteBuffer.wrap(qualifierBname), ByteBuffer - .wrap(valueBname))); + columnValues.add(new TColumnValue(wrap(familyAname), wrap(qualifierAname), wrap(valueAname))); + columnValues.add(new TColumnValue(wrap(familyBname), wrap(qualifierBname), wrap(valueBname))); List puts = new ArrayList(); - puts.add(new TPut(ByteBuffer.wrap(rowName1), columnValues)); - puts.add(new TPut(ByteBuffer.wrap(rowName2), columnValues)); + puts.add(new TPut(wrap(rowName1), columnValues)); + puts.add(new TPut(wrap(rowName2), columnValues)); handler.putMultiple(table, puts); List gets = new ArrayList(); - gets.add(new TGet(ByteBuffer.wrap(rowName1))); - gets.add(new TGet(ByteBuffer.wrap(rowName2))); + gets.add(new TGet(wrap(rowName1))); + gets.add(new TGet(wrap(rowName2))); List results = handler.getMultiple(table, gets); assertEquals(2, results.size()); @@ -219,62 +225,60 @@ public void testPutGetMultiple() throws Exception { @Test public void testDeleteMultiple() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); byte[] rowName1 = "testDeleteMultiple1".getBytes(); byte[] rowName2 = "testDeleteMultiple2".getBytes(); List columnValues = new ArrayList(); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), ByteBuffer - .wrap(valueAname))); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyBname), ByteBuffer.wrap(qualifierBname), ByteBuffer - .wrap(valueBname))); + columnValues.add(new TColumnValue(wrap(familyAname), wrap(qualifierAname), wrap(valueAname))); + columnValues.add(new TColumnValue(wrap(familyBname), wrap(qualifierBname), wrap(valueBname))); List puts = new ArrayList(); - puts.add(new TPut(ByteBuffer.wrap(rowName1), columnValues)); - puts.add(new TPut(ByteBuffer.wrap(rowName2), columnValues)); + puts.add(new TPut(wrap(rowName1), columnValues)); + puts.add(new TPut(wrap(rowName2), columnValues)); handler.putMultiple(table, puts); List deletes = new ArrayList(); - deletes.add(new TDelete(ByteBuffer.wrap(rowName1))); - deletes.add(new TDelete(ByteBuffer.wrap(rowName2))); + deletes.add(new TDelete(wrap(rowName1))); + deletes.add(new TDelete(wrap(rowName2))); List deleteResults = handler.deleteMultiple(table, deletes); // 0 means they were all successfully applies assertEquals(0, deleteResults.size()); - assertFalse(handler.exists(table, new TGet(ByteBuffer.wrap(rowName1)))); - assertFalse(handler.exists(table, new TGet(ByteBuffer.wrap(rowName2)))); + assertFalse(handler.exists(table, new TGet(wrap(rowName1)))); + assertFalse(handler.exists(table, new TGet(wrap(rowName2)))); } @Test public void testDelete() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); byte[] rowName = "testDelete".getBytes(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); List columnValues = new ArrayList(); - TColumnValue columnValueA = new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), - ByteBuffer.wrap(valueAname)); - TColumnValue columnValueB = new TColumnValue(ByteBuffer.wrap(familyBname), ByteBuffer.wrap(qualifierBname), - ByteBuffer.wrap(valueBname)); + TColumnValue columnValueA = new TColumnValue(wrap(familyAname), wrap(qualifierAname), + wrap(valueAname)); + TColumnValue columnValueB = new TColumnValue(wrap(familyBname), wrap(qualifierBname), + wrap(valueBname)); columnValues.add(columnValueA); columnValues.add(columnValueB); - TPut put = new TPut(ByteBuffer.wrap(rowName), columnValues); + TPut put = new TPut(wrap(rowName), columnValues); put.setColumnValues(columnValues); handler.put(table, put); - TDelete delete = new TDelete(ByteBuffer.wrap(rowName)); + TDelete delete = new TDelete(wrap(rowName)); List deleteColumns = new ArrayList(); - TColumn deleteColumn = new TColumn(ByteBuffer.wrap(familyAname)); + TColumn deleteColumn = new TColumn(wrap(familyAname)); deleteColumn.setQualifier(qualifierAname); deleteColumns.add(deleteColumn); delete.setColumns(deleteColumns); handler.deleteSingle(table, delete); - TGet get = new TGet(ByteBuffer.wrap(rowName)); + TGet get = new TGet(wrap(rowName)); TResult result = handler.get(table, get); assertArrayEquals(rowName, result.getRow()); List returnedColumnValues = result.getColumnValues(); @@ -287,14 +291,14 @@ public void testDelete() throws Exception { public void testDeleteAllTimestamps() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); byte[] rowName = "testDeleteAllTimestamps".getBytes(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); List columnValues = new ArrayList(); - TColumnValue columnValueA = new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), - ByteBuffer.wrap(valueAname)); + TColumnValue columnValueA = new TColumnValue(wrap(familyAname), wrap(qualifierAname), + wrap(valueAname)); columnValueA.setTimestamp(System.currentTimeMillis() - 10); columnValues.add(columnValueA); - TPut put = new TPut(ByteBuffer.wrap(rowName), columnValues); + TPut put = new TPut(wrap(rowName), columnValues); put.setColumnValues(columnValues); @@ -302,14 +306,14 @@ public void testDeleteAllTimestamps() throws Exception { columnValueA.setTimestamp(System.currentTimeMillis()); handler.put(table, put); - TGet get = new TGet(ByteBuffer.wrap(rowName)); + TGet get = new TGet(wrap(rowName)); get.setMaxVersions(2); TResult result = handler.get(table, get); assertEquals(2, result.getColumnValuesSize()); - TDelete delete = new TDelete(ByteBuffer.wrap(rowName)); + TDelete delete = new TDelete(wrap(rowName)); List deleteColumns = new ArrayList(); - TColumn deleteColumn = new TColumn(ByteBuffer.wrap(familyAname)); + TColumn deleteColumn = new TColumn(wrap(familyAname)); deleteColumn.setQualifier(qualifierAname); deleteColumns.add(deleteColumn); delete.setColumns(deleteColumns); @@ -317,7 +321,7 @@ public void testDeleteAllTimestamps() throws Exception { handler.deleteSingle(table, delete); - get = new TGet(ByteBuffer.wrap(rowName)); + get = new TGet(wrap(rowName)); result = handler.get(table, get); assertNull(result.getRow()); assertEquals(0, result.getColumnValuesSize()); @@ -327,17 +331,17 @@ public void testDeleteAllTimestamps() throws Exception { public void testDeleteSingleTimestamp() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); byte[] rowName = "testDeleteSingleTimestamp".getBytes(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); long timestamp1 = System.currentTimeMillis() - 10; long timestamp2 = System.currentTimeMillis(); List columnValues = new ArrayList(); - TColumnValue columnValueA = new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), - ByteBuffer.wrap(valueAname)); + TColumnValue columnValueA = new TColumnValue(wrap(familyAname), wrap(qualifierAname), + wrap(valueAname)); columnValueA.setTimestamp(timestamp1); columnValues.add(columnValueA); - TPut put = new TPut(ByteBuffer.wrap(rowName), columnValues); + TPut put = new TPut(wrap(rowName), columnValues); put.setColumnValues(columnValues); @@ -345,14 +349,14 @@ public void testDeleteSingleTimestamp() throws Exception { columnValueA.setTimestamp(timestamp2); handler.put(table, put); - TGet get = new TGet(ByteBuffer.wrap(rowName)); + TGet get = new TGet(wrap(rowName)); get.setMaxVersions(2); TResult result = handler.get(table, get); assertEquals(2, result.getColumnValuesSize()); - TDelete delete = new TDelete(ByteBuffer.wrap(rowName)); + TDelete delete = new TDelete(wrap(rowName)); List deleteColumns = new ArrayList(); - TColumn deleteColumn = new TColumn(ByteBuffer.wrap(familyAname)); + TColumn deleteColumn = new TColumn(wrap(familyAname)); deleteColumn.setQualifier(qualifierAname); deleteColumns.add(deleteColumn); delete.setColumns(deleteColumns); @@ -360,7 +364,7 @@ public void testDeleteSingleTimestamp() throws Exception { handler.deleteSingle(table, delete); - get = new TGet(ByteBuffer.wrap(rowName)); + get = new TGet(wrap(rowName)); result = handler.get(table, get); assertArrayEquals(rowName, result.getRow()); assertEquals(1, result.getColumnValuesSize()); @@ -372,21 +376,21 @@ public void testDeleteSingleTimestamp() throws Exception { public void testIncrement() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); byte[] rowName = "testIncrement".getBytes(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); List columnValues = new ArrayList(); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), ByteBuffer - .wrap(Bytes.toBytes(1L)))); - TPut put = new TPut(ByteBuffer.wrap(rowName), columnValues); + columnValues.add(new TColumnValue(wrap(familyAname), wrap(qualifierAname), + wrap(Bytes.toBytes(1L)))); + TPut put = new TPut(wrap(rowName), columnValues); put.setColumnValues(columnValues); handler.put(table, put); List incrementColumns = new ArrayList(); - incrementColumns.add(new TColumnIncrement(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname))); - TIncrement increment = new TIncrement(ByteBuffer.wrap(rowName), incrementColumns); + incrementColumns.add(new TColumnIncrement(wrap(familyAname), wrap(qualifierAname))); + TIncrement increment = new TIncrement(wrap(rowName), incrementColumns); handler.increment(table, increment); - TGet get = new TGet(ByteBuffer.wrap(rowName)); + TGet get = new TGet(wrap(rowName)); TResult result = handler.get(table, get); assertArrayEquals(rowName, result.getRow()); @@ -396,8 +400,8 @@ public void testIncrement() throws Exception { } /** - * check that checkAndPut fails if the cell does not exist, then put in the cell, then check that the checkAndPut - * succeeds. + * check that checkAndPut fails if the cell does not exist, then put in the cell, then check + * that the checkAndPut succeeds. * * @throws Exception */ @@ -405,33 +409,33 @@ public void testIncrement() throws Exception { public void testCheckAndPut() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); byte[] rowName = "testCheckAndPut".getBytes(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); List columnValuesA = new ArrayList(); - TColumnValue columnValueA = new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), - ByteBuffer.wrap(valueAname)); + TColumnValue columnValueA = new TColumnValue(wrap(familyAname), wrap(qualifierAname), + wrap(valueAname)); columnValuesA.add(columnValueA); - TPut putA = new TPut(ByteBuffer.wrap(rowName), columnValuesA); + TPut putA = new TPut(wrap(rowName), columnValuesA); putA.setColumnValues(columnValuesA); List columnValuesB = new ArrayList(); - TColumnValue columnValueB = new TColumnValue(ByteBuffer.wrap(familyBname), ByteBuffer.wrap(qualifierBname), - ByteBuffer.wrap(valueBname)); + TColumnValue columnValueB = new TColumnValue(wrap(familyBname), wrap(qualifierBname), + wrap(valueBname)); columnValuesB.add(columnValueB); - TPut putB = new TPut(ByteBuffer.wrap(rowName), columnValuesB); + TPut putB = new TPut(wrap(rowName), columnValuesB); putB.setColumnValues(columnValuesB); - assertFalse(handler.checkAndPut(table, ByteBuffer.wrap(rowName), ByteBuffer.wrap(familyAname), - ByteBuffer.wrap(qualifierAname), ByteBuffer.wrap(valueAname), putB)); + assertFalse(handler.checkAndPut(table, wrap(rowName), wrap(familyAname), + wrap(qualifierAname), wrap(valueAname), putB)); - TGet get = new TGet(ByteBuffer.wrap(rowName)); + TGet get = new TGet(wrap(rowName)); TResult result = handler.get(table, get); assertEquals(0, result.getColumnValuesSize()); handler.put(table, putA); - assertTrue(handler.checkAndPut(table, ByteBuffer.wrap(rowName), ByteBuffer.wrap(familyAname), - ByteBuffer.wrap(qualifierAname), ByteBuffer.wrap(valueAname), putB)); + assertTrue(handler.checkAndPut(table, wrap(rowName), wrap(familyAname), + wrap(qualifierAname), wrap(valueAname), putB)); result = handler.get(table, get); assertArrayEquals(rowName, result.getRow()); @@ -443,8 +447,8 @@ public void testCheckAndPut() throws Exception { } /** - * check that checkAndDelete fails if the cell does not exist, then put in the cell, then check that the - * checkAndDelete succeeds. + * check that checkAndDelete fails if the cell does not exist, then put in the cell, then + * check that the checkAndDelete succeeds. * * @throws Exception */ @@ -452,39 +456,39 @@ public void testCheckAndPut() throws Exception { public void testCheckAndDelete() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); byte[] rowName = "testCheckAndDelete".getBytes(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); List columnValuesA = new ArrayList(); - TColumnValue columnValueA = new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), - ByteBuffer.wrap(valueAname)); + TColumnValue columnValueA = new TColumnValue(wrap(familyAname), wrap(qualifierAname), + wrap(valueAname)); columnValuesA.add(columnValueA); - TPut putA = new TPut(ByteBuffer.wrap(rowName), columnValuesA); + TPut putA = new TPut(wrap(rowName), columnValuesA); putA.setColumnValues(columnValuesA); List columnValuesB = new ArrayList(); - TColumnValue columnValueB = new TColumnValue(ByteBuffer.wrap(familyBname), ByteBuffer.wrap(qualifierBname), - ByteBuffer.wrap(valueBname)); + TColumnValue columnValueB = new TColumnValue(wrap(familyBname), wrap(qualifierBname), + wrap(valueBname)); columnValuesB.add(columnValueB); - TPut putB = new TPut(ByteBuffer.wrap(rowName), columnValuesB); + TPut putB = new TPut(wrap(rowName), columnValuesB); putB.setColumnValues(columnValuesB); // put putB so that we know whether the row has been deleted or not handler.put(table, putB); - TDelete delete = new TDelete(ByteBuffer.wrap(rowName)); + TDelete delete = new TDelete(wrap(rowName)); - assertFalse(handler.checkAndDelete(table, ByteBuffer.wrap(rowName), ByteBuffer.wrap(familyAname), - ByteBuffer.wrap(qualifierAname), ByteBuffer.wrap(valueAname), delete)); + assertFalse(handler.checkAndDelete(table, wrap(rowName), wrap(familyAname), + wrap(qualifierAname), wrap(valueAname), delete)); - TGet get = new TGet(ByteBuffer.wrap(rowName)); + TGet get = new TGet(wrap(rowName)); TResult result = handler.get(table, get); assertArrayEquals(rowName, result.getRow()); assertTColumnValuesEqual(columnValuesB, result.getColumnValues()); handler.put(table, putA); - assertTrue(handler.checkAndDelete(table, ByteBuffer.wrap(rowName), ByteBuffer.wrap(familyAname), - ByteBuffer.wrap(qualifierAname), ByteBuffer.wrap(valueAname), delete)); + assertTrue(handler.checkAndDelete(table, wrap(rowName), wrap(familyAname), + wrap(qualifierAname), wrap(valueAname), delete)); result = handler.get(table, get); assertFalse(result.isSetRow()); @@ -494,8 +498,19 @@ public void testCheckAndDelete() throws Exception { @Test public void testScan() throws Exception { ThriftHBaseServiceHandler handler = createHandler(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); + // insert data + TColumnValue columnValue = new TColumnValue(wrap(familyAname), wrap(qualifierAname), + wrap(valueAname)); + List columnValues = new ArrayList(); + columnValues.add(columnValue); + for (int i = 0; i < 10; i++) { + TPut put = new TPut(wrap(("testScan" + i).getBytes()), columnValues); + handler.put(table, put); + } + + // create scan instance TScan scan = new TScan(); List columns = new ArrayList(); TColumn column = new TColumn(); @@ -504,28 +519,75 @@ public void testScan() throws Exception { columns.add(column); scan.setColumns(columns); scan.setStartRow("testScan".getBytes()); + scan.setStopRow("testScan\uffff".getBytes()); + + // get scanner and rows + int scanId = handler.openScanner(table, scan); + List results = handler.getScannerRows(scanId, 10); + assertEquals(10, results.size()); + for (int i = 0; i < 10; i++) { + // check if the rows are returned and in order + assertArrayEquals(("testScan" + i).getBytes(), results.get(i).getRow()); + } - TColumnValue columnValue = new TColumnValue(ByteBuffer.wrap(familyAname), ByteBuffer.wrap(qualifierAname), - ByteBuffer.wrap(valueAname)); + // check that we are at the end of the scan + results = handler.getScannerRows(scanId, 10); + assertEquals(0, results.size()); + + // close scanner and check that it was indeed closed + handler.closeScanner(scanId); + try { + handler.getScannerRows(scanId, 10); + fail("Scanner id should be invalid"); + } catch (TIllegalArgument e) { + } + } + + @Test + public void testScanWithFilter() throws Exception { + ThriftHBaseServiceHandler handler = createHandler(); + ByteBuffer table = wrap(tableAname); + + // insert data + TColumnValue columnValue = new TColumnValue(wrap(familyAname), wrap(qualifierAname), + wrap(valueAname)); List columnValues = new ArrayList(); columnValues.add(columnValue); for (int i = 0; i < 10; i++) { - TPut put = new TPut(ByteBuffer.wrap(("testScan" + i).getBytes()), columnValues); + TPut put = new TPut(wrap(("testScanWithFilter" + i).getBytes()), columnValues); handler.put(table, put); } + // create scan instance with filter + TScan scan = new TScan(); + List columns = new ArrayList(); + TColumn column = new TColumn(); + column.setFamily(familyAname); + column.setQualifier(qualifierAname); + columns.add(column); + scan.setColumns(columns); + scan.setStartRow("testScanWithFilter".getBytes()); + scan.setStopRow("testScanWithFilter\uffff".getBytes()); + // only get the key part + scan.setFilterString(wrap(("KeyOnlyFilter()").getBytes())); + + // get scanner and rows int scanId = handler.openScanner(table, scan); List results = handler.getScannerRows(scanId, 10); assertEquals(10, results.size()); for (int i = 0; i < 10; i++) { - assertArrayEquals(("testScan" + i).getBytes(), results.get(i).getRow()); + // check if the rows are returned and in order + assertArrayEquals(("testScanWithFilter" + i).getBytes(), results.get(i).getRow()); + // check that the value is indeed stripped by the filter + assertEquals(0, results.get(i).getColumnValues().get(0).getValue().length); } + // check that we are at the end of the scan results = handler.getScannerRows(scanId, 10); assertEquals(0, results.size()); + // close scanner and check that it was indeed closed handler.closeScanner(scanId); - try { handler.getScannerRows(scanId, 10); fail("Scanner id should be invalid"); @@ -533,6 +595,143 @@ public void testScan() throws Exception { } } + /** + * Padding numbers to make comparison of sort order easier in a for loop + * + * @param n The number to pad. + * @param pad The length to pad up to. + * @return The padded number as a string. + */ + private String pad(int n, byte pad) { + String res = Integer.toString(n); + while (res.length() < pad) res = "0" + res; + return res; + } + + @Test + public void testScanWithBatchSize() throws Exception { + ThriftHBaseServiceHandler handler = createHandler(); + ByteBuffer table = wrap(tableAname); + + // insert data + List columnValues = new ArrayList(); + for (int i = 0; i < 100; i++) { + String colNum = pad(i, (byte) 3); + TColumnValue columnValue = new TColumnValue(wrap(familyAname), + wrap(("col" + colNum).getBytes()), wrap(("val" + colNum).getBytes())); + columnValues.add(columnValue); + } + TPut put = new TPut(wrap(("testScanWithBatchSize").getBytes()), columnValues); + handler.put(table, put); + + // create scan instance + TScan scan = new TScan(); + List columns = new ArrayList(); + TColumn column = new TColumn(); + column.setFamily(familyAname); + columns.add(column); + scan.setColumns(columns); + scan.setStartRow("testScanWithBatchSize".getBytes()); + scan.setStopRow("testScanWithBatchSize\uffff".getBytes()); + // set batch size to 10 columns per call + scan.setBatchSize(10); + + // get scanner + int scanId = handler.openScanner(table, scan); + List results = null; + for (int i = 0; i < 10; i++) { + // get batch for single row (10x10 is what we expect) + results = handler.getScannerRows(scanId, 1); + assertEquals(1, results.size()); + // check length of batch + List cols = results.get(0).getColumnValues(); + assertEquals(10, cols.size()); + // check if the columns are returned and in order + for (int y = 0; y < 10; y++) { + int colNum = y + (10 * i); + String colNumPad = pad(colNum, (byte) 3); + assertArrayEquals(("col" + colNumPad).getBytes(), cols.get(y).getQualifier()); + } + } + + // check that we are at the end of the scan + results = handler.getScannerRows(scanId, 1); + assertEquals(0, results.size()); + + // close scanner and check that it was indeed closed + handler.closeScanner(scanId); + try { + handler.getScannerRows(scanId, 1); + fail("Scanner id should be invalid"); + } catch (TIllegalArgument e) { + } + } + @Test + public void testGetScannerResults() throws Exception { + ThriftHBaseServiceHandler handler = createHandler(); + ByteBuffer table = wrap(tableAname); + + // insert data + TColumnValue columnValue = + new TColumnValue(wrap(familyAname), wrap(qualifierAname), wrap(valueAname)); + List columnValues = new ArrayList(); + columnValues.add(columnValue); + for (int i = 0; i < 20; i++) { + TPut put = + new TPut(wrap(("testGetScannerResults" + pad(i, (byte) 2)).getBytes()), columnValues); + handler.put(table, put); + } + + // create scan instance + TScan scan = new TScan(); + List columns = new ArrayList(); + TColumn column = new TColumn(); + column.setFamily(familyAname); + column.setQualifier(qualifierAname); + columns.add(column); + scan.setColumns(columns); + scan.setStartRow("testGetScannerResults".getBytes()); + + // get 5 rows and check the returned results + scan.setStopRow("testGetScannerResults05".getBytes()); + List results = handler.getScannerResults(table, scan, 5); + assertEquals(5, results.size()); + for (int i = 0; i < 5; i++) { + // check if the rows are returned and in order + assertArrayEquals(("testGetScannerResults" + pad(i, (byte) 2)).getBytes(), results.get(i) + .getRow()); + } + + // get 10 rows and check the returned results + scan.setStopRow("testGetScannerResults10".getBytes()); + results = handler.getScannerResults(table, scan, 10); + assertEquals(10, results.size()); + for (int i = 0; i < 10; i++) { + // check if the rows are returned and in order + assertArrayEquals(("testGetScannerResults" + pad(i, (byte) 2)).getBytes(), results.get(i) + .getRow()); + } + + // get 20 rows and check the returned results + scan.setStopRow("testGetScannerResults20".getBytes()); + results = handler.getScannerResults(table, scan, 20); + assertEquals(20, results.size()); + for (int i = 0; i < 20; i++) { + // check if the rows are returned and in order + assertArrayEquals(("testGetScannerResults" + pad(i, (byte) 2)).getBytes(), results.get(i) + .getRow()); + } + } + + @Test + public void testFilterRegistration() throws Exception { + Configuration conf = UTIL.getConfiguration(); + conf.set("hbase.thrift.filters", "MyFilter:filterclass"); + ThriftServer.registerFilters(conf); + Map registeredFilters = ParseFilter.getAllFilters(); + assertEquals("filterclass", registeredFilters.get("MyFilter")); + } + @Test public void testMetrics() throws Exception { Configuration conf = UTIL.getConfiguration(); @@ -540,19 +739,15 @@ public void testMetrics() throws Exception { THBaseService.Iface handler = ThriftHBaseServiceHandler.newInstance(conf, metrics); byte[] rowName = "testMetrics".getBytes(); - ByteBuffer table = ByteBuffer.wrap(tableAname); + ByteBuffer table = wrap(tableAname); - TGet get = new TGet(ByteBuffer.wrap(rowName)); + TGet get = new TGet(wrap(rowName)); assertFalse(handler.exists(table, get)); List columnValues = new ArrayList(); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyAname), - ByteBuffer.wrap(qualifierAname), - ByteBuffer.wrap(valueAname))); - columnValues.add(new TColumnValue(ByteBuffer.wrap(familyBname), - ByteBuffer.wrap(qualifierBname), - ByteBuffer.wrap(valueBname))); - TPut put = new TPut(ByteBuffer.wrap(rowName), columnValues); + columnValues.add(new TColumnValue(wrap(familyAname), wrap(qualifierAname), wrap(valueAname))); + columnValues.add(new TColumnValue(wrap(familyBname), wrap(qualifierBname), wrap(valueBname))); + TPut put = new TPut(wrap(rowName), columnValues); put.setColumnValues(columnValues); handler.put(table, put); @@ -562,7 +757,183 @@ public void testMetrics() throws Exception { verifyMetrics(metrics, "put_num_ops", 1); verifyMetrics(metrics, "exists_num_ops", 2); } - + + @Test + public void testAttribute() throws Exception { + byte[] rowName = "testAttribute".getBytes(); + byte[] attributeKey = "attribute1".getBytes(); + byte[] attributeValue = "value1".getBytes(); + Map attributes = new HashMap(); + attributes.put(wrap(attributeKey), wrap(attributeValue)); + + TGet tGet = new TGet(wrap(rowName)); + tGet.setAttributes(attributes); + Get get = getFromThrift(tGet); + assertArrayEquals(get.getAttribute("attribute1"), attributeValue); + + List columnValues = new ArrayList(); + columnValues.add(new TColumnValue(wrap(familyAname), wrap(qualifierAname), wrap(valueAname))); + TPut tPut = new TPut(wrap(rowName) , columnValues); + tPut.setAttributes(attributes); + Put put = putFromThrift(tPut); + assertArrayEquals(put.getAttribute("attribute1"), attributeValue); + + TScan tScan = new TScan(); + tScan.setAttributes(attributes); + Scan scan = scanFromThrift(tScan); + assertArrayEquals(scan.getAttribute("attribute1"), attributeValue); + + TDelete tDelete = new TDelete(wrap(rowName)); + tDelete.setAttributes(attributes); + Delete delete = deleteFromThrift(tDelete); + assertArrayEquals(delete.getAttribute("attribute1"), attributeValue); + } + + /** + * Put valueA to a row, make sure put has happened, then create a mutation object to put valueB + * and delete ValueA, then check that the row value is only valueB. + * + * @throws Exception + */ + @Test + public void testMutateRow() throws Exception { + ThriftHBaseServiceHandler handler = createHandler(); + byte[] rowName = "testMutateRow".getBytes(); + ByteBuffer table = wrap(tableAname); + + List columnValuesA = new ArrayList(); + TColumnValue columnValueA = new TColumnValue(wrap(familyAname), wrap(qualifierAname), + wrap(valueAname)); + columnValuesA.add(columnValueA); + TPut putA = new TPut(wrap(rowName), columnValuesA); + putA.setColumnValues(columnValuesA); + + handler.put(table,putA); + + TGet get = new TGet(wrap(rowName)); + TResult result = handler.get(table, get); + assertArrayEquals(rowName, result.getRow()); + List returnedColumnValues = result.getColumnValues(); + + List expectedColumnValues = new ArrayList(); + expectedColumnValues.add(columnValueA); + assertTColumnValuesEqual(expectedColumnValues, returnedColumnValues); + + List columnValuesB = new ArrayList(); + TColumnValue columnValueB = new TColumnValue(wrap(familyAname), wrap(qualifierBname), + wrap(valueBname)); + columnValuesB.add(columnValueB); + TPut putB = new TPut(wrap(rowName), columnValuesB); + putB.setColumnValues(columnValuesB); + + TDelete delete = new TDelete(wrap(rowName)); + List deleteColumns = new ArrayList(); + TColumn deleteColumn = new TColumn(wrap(familyAname)); + deleteColumn.setQualifier(qualifierAname); + deleteColumns.add(deleteColumn); + delete.setColumns(deleteColumns); + + List mutations = new ArrayList(); + TMutation mutationA = TMutation.put(putB); + mutations.add(mutationA); + + TMutation mutationB = TMutation.deleteSingle(delete); + mutations.add(mutationB); + + TRowMutations tRowMutations = new TRowMutations(wrap(rowName),mutations); + handler.mutateRow(table,tRowMutations); + + result = handler.get(table, get); + assertArrayEquals(rowName, result.getRow()); + returnedColumnValues = result.getColumnValues(); + + expectedColumnValues = new ArrayList(); + expectedColumnValues.add(columnValueB); + assertTColumnValuesEqual(expectedColumnValues, returnedColumnValues); + } + + /** + * Create TPut, TDelete , TIncrement objects, set durability then call ThriftUtility + * functions to get Put , Delete and Increment respectively. Use getDurability to make sure + * the returned objects have the appropriate durability setting. + * + * @throws Exception + */ + @Test + public void testDurability() throws Exception { + byte[] rowName = "testDurability".getBytes(); + List columnValues = new ArrayList(); + columnValues.add(new TColumnValue(wrap(familyAname), wrap(qualifierAname), wrap(valueAname))); + + List incrementColumns = new ArrayList(); + incrementColumns.add(new TColumnIncrement(wrap(familyAname), wrap(qualifierAname))); + + TDelete tDelete = new TDelete(wrap(rowName)); + + //if not setting writeToWal, check for default value + Delete delete = deleteFromThrift(tDelete); + assertEquals(delete.getDurability(), Durability.USE_DEFAULT); + + //if setting writeToWal to true, durability should be CF default + tDelete.setWriteToWal(true); + delete = deleteFromThrift(tDelete); + assertEquals(delete.getDurability(), Durability.USE_DEFAULT); + + //if setting writeToWal to false, durability should be SKIP_WAL + tDelete.setWriteToWal(false); + delete = deleteFromThrift(tDelete); + assertEquals(delete.getDurability(), Durability.SKIP_WAL); + + + tDelete.setDurability(TDurability.SKIP_WAL); + delete = deleteFromThrift(tDelete); + assertEquals(delete.getDurability(), Durability.SKIP_WAL); + + tDelete.setDurability(TDurability.ASYNC_WAL); + delete = deleteFromThrift(tDelete); + assertEquals(delete.getDurability(), Durability.ASYNC_WAL); + + tDelete.setDurability(TDurability.SYNC_WAL); + delete = deleteFromThrift(tDelete); + assertEquals(delete.getDurability(), Durability.SYNC_WAL); + + tDelete.setDurability(TDurability.FSYNC_WAL); + delete = deleteFromThrift(tDelete); + assertEquals(delete.getDurability(), Durability.FSYNC_WAL); + + TPut tPut = new TPut(wrap(rowName), columnValues); + + //if not setting writeToWal, check for default value + Put put = putFromThrift(tPut); + assertEquals(put.getDurability(), Durability.USE_DEFAULT); + + //if setting writeToWal to true, durability should be CF default + tPut.setWriteToWal(true); + put = putFromThrift(tPut); + assertEquals(put.getDurability(), Durability.USE_DEFAULT); + + //if setting writeToWal to false, durability should be SKIP_WAL + tPut.setWriteToWal(false); + put = putFromThrift(tPut); + assertEquals(put.getDurability(), Durability.SKIP_WAL); + + tPut.setDurability(TDurability.SKIP_WAL); + put = putFromThrift(tPut); + assertEquals(put.getDurability(), Durability.SKIP_WAL); + + tPut.setDurability(TDurability.ASYNC_WAL); + put = putFromThrift(tPut); + assertEquals(put.getDurability(), Durability.ASYNC_WAL); + + tPut.setDurability(TDurability.SYNC_WAL); + put = putFromThrift(tPut); + assertEquals(put.getDurability(), Durability.SYNC_WAL); + + tPut.setDurability(TDurability.FSYNC_WAL); + put = putFromThrift(tPut); + assertEquals(put.getDurability(), Durability.FSYNC_WAL); + } + private static ThriftMetrics getMetrics(Configuration conf) throws Exception { setupMetricsContext(); return new ThriftMetrics(Integer.parseInt(ThriftServer.DEFAULT_LISTEN_PORT), diff --git a/src/test/java/org/apache/hadoop/hbase/util/ChaosMonkey.java b/src/test/java/org/apache/hadoop/hbase/util/ChaosMonkey.java new file mode 100644 index 000000000000..8c6ebda0084e --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/ChaosMonkey.java @@ -0,0 +1,828 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.util; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.Queue; +import java.util.Random; + +import org.apache.commons.cli.CommandLine; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.ClusterStatus; +import org.apache.hadoop.hbase.HBaseCluster; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HServerLoad; +import org.apache.hadoop.hbase.IntegrationTestDataIngestWithChaosMonkey; +import org.apache.hadoop.hbase.IntegrationTestingUtility; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.Stoppable; +import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.util.ToolRunner; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; + +/** + * A utility to injects faults in a running cluster. + *

+ * ChaosMonkey defines Action's and Policy's. Actions are sequences of events, like + * - Select a random server to kill + * - Sleep for 5 sec + * - Start the server on the same host + * Actions can also be complex events, like rolling restart of all of the servers. + *

+ * Policies on the other hand are responsible for executing the actions based on a strategy. + * The default policy is to execute a random action every minute based on predefined action + * weights. ChaosMonkey executes predefined named policies until it is stopped. More than one + * policy can be active at any time. + *

+ * Chaos monkey can be run from the command line, or can be invoked from integration tests. + * See {@link IntegrationTestDataIngestWithChaosMonkey} or other integration tests that use + * chaos monkey for code examples. + *

+ * ChaosMonkey class is indeed inspired by the Netflix's same-named tool: + * http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html + */ +public class ChaosMonkey extends AbstractHBaseTool implements Stoppable { + + private static final Log LOG = LogFactory.getLog(ChaosMonkey.class); + + private static final long ONE_SEC = 1000; + private static final long FIVE_SEC = 5 * ONE_SEC; + private static final long ONE_MIN = 60 * ONE_SEC; + private static final long TIMEOUT = ONE_MIN; + + final IntegrationTestingUtility util; + + /** + * Construct a new ChaosMonkey + * @param util the HBaseIntegrationTestingUtility already configured + */ + public ChaosMonkey(IntegrationTestingUtility util) { + this.util = util; + } + + /** + * Construct a new ChaosMonkey + * @param util the HBaseIntegrationTestingUtility already configured + * @param policies names of pre-defined policies to use + */ + public ChaosMonkey(IntegrationTestingUtility util, String... policies) { + this.util = util; + setPoliciesByName(policies); + } + + /** + * Construct a new ChaosMonkey + * @param util the HBaseIntegrationTestingUtility already configured + * @param policies custom policies to use + */ + public ChaosMonkey(IntegrationTestingUtility util, Policy... policies) { + this.util = util; + this.policies = policies; + } + + private void setPoliciesByName(String... policies) { + this.policies = new Policy[policies.length]; + for (int i=0; i < policies.length; i++) { + this.policies[i] = NAMED_POLICIES.get(policies[i]); + } + } + + private void setPolicies(Policy... policies) { + this.policies = new Policy[policies.length]; + for (int i = 0; i < policies.length; i++) { + this.policies[i] = policies[i]; + } + } + + /** + * Context for Action's + */ + public static class ActionContext { + private IntegrationTestingUtility util; + + public ActionContext(IntegrationTestingUtility util) { + this.util = util; + } + + public IntegrationTestingUtility getHBaseIntegrationTestingUtility() { + return util; + } + + public HBaseCluster getHBaseCluster() { + return util.getHBaseClusterInterface(); + } + } + + /** + * A (possibly mischievous) action that the ChaosMonkey can perform. + */ + public static class Action { + // TODO: interesting question - should actions be implemented inside + // ChaosMonkey, or outside? If they are inside (initial), the class becomes + // huge and all-encompassing; if they are outside ChaosMonkey becomes just + // a random task scheduler. For now, keep inside. + + protected ActionContext context; + protected HBaseCluster cluster; + protected ClusterStatus initialStatus; + protected ServerName[] initialServers; + + public void init(ActionContext context) throws IOException { + this.context = context; + cluster = context.getHBaseCluster(); + initialStatus = cluster.getInitialClusterStatus(); + Collection regionServers = initialStatus.getServers(); + initialServers = regionServers.toArray(new ServerName[regionServers.size()]); + } + + public void perform() throws Exception { }; + + // TODO: perhaps these methods should be elsewhere? + /** Returns current region servers */ + protected ServerName[] getCurrentServers() throws IOException { + Collection regionServers = cluster.getClusterStatus().getServers(); + return regionServers.toArray(new ServerName[regionServers.size()]); + } + + protected void killMaster(ServerName server) throws IOException { + LOG.info("Killing master:" + server); + cluster.killMaster(server); + cluster.waitForMasterToStop(server, TIMEOUT); + LOG.info("Killed master server:" + server); + } + + protected void startMaster(ServerName server) throws IOException { + LOG.info("Starting master:" + server.getHostname()); + cluster.startMaster(server.getHostname()); + cluster.waitForActiveAndReadyMaster(TIMEOUT); + LOG.info("Started master: " + server); + } + + protected void killRs(ServerName server) throws IOException { + LOG.info("Killing region server:" + server); + cluster.killRegionServer(server); + cluster.waitForRegionServerToStop(server, TIMEOUT); + LOG.info("Killed region server:" + server + ". Reported num of rs:" + + cluster.getClusterStatus().getServersSize()); + } + + protected void startRs(ServerName server) throws IOException { + LOG.info("Starting region server:" + server.getHostname()); + cluster.startRegionServer(server.getHostname()); + cluster.waitForRegionServerToStart(server.getHostname(), TIMEOUT); + LOG.info("Started region server:" + server + ". Reported num of rs:" + + cluster.getClusterStatus().getServersSize()); + } + } + + private static class RestartActionBase extends Action { + long sleepTime; // how long should we sleep + + public RestartActionBase(long sleepTime) { + this.sleepTime = sleepTime; + } + + void sleep(long sleepTime) { + LOG.info("Sleeping for:" + sleepTime); + Threads.sleep(sleepTime); + } + + void restartMaster(ServerName server, long sleepTime) throws IOException { + sleepTime = Math.max(sleepTime, 1000); + killMaster(server); + sleep(sleepTime); + startMaster(server); + } + + void restartRs(ServerName server, long sleepTime) throws IOException { + sleepTime = Math.max(sleepTime, 1000); + killRs(server); + sleep(sleepTime); + startRs(server); + } + } + + public static class RestartActiveMaster extends RestartActionBase { + public RestartActiveMaster(long sleepTime) { + super(sleepTime); + } + @Override + public void perform() throws Exception { + LOG.info("Performing action: Restart active master"); + + ServerName master = cluster.getClusterStatus().getMaster(); + restartMaster(master, sleepTime); + } + } + + public static class RestartRandomRs extends RestartActionBase { + public RestartRandomRs(long sleepTime) { + super(sleepTime); + } + + @Override + public void perform() throws Exception { + LOG.info("Performing action: Restart random region server"); + ServerName server = selectRandomItem(getCurrentServers()); + + restartRs(server, sleepTime); + } + } + + public static class RestartRsHoldingMeta extends RestartActionBase { + public RestartRsHoldingMeta(long sleepTime) { + super(sleepTime); + } + @Override + public void perform() throws Exception { + LOG.info("Performing action: Restart region server holding META"); + ServerName server = cluster.getServerHoldingMeta(); + if (server == null) { + LOG.warn("No server is holding .META. right now."); + return; + } + restartRs(server, sleepTime); + } + } + + public static class RestartRsHoldingRoot extends RestartRandomRs { + public RestartRsHoldingRoot(long sleepTime) { + super(sleepTime); + } + @Override + public void perform() throws Exception { + LOG.info("Performing action: Restart region server holding ROOT"); + ServerName server = cluster.getServerHoldingRoot(); + if (server == null) { + LOG.warn("No server is holding -ROOT- right now."); + return; + } + restartRs(server, sleepTime); + } + } + + public static class RestartRsHoldingTable extends RestartActionBase { + + private final String tableName; + + public RestartRsHoldingTable(long sleepTime, String tableName) { + super(sleepTime); + this.tableName = tableName; + } + + @Override + public void perform() throws Exception { + HTable table = null; + Collection serverNames; + try { + Configuration conf = context.getHBaseIntegrationTestingUtility().getConfiguration(); + table = new HTable(conf, tableName); + serverNames = table.getRegionLocations().values(); + } catch (IOException e) { + LOG.debug("Error creating HTable used to get list of region locations.", e); + return; + } finally { + if (table != null) { + table.close(); + } + } + Random random = new Random(); + ServerName[] nameArray = serverNames.toArray(new ServerName[serverNames.size()]); + restartRs(nameArray[random.nextInt(nameArray.length)], sleepTime); + } + } + + public static class MoveRegionsOfTable extends Action { + private final long sleepTime; + private final byte[] tableNameBytes; + + public MoveRegionsOfTable(long sleepTime, String tableName) { + this.sleepTime = sleepTime; + this.tableNameBytes = Bytes.toBytes(tableName); + } + + @Override + public void perform() throws Exception { + try { + HBaseAdmin admin = this.context.getHBaseIntegrationTestingUtility().getHBaseAdmin(); + List regions = admin.getTableRegions(tableNameBytes); + Collection serversList = admin.getClusterStatus().getServers(); + ServerName[] servers = serversList.toArray(new ServerName[serversList.size()]); + Random random = new Random(); + for (HRegionInfo regionInfo:regions) { + try { + byte[] destServerName = + Bytes.toBytes(servers[random.nextInt(servers.length)].getServerName()); + admin.move(regionInfo.getRegionName(), destServerName); + } catch (Exception e) { + LOG.debug("Error moving region", e); + } + } + Thread.sleep(sleepTime); + } catch (Exception e) { + LOG.debug("Error performing MoveRegionsOfTable", e); + } + } + } + + /** + * Restarts a ratio of the running regionservers at the same time + */ + public static class BatchRestartRs extends RestartActionBase { + float ratio; //ratio of regionservers to restart + + public BatchRestartRs(long sleepTime, float ratio) { + super(sleepTime); + this.ratio = ratio; + } + + @Override + public void perform() throws Exception { + LOG.info(String.format("Performing action: Batch restarting %d%% of region servers", + (int)(ratio * 100))); + List selectedServers = selectRandomItems(getCurrentServers(), ratio); + + for (ServerName server : selectedServers) { + LOG.info("Killing region server:" + server); + cluster.killRegionServer(server); + } + + for (ServerName server : selectedServers) { + cluster.waitForRegionServerToStop(server, TIMEOUT); + } + + LOG.info("Killed " + selectedServers.size() + " region servers. Reported num of rs:" + + cluster.getClusterStatus().getServersSize()); + + sleep(sleepTime); + + for (ServerName server : selectedServers) { + LOG.info("Starting region server:" + server.getHostname()); + cluster.startRegionServer(server.getHostname()); + + } + for (ServerName server : selectedServers) { + cluster.waitForRegionServerToStart(server.getHostname(), TIMEOUT); + } + LOG.info("Started " + selectedServers.size() +" region servers. Reported num of rs:" + + cluster.getClusterStatus().getServersSize()); + } + } + + /** + * Restarts a ratio of the regionservers in a rolling fashion. At each step, either kills a + * server, or starts one, sleeping randomly (0-sleepTime) in between steps. + */ + public static class RollingBatchRestartRs extends BatchRestartRs { + public RollingBatchRestartRs(long sleepTime, float ratio) { + super(sleepTime, ratio); + } + + @Override + public void perform() throws Exception { + Random random = new Random(); + LOG.info(String.format("Performing action: Rolling batch restarting %d%% of region servers", + (int)(ratio * 100))); + List selectedServers = selectRandomItems(getCurrentServers(), ratio); + + Queue serversToBeKilled = new LinkedList(selectedServers); + Queue deadServers = new LinkedList(); + + // + while (!serversToBeKilled.isEmpty() || !deadServers.isEmpty()) { + boolean action = true; //action true = kill server, false = start server + + if (serversToBeKilled.isEmpty() || deadServers.isEmpty()) { + action = deadServers.isEmpty(); + } else { + action = random.nextBoolean(); + } + + if (action) { + ServerName server = serversToBeKilled.remove(); + killRs(server); + deadServers.add(server); + } else { + ServerName server = deadServers.remove(); + startRs(server); + } + + sleep(random.nextInt((int)sleepTime)); + } + } + } + + public static class UnbalanceRegionsAction extends Action { + private double fractionOfRegions; + private double fractionOfServers; + private Random random = new Random(); + + /** + * Unbalances the regions on the cluster by choosing "target" servers, and moving + * some regions from each of the non-target servers to random target servers. + * @param fractionOfRegions Fraction of regions to move from each server. + * @param fractionOfServers Fraction of servers to be chosen as targets. + */ + public UnbalanceRegionsAction(double fractionOfRegions, double fractionOfServers) { + this.fractionOfRegions = fractionOfRegions; + this.fractionOfServers = fractionOfServers; + } + + @Override + public void perform() throws Exception { + LOG.info("Unbalancing regions"); + ClusterStatus status = this.cluster.getClusterStatus(); + List victimServers = new LinkedList(status.getServers()); + int targetServerCount = (int)Math.ceil(fractionOfServers * victimServers.size()); + List targetServers = new ArrayList(targetServerCount); + for (int i = 0; i < targetServerCount; ++i) { + int victimIx = random.nextInt(victimServers.size()); + String serverName = victimServers.remove(victimIx).getServerName(); + targetServers.add(Bytes.toBytes(serverName)); + } + + List victimRegions = new LinkedList(); + for (ServerName server : victimServers) { + HServerLoad serverLoad = status.getLoad(server); + // Ugh. + List regions = new LinkedList(serverLoad.getRegionsLoad().keySet()); + int victimRegionCount = (int)Math.ceil(fractionOfRegions * regions.size()); + LOG.debug("Removing " + victimRegionCount + " regions from " + server.getServerName()); + for (int i = 0; i < victimRegionCount; ++i) { + int victimIx = random.nextInt(regions.size()); + String regionId = HRegionInfo.encodeRegionName(regions.remove(victimIx)); + victimRegions.add(Bytes.toBytes(regionId)); + } + } + + LOG.info("Moving " + victimRegions.size() + " regions from " + victimServers.size() + + " servers to " + targetServers.size() + " different servers"); + HBaseAdmin admin = this.context.getHBaseIntegrationTestingUtility().getHBaseAdmin(); + for (byte[] victimRegion : victimRegions) { + int targetIx = random.nextInt(targetServers.size()); + admin.move(victimRegion, targetServers.get(targetIx)); + } + } + } + + public static class ForceBalancerAction extends Action { + @Override + public void perform() throws Exception { + LOG.info("Balancing regions"); + HBaseAdmin admin = this.context.getHBaseIntegrationTestingUtility().getHBaseAdmin(); + boolean result = admin.balancer(); + if (!result) { + LOG.error("Balancer didn't succeed"); + } + } + } + + /** + * A context for a Policy + */ + public static class PolicyContext extends ActionContext { + public PolicyContext(IntegrationTestingUtility util) { + super(util); + } + } + + /** + * A policy to introduce chaos to the cluster + */ + public static abstract class Policy extends StoppableImplementation implements Runnable { + protected PolicyContext context; + public void init(PolicyContext context) throws Exception { + this.context = context; + } + } + + /** A policy that runs multiple other policies one after the other */ + public static class CompositeSequentialPolicy extends Policy { + private List policies; + public CompositeSequentialPolicy(Policy... policies) { + this.policies = Arrays.asList(policies); + } + + @Override + public void stop(String why) { + super.stop(why); + for (Policy p : policies) { + p.stop(why); + } + } + + @Override + public void run() { + for (Policy p : policies) { + p.run(); + } + } + + @Override + public void init(PolicyContext context) throws Exception { + super.init(context); + for (Policy p : policies) { + p.init(context); + } + } + } + + /** A policy which does stuff every time interval. */ + public static abstract class PeriodicPolicy extends Policy { + private long periodMs; + + public PeriodicPolicy(long periodMs) { + this.periodMs = periodMs; + } + + @Override + public void run() { + // Add some jitter. + int jitter = new Random().nextInt((int)periodMs); + LOG.info("Sleeping for " + jitter + " to add jitter"); + Threads.sleep(jitter); + + while (!isStopped()) { + long start = System.currentTimeMillis(); + runOneIteration(); + + if (isStopped()) return; + long sleepTime = periodMs - (System.currentTimeMillis() - start); + if (sleepTime > 0) { + LOG.info("Sleeping for: " + sleepTime); + Threads.sleep(sleepTime); + } + } + } + + protected abstract void runOneIteration(); + + @Override + public void init(PolicyContext context) throws Exception { + super.init(context); + LOG.info("Using ChaosMonkey Policy: " + this.getClass() + ", period: " + periodMs); + } + } + + /** A policy which performs a sequence of actions deterministically. */ + public static class DoActionsOncePolicy extends PeriodicPolicy { + private List actions; + + public DoActionsOncePolicy(long periodMs, List actions) { + super(periodMs); + this.actions = new ArrayList(actions); + } + + public DoActionsOncePolicy(long periodMs, Action... actions) { + this(periodMs, Arrays.asList(actions)); + } + + @Override + protected void runOneIteration() { + if (actions.isEmpty()) { + this.stop("done"); + return; + } + Action action = actions.remove(0); + + try { + action.perform(); + } catch (Exception ex) { + LOG.warn("Exception occured during performing action: " + + StringUtils.stringifyException(ex)); + } + } + + @Override + public void init(PolicyContext context) throws Exception { + super.init(context); + for (Action action : actions) { + action.init(this.context); + } + } + } + + /** + * A policy, which picks a random action according to the given weights, + * and performs it every configurable period. + */ + public static class PeriodicRandomActionPolicy extends PeriodicPolicy { + private List> actions; + + public PeriodicRandomActionPolicy(long periodMs, List> actions) { + super(periodMs); + this.actions = actions; + } + + public PeriodicRandomActionPolicy(long periodMs, Pair... actions) { + // We don't expect it to be modified. + this(periodMs, Arrays.asList(actions)); + } + + public PeriodicRandomActionPolicy(long periodMs, Action... actions) { + super(periodMs); + this.actions = new ArrayList>(actions.length); + for (Action action : actions) { + this.actions.add(new Pair(action, 1)); + } + } + + @Override + protected void runOneIteration() { + Action action = selectWeightedRandomItem(actions); + try { + action.perform(); + } catch (Exception ex) { + LOG.warn("Exception occured during performing action: " + + StringUtils.stringifyException(ex)); + } + } + + @Override + public void init(PolicyContext context) throws Exception { + super.init(context); + for (Pair action : actions) { + action.getFirst().init(this.context); + } + } + } + + /** Selects a random item from the given items */ + static T selectRandomItem(T[] items) { + Random random = new Random(); + return items[random.nextInt(items.length)]; + } + + /** Selects a random item from the given items with weights*/ + static T selectWeightedRandomItem(List> items) { + Random random = new Random(); + int totalWeight = 0; + for (Pair pair : items) { + totalWeight += pair.getSecond(); + } + + int cutoff = random.nextInt(totalWeight); + int cummulative = 0; + T item = null; + + //warn: O(n) + for (int i=0; i List selectRandomItems(T[] items, float ratio) { + Random random = new Random(); + int remaining = (int)Math.ceil(items.length * ratio); + + List selectedItems = new ArrayList(remaining); + + for (int i=0; i 0; i++) { + if (random.nextFloat() < ((float)remaining/(items.length-i))) { + selectedItems.add(items[i]); + remaining--; + } + } + + return selectedItems; + } + + /** + * All actions that deal with RS's with the following weights (relative probabilities): + * - Restart active master (sleep 5 sec) : 2 + * - Restart random regionserver (sleep 5 sec) : 2 + * - Restart random regionserver (sleep 60 sec) : 2 + * - Restart META regionserver (sleep 5 sec) : 1 + * - Restart ROOT regionserver (sleep 5 sec) : 1 + * - Batch restart of 50% of regionservers (sleep 5 sec) : 2 + * - Rolling restart of 100% of regionservers (sleep 5 sec) : 2 + */ + @SuppressWarnings("unchecked") + private static final List> ALL_ACTIONS = Lists.newArrayList( + new Pair(new RestartActiveMaster(FIVE_SEC), 2), + new Pair(new RestartRandomRs(FIVE_SEC), 2), + new Pair(new RestartRandomRs(ONE_MIN), 2), + new Pair(new RestartRsHoldingMeta(FIVE_SEC), 1), + new Pair(new RestartRsHoldingRoot(FIVE_SEC), 1), + new Pair(new BatchRestartRs(FIVE_SEC, 0.5f), 2), + new Pair(new RollingBatchRestartRs(FIVE_SEC, 1.0f), 2) + ); + + public static final String EVERY_MINUTE_RANDOM_ACTION_POLICY = "EVERY_MINUTE_RANDOM_ACTION_POLICY"; + + private Policy[] policies; + private Thread[] monkeyThreads; + + public void start() throws Exception { + monkeyThreads = new Thread[policies.length]; + + for (int i=0; i NAMED_POLICIES = Maps.newHashMap(); + static { + NAMED_POLICIES.put(EVERY_MINUTE_RANDOM_ACTION_POLICY, + new PeriodicRandomActionPolicy(ONE_MIN, ALL_ACTIONS)); + } + + @Override + protected void addOptions() { + addOptWithArg("policy", "a named policy defined in ChaosMonkey.java. Possible values: " + + NAMED_POLICIES.keySet()); + //we can add more options, and make policies more configurable + } + + @Override + protected void processOptions(CommandLine cmd) { + String[] policies = cmd.getOptionValues("policy"); + if (policies != null) { + setPoliciesByName(policies); + } else { + // Set a default policy if none is provided + setPolicies(NAMED_POLICIES.get(EVERY_MINUTE_RANDOM_ACTION_POLICY)); + } + } + + @Override + protected int doWork() throws Exception { + start(); + waitForStop(); + return 0; + } + + public static void main(String[] args) throws Exception { + Configuration conf = HBaseConfiguration.create(); + IntegrationTestingUtility.setUseDistributedCluster(conf); + IntegrationTestingUtility util = new IntegrationTestingUtility(conf); + util.initializeCluster(1); + + ChaosMonkey monkey = new ChaosMonkey(util); + int ret = ToolRunner.run(conf, monkey, args); + System.exit(ret); + } + +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/ClassLoaderTestHelper.java b/src/test/java/org/apache/hadoop/hbase/util/ClassLoaderTestHelper.java new file mode 100644 index 000000000000..55a3feb0d691 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/ClassLoaderTestHelper.java @@ -0,0 +1,164 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.util; + +import static org.junit.Assert.assertTrue; + +import java.io.BufferedWriter; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.FileWriter; +import java.util.ArrayList; +import java.util.List; +import java.util.jar.JarEntry; +import java.util.jar.JarOutputStream; +import java.util.jar.Manifest; + +import javax.tools.JavaCompiler; +import javax.tools.JavaFileObject; +import javax.tools.StandardJavaFileManager; +import javax.tools.ToolProvider; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.Path; + +/** + * Some utilities to help class loader testing + */ +public class ClassLoaderTestHelper { + private static final Log LOG = LogFactory.getLog(ClassLoaderTestHelper.class); + + /** + * Jar a list of files into a jar archive. + * + * @param archiveFile the target jar archive + * @param tobejared a list of files to be jared + */ + private static boolean createJarArchive(File archiveFile, File[] tobeJared) { + try { + byte buffer[] = new byte[4096]; + // Open archive file + FileOutputStream stream = new FileOutputStream(archiveFile); + JarOutputStream out = new JarOutputStream(stream, new Manifest()); + + for (int i = 0; i < tobeJared.length; i++) { + if (tobeJared[i] == null || !tobeJared[i].exists() + || tobeJared[i].isDirectory()) { + continue; + } + + // Add archive entry + JarEntry jarAdd = new JarEntry(tobeJared[i].getName()); + jarAdd.setTime(tobeJared[i].lastModified()); + out.putNextEntry(jarAdd); + + // Write file to archive + FileInputStream in = new FileInputStream(tobeJared[i]); + while (true) { + int nRead = in.read(buffer, 0, buffer.length); + if (nRead <= 0) + break; + out.write(buffer, 0, nRead); + } + in.close(); + } + out.close(); + stream.close(); + LOG.info("Adding classes to jar file completed"); + return true; + } catch (Exception ex) { + LOG.error("Error: " + ex.getMessage()); + return false; + } + } + + /** + * Create a test jar for testing purpose for a given class + * name with specified code string: save the class to a file, + * compile it, and jar it up. If the code string passed in is + * null, a bare empty class will be created and used. + * + * @param testDir the folder under which to store the test class and jar + * @param className the test class name + * @param code the optional test class code, which can be null. + * If null, a bare empty class will be used + * @return the test jar file generated + */ + public static File buildJar(String testDir, + String className, String code) throws Exception { + return buildJar(testDir, className, code, testDir); + } + + /** + * Create a test jar for testing purpose for a given class + * name with specified code string. + * + * @param testDir the folder under which to store the test class + * @param className the test class name + * @param code the optional test class code, which can be null. + * If null, an empty class will be used + * @param folder the folder under which to store the generated jar + * @return the test jar file generated + */ + public static File buildJar(String testDir, + String className, String code, String folder) throws Exception { + String javaCode = code != null ? code : "public class " + className + " {}"; + Path srcDir = new Path(testDir, "src"); + File srcDirPath = new File(srcDir.toString()); + srcDirPath.mkdirs(); + File sourceCodeFile = new File(srcDir.toString(), className + ".java"); + BufferedWriter bw = new BufferedWriter(new FileWriter(sourceCodeFile)); + bw.write(javaCode); + bw.close(); + + // compile it by JavaCompiler + JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + ArrayList srcFileNames = new ArrayList(); + srcFileNames.add(sourceCodeFile.toString()); + StandardJavaFileManager fm = compiler.getStandardFileManager(null, null, + null); + Iterable cu = + fm.getJavaFileObjects(sourceCodeFile); + List options = new ArrayList(); + options.add("-classpath"); + // only add hbase classes to classpath. This is a little bit tricky: assume + // the classpath is {hbaseSrc}/target/classes. + String currentDir = new File(".").getAbsolutePath(); + String classpath = currentDir + File.separator + "target"+ File.separator + + "classes" + System.getProperty("path.separator") + + System.getProperty("java.class.path") + System.getProperty("path.separator") + + System.getProperty("surefire.test.class.path"); + options.add(classpath); + LOG.debug("Setting classpath to: " + classpath); + + JavaCompiler.CompilationTask task = compiler.getTask(null, fm, null, + options, null, cu); + assertTrue("Compile file " + sourceCodeFile + " failed.", task.call()); + + // build a jar file by the classes files + String jarFileName = className + ".jar"; + File jarFile = new File(folder, jarFileName); + if (!createJarArchive(jarFile, + new File[]{new File(srcDir.toString(), className + ".class")})){ + assertTrue("Build jar file failed.", false); + } + return jarFile; + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/EnvironmentEdgeManagerTestHelper.java b/src/test/java/org/apache/hadoop/hbase/util/EnvironmentEdgeManagerTestHelper.java index 730f4e3af6a8..47d79cbcf8bb 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/EnvironmentEdgeManagerTestHelper.java +++ b/src/test/java/org/apache/hadoop/hbase/util/EnvironmentEdgeManagerTestHelper.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java b/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java new file mode 100644 index 000000000000..7609e0ee5405 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java @@ -0,0 +1,239 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.util; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.Store; + +/** + * Test helper for testing archiving of HFiles + */ +public class HFileArchiveTestingUtil { + + private static final Log LOG = LogFactory.getLog(HFileArchiveTestingUtil.class); + + private HFileArchiveTestingUtil() { + // NOOP private ctor since this is just a utility class + } + + public static boolean compareArchiveToOriginal(FileStatus[] previous, FileStatus[] archived, + FileSystem fs, boolean hasTimedBackup) { + + List> lists = getFileLists(previous, archived); + List original = lists.get(0); + Collections.sort(original); + + List currentFiles = lists.get(1); + Collections.sort(currentFiles); + + List backedup = lists.get(2); + Collections.sort(backedup); + + // check the backed up files versus the current (should match up, less the + // backup time in the name) + if (!hasTimedBackup == (backedup.size() > 0)) { + LOG.debug("backedup files doesn't match expected."); + return false; + } + String msg = null; + if (hasTimedBackup) { + msg = assertArchiveEquality(original, backedup); + if (msg != null) { + LOG.debug(msg); + return false; + } + } + msg = assertArchiveEquality(original, currentFiles); + if (msg != null) { + LOG.debug(msg); + return false; + } + return true; + } + + /** + * Compare the archived files to the files in the original directory + * @param previous original files that should have been archived + * @param archived files that were archived + * @param fs filessystem on which the archiving took place + * @throws IOException + */ + public static void assertArchiveEqualToOriginal(FileStatus[] previous, FileStatus[] archived, + FileSystem fs) throws IOException { + assertArchiveEqualToOriginal(previous, archived, fs, false); + } + + /** + * Compare the archived files to the files in the original directory + * @param previous original files that should have been archived + * @param archived files that were archived + * @param fs {@link FileSystem} on which the archiving took place + * @param hasTimedBackup true if we expect to find an archive backup directory with a + * copy of the files in the archive directory (and the original files). + * @throws IOException + */ + public static void assertArchiveEqualToOriginal(FileStatus[] previous, FileStatus[] archived, + FileSystem fs, boolean hasTimedBackup) throws IOException { + + List> lists = getFileLists(previous, archived); + List original = lists.get(0); + Collections.sort(original); + + List currentFiles = lists.get(1); + Collections.sort(currentFiles); + + List backedup = lists.get(2); + Collections.sort(backedup); + + // check the backed up files versus the current (should match up, less the + // backup time in the name) + assertEquals("Didn't expect any backup files, but got: " + backedup, hasTimedBackup, + backedup.size() > 0); + String msg = null; + if (hasTimedBackup) { + assertArchiveEquality(original, backedup); + assertNull(msg, msg); + } + + // do the rest of the comparison + msg = assertArchiveEquality(original, currentFiles); + assertNull(msg, msg); + } + + private static String assertArchiveEquality(List expected, List archived) { + String compare = compareFileLists(expected, archived); + if (!(expected.size() == archived.size())) return "Not the same number of current files\n" + + compare; + if (!expected.equals(archived)) return "Different backup files, but same amount\n" + compare; + return null; + } + + /** + * @return , where each is sorted + */ + private static List> getFileLists(FileStatus[] previous, FileStatus[] archived) { + List> files = new ArrayList>(); + + // copy over the original files + List originalFileNames = convertToString(previous); + files.add(originalFileNames); + + List currentFiles = new ArrayList(previous.length); + List backedupFiles = new ArrayList(previous.length); + for (FileStatus f : archived) { + String name = f.getPath().getName(); + // if the file has been backed up + if (name.contains(".")) { + Path parent = f.getPath().getParent(); + String shortName = name.split("[.]")[0]; + Path modPath = new Path(parent, shortName); + FileStatus file = new FileStatus(f.getLen(), f.isDir(), f.getReplication(), + f.getBlockSize(), f.getModificationTime(), modPath); + backedupFiles.add(file); + } else { + // otherwise, add it to the list to compare to the original store files + currentFiles.add(name); + } + } + + files.add(currentFiles); + files.add(convertToString(backedupFiles)); + return files; + } + + private static List convertToString(FileStatus[] files) { + return convertToString(Arrays.asList(files)); + } + + private static List convertToString(List files) { + List originalFileNames = new ArrayList(files.size()); + for (FileStatus f : files) { + originalFileNames.add(f.getPath().getName()); + } + return originalFileNames; + } + + /* Get a pretty representation of the differences */ + private static String compareFileLists(List expected, List gotten) { + StringBuilder sb = new StringBuilder("Expected (" + expected.size() + "): \t\t Gotten (" + + gotten.size() + "):\n"); + List notFound = new ArrayList(); + for (String s : expected) { + if (gotten.contains(s)) sb.append(s + "\t\t" + s + "\n"); + else notFound.add(s); + } + sb.append("Not Found:\n"); + for (String s : notFound) { + sb.append(s + "\n"); + } + sb.append("\nExtra:\n"); + for (String s : gotten) { + if (!expected.contains(s)) sb.append(s + "\n"); + } + return sb.toString(); + } + + /** + * Helper method to get the archive directory for the specified region + * @param conf {@link Configuration} to check for the name of the archive directory + * @param region region that is being archived + * @return {@link Path} to the archive directory for the given region + */ + public static Path getRegionArchiveDir(Configuration conf, HRegion region) { + return HFileArchiveUtil.getRegionArchiveDir(conf, region.getTableDir(), region.getRegionDir()); + } + + /** + * Helper method to get the store archive directory for the specified region + * @param conf {@link Configuration} to check for the name of the archive directory + * @param region region that is being archived + * @param store store that is archiving files + * @return {@link Path} to the store archive directory for the given region + */ + public static Path getStoreArchivePath(Configuration conf, HRegion region, Store store) { + return HFileArchiveUtil.getStoreArchivePath(conf, region, store.getFamily().getName()); + } + + public static Path getStoreArchivePath(HBaseTestingUtility util, String tableName, + byte[] storeName) throws IOException { + byte[] table = Bytes.toBytes(tableName); + // get the RS and region serving our table + List servingRegions = util.getHBaseCluster().getRegions(table); + HRegion region = servingRegions.get(0); + + // check that we actually have some store files that were archived + Store store = region.getStore(storeName); + return HFileArchiveTestingUtil.getStoreArchivePath(util.getConfiguration(), region, store); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/HFileTestUtil.java b/src/test/java/org/apache/hadoop/hbase/util/HFileTestUtil.java new file mode 100644 index 000000000000..6bc2866660d9 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/HFileTestUtil.java @@ -0,0 +1,63 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.util; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.io.hfile.CacheConfig; +import org.apache.hadoop.hbase.io.hfile.HFile; +import org.apache.hadoop.hbase.regionserver.StoreFile; + +import java.io.IOException; + +/** + * Utility class for HFile-related testing. + */ +public class HFileTestUtil { + + /** + * Create an HFile with the given number of rows between a given + * start key and end key. + */ + public static void createHFile( + Configuration configuration, + FileSystem fs, Path path, + byte[] family, byte[] qualifier, + byte[] startKey, byte[] endKey, int numRows) throws IOException + { + HFile.Writer writer = HFile.getWriterFactory(configuration, new CacheConfig(configuration)) + .withPath(fs, path) + .withComparator(KeyValue.KEY_COMPARATOR) + .create(); + long now = System.currentTimeMillis(); + try { + // subtract 2 since iterateOnSplits doesn't include boundary keys + for (byte[] key : Bytes.iterateOnSplits(startKey, endKey, numRows-2)) { + KeyValue kv = new KeyValue(key, family, qualifier, now, key); + writer.append(kv); + } + } finally { + writer.appendFileInfo(StoreFile.BULKLOAD_TIME_KEY, + Bytes.toBytes(System.currentTimeMillis())); + writer.close(); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/LoadTestDataGenerator.java b/src/test/java/org/apache/hadoop/hbase/util/LoadTestDataGenerator.java new file mode 100644 index 000000000000..c7f6dd6b6e26 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/LoadTestDataGenerator.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations + * under the License. + */ + +package org.apache.hadoop.hbase.util; + +import java.util.Set; + +/** + * A generator of random data (keys/cfs/columns/values) for load testing. + * Contains LoadTestKVGenerator as a matter of convenience... + */ +public abstract class LoadTestDataGenerator { + protected final LoadTestKVGenerator kvGenerator; + + /** + * Initializes the object. + * @param minValueSize minimum size of the value generated by + * {@link #generateValue(byte[], byte[], byte[])}. + * @param maxValueSize maximum size of the value generated by + * {@link #generateValue(byte[], byte[], byte[])}. + */ + public LoadTestDataGenerator(int minValueSize, int maxValueSize) { + this.kvGenerator = new LoadTestKVGenerator(minValueSize, maxValueSize); + } + /** + * Generates a deterministic, unique hashed row key from a number. That way, the user can + * keep track of numbers, without messing with byte array and ensuring key distribution. + * @param keyBase Base number for a key, such as a loop counter. + */ + public abstract byte[] getDeterministicUniqueKey(long keyBase); + + /** + * Gets column families for the load test table. + * @return The array of byte[]s representing column family names. + */ + public abstract byte[][] getColumnFamilies(); + + /** + * Generates an applicable set of columns to be used for a particular key and family. + * @param rowKey The row key to generate for. + * @param cf The column family name to generate for. + * @return The array of byte[]s representing column names. + */ + public abstract byte[][] generateColumnsForCf(byte[] rowKey, byte[] cf); + + /** + * Generates a value to be used for a particular row/cf/column. + * @param rowKey The row key to generate for. + * @param cf The column family name to generate for. + * @param column The column name to generate for. + * @return The value to use. + */ + public abstract byte[] generateValue(byte[] rowKey, byte[] cf, byte[] column); + + /** + * Checks that columns for a rowKey and cf are valid if generated via + * {@link #generateColumnsForCf(byte[], byte[])} + * @param rowKey The row key to verify for. + * @param cf The column family name to verify for. + * @param columnSet The column set (for example, encountered by read). + * @return True iff valid. + */ + public abstract boolean verify(byte[] rowKey, byte[] cf, Set columnSet); + + /** + * Checks that value for a rowKey/cf/column is valid if generated via + * {@link #generateValue(byte[], byte[], byte[])} + * @param rowKey The row key to verify for. + * @param cf The column family name to verify for. + * @param column The column name to verify for. + * @param value The value (for example, encountered by read). + * @return True iff valid. + */ + public abstract boolean verify(byte[] rowKey, byte[] cf, byte[] column, byte[] value); +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java b/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java index ba125a62e9a7..77f5420c60d9 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java +++ b/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java @@ -27,8 +27,6 @@ * hash. Values are generated by selecting value size in the configured range * and generating a pseudo-random sequence of bytes seeded by key, column * qualifier, and value size. - *

- * Not thread-safe, so a separate instance is needed for every writer thread/ */ public class LoadTestKVGenerator { @@ -49,13 +47,13 @@ public LoadTestKVGenerator(int minValueSize, int maxValueSize) { /** * Verifies that the given byte array is the same as what would be generated - * for the given row key and qualifier. We are assuming that the value size - * is correct, and only verify the actual bytes. However, if the min/max - * value sizes are set sufficiently high, an accidental match should be + * for the given seed strings (row/cf/column/...). We are assuming that the + * value size is correct, and only verify the actual bytes. However, if the + * min/max value sizes are set sufficiently high, an accidental match should be * extremely improbable. */ - public static boolean verify(String rowKey, String qual, byte[] value) { - byte[] expectedData = getValueForRowColumn(rowKey, qual, value.length); + public static boolean verify(byte[] value, byte[]... seedStrings) { + byte[] expectedData = getValueForRowColumn(value.length, seedStrings); return Bytes.equals(expectedData, value); } @@ -74,25 +72,31 @@ public static String md5PrefixedKey(long key) { /** * Generates a value for the given key index and column qualifier. Size is * selected randomly in the configured range. The generated value depends - * only on the combination of the key, qualifier, and the selected value - * size. This allows to verify the actual value bytes when reading, as done - * in {@link #verify(String, String, byte[])}. + * only on the combination of the strings passed (key/cf/column/...) and the selected + * value size. This allows to verify the actual value bytes when reading, as done + * in {#verify(byte[], byte[]...)} + * This method is as thread-safe as Random class. It appears that the worst bug ever + * found with the latter is that multiple threads will get some duplicate values, which + * we don't care about. */ - public byte[] generateRandomSizeValue(long key, String qual) { - String rowKey = md5PrefixedKey(key); - int dataSize = minValueSize + randomForValueSize.nextInt( - Math.abs(maxValueSize - minValueSize)); - return getValueForRowColumn(rowKey, qual, dataSize); - } + public byte[] generateRandomSizeValue(byte[]... seedStrings) { + int dataSize = minValueSize; + if (minValueSize != maxValueSize) { + dataSize = minValueSize + randomForValueSize.nextInt(Math.abs(maxValueSize - minValueSize)); + } + return getValueForRowColumn(dataSize, seedStrings); + } /** * Generates random bytes of the given size for the given row and column * qualifier. The random seed is fully determined by these parameters. */ - private static byte[] getValueForRowColumn(String rowKey, String qual, - int dataSize) { - Random seededRandom = new Random(rowKey.hashCode() + qual.hashCode() + - dataSize); + private static byte[] getValueForRowColumn(int dataSize, byte[]... seedStrings) { + long seed = dataSize; + for (byte[] str : seedStrings) { + seed += Bytes.toString(str).hashCode(); + } + Random seededRandom = new Random(seed); byte[] randomBytes = new byte[dataSize]; seededRandom.nextBytes(randomBytes); return randomBytes; diff --git a/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java b/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java index b70ec784d230..dfefe9de5bf5 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java +++ b/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java @@ -17,11 +17,16 @@ package org.apache.hadoop.hbase.util; import java.io.IOException; +import java.io.InterruptedIOException; +import java.util.ArrayList; import java.util.Arrays; +import java.util.List; +import java.util.concurrent.atomic.AtomicReference; import org.apache.commons.cli.CommandLine; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HConstants; @@ -31,6 +36,7 @@ import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; import org.apache.hadoop.hbase.io.hfile.Compression; import org.apache.hadoop.hbase.regionserver.StoreFile; +import org.apache.hadoop.util.ToolRunner; /** * A command-line utility that reads, writes, and verifies data. Unlike @@ -42,33 +48,36 @@ public class LoadTestTool extends AbstractHBaseTool { private static final Log LOG = LogFactory.getLog(LoadTestTool.class); /** Table name for the test */ - private byte[] tableName; + protected byte[] tableName; /** Table name to use of not overridden on the command line */ - private static final String DEFAULT_TABLE_NAME = "cluster_test"; + protected static final String DEFAULT_TABLE_NAME = "cluster_test"; /** Column family used by the test */ - static byte[] COLUMN_FAMILY = Bytes.toBytes("test_cf"); + protected static byte[] COLUMN_FAMILY = Bytes.toBytes("test_cf"); /** Column families used by the test */ - static final byte[][] COLUMN_FAMILIES = { COLUMN_FAMILY }; + protected static final byte[][] COLUMN_FAMILIES = { COLUMN_FAMILY }; + + /** The default data size if not specified */ + protected static final int DEFAULT_DATA_SIZE = 64; /** The number of reader/writer threads if not specified */ - private static final int DEFAULT_NUM_THREADS = 20; + protected static final int DEFAULT_NUM_THREADS = 20; /** Usage string for the load option */ - private static final String OPT_USAGE_LOAD = + protected static final String OPT_USAGE_LOAD = ":" + "[:<#threads=" + DEFAULT_NUM_THREADS + ">]"; /** Usa\ge string for the read option */ - private static final String OPT_USAGE_READ = + protected static final String OPT_USAGE_READ = "[:<#threads=" + DEFAULT_NUM_THREADS + ">]"; - private static final String OPT_USAGE_BLOOM = "Bloom filter type, one of " + + protected static final String OPT_USAGE_BLOOM = "Bloom filter type, one of " + Arrays.toString(StoreFile.BloomType.values()); - private static final String OPT_USAGE_COMPRESSION = "Compression type, " + + protected static final String OPT_USAGE_COMPRESSION = "Compression type, " + "one of " + Arrays.toString(Compression.Algorithm.values()); public static final String OPT_DATA_BLOCK_ENCODING_USAGE = @@ -76,6 +85,10 @@ public class LoadTestTool extends AbstractHBaseTool { + "compression) to use for data blocks in the test column family, " + "one of " + Arrays.toString(DataBlockEncoding.values()) + "."; + public static final String OPT_INMEMORY = "in_memory"; + public static final String OPT_USAGE_IN_MEMORY = "Tries to keep the HFiles of the CF " + + "inmemory as far as possible. Not guaranteed that reads are always served from inmemory"; + private static final String OPT_BLOOM = "bloom"; private static final String OPT_COMPRESSION = "compression"; public static final String OPT_DATA_BLOCK_ENCODING = @@ -86,47 +99,58 @@ public class LoadTestTool extends AbstractHBaseTool { "If this is specified, data blocks will only be encoded in block " + "cache but not on disk"; - private static final String OPT_KEY_WINDOW = "key_window"; - private static final String OPT_WRITE = "write"; - private static final String OPT_MAX_READ_ERRORS = "max_read_errors"; - private static final String OPT_MULTIPUT = "multiput"; - private static final String OPT_NUM_KEYS = "num_keys"; - private static final String OPT_READ = "read"; - private static final String OPT_START_KEY = "start_key"; - private static final String OPT_TABLE_NAME = "tn"; - private static final String OPT_ZK_QUORUM = "zk"; - - private static final long DEFAULT_START_KEY = 0; + protected static final String OPT_KEY_WINDOW = "key_window"; + protected static final String OPT_WRITE = "write"; + protected static final String OPT_MAX_READ_ERRORS = "max_read_errors"; + protected static final String OPT_MULTIPUT = "multiput"; + protected static final String OPT_NUM_KEYS = "num_keys"; + protected static final String OPT_READ = "read"; + protected static final String OPT_START_KEY = "start_key"; + protected static final String OPT_TABLE_NAME = "tn"; + protected static final String OPT_ZK_QUORUM = "zk"; + protected static final String OPT_SKIP_INIT = "skip_init"; + protected static final String OPT_INIT_ONLY = "init_only"; + private static final String NUM_TABLES = "num_tables"; + + protected static final long DEFAULT_START_KEY = 0; /** This will be removed as we factor out the dependency on command line */ - private CommandLine cmd; + protected CommandLine cmd; - private MultiThreadedWriter writerThreads = null; - private MultiThreadedReader readerThreads = null; + protected MultiThreadedWriter writerThreads = null; + protected MultiThreadedReader readerThreads = null; - private long startKey, endKey; + protected long startKey, endKey; - private boolean isWrite, isRead; + protected boolean isWrite, isRead; // Column family options - private DataBlockEncoding dataBlockEncodingAlgo; - private boolean encodeInCacheOnly; - private Compression.Algorithm compressAlgo; - private StoreFile.BloomType bloomType; + protected DataBlockEncoding dataBlockEncodingAlgo; + protected boolean encodeInCacheOnly; + protected Compression.Algorithm compressAlgo; + protected StoreFile.BloomType bloomType; + private boolean inMemoryCF; // Writer options - private int numWriterThreads = DEFAULT_NUM_THREADS; - private long minColsPerKey, maxColsPerKey; - private int minColDataSize, maxColDataSize; - private boolean isMultiPut; + protected int numWriterThreads = DEFAULT_NUM_THREADS; + protected int minColsPerKey, maxColsPerKey; + protected int minColDataSize = DEFAULT_DATA_SIZE, maxColDataSize = DEFAULT_DATA_SIZE; + protected boolean isMultiPut; // Reader options private int numReaderThreads = DEFAULT_NUM_THREADS; private int keyWindow = MultiThreadedReader.DEFAULT_KEY_WINDOW; private int maxReadErrors = MultiThreadedReader.DEFAULT_MAX_ERRORS; private int verifyPercent; + + private int numTables = 1; + + // TODO: refactor LoadTestToolImpl somewhere to make the usage from tests less bad, + // console tool itself should only be used from console. + protected boolean isSkipInit = false; + protected boolean isInitOnly = false; - private String[] splitColonSeparated(String option, + protected String[] splitColonSeparated(String option, int minNumCols, int maxNumCols) { String optVal = cmd.getOptionValue(option); String[] cols = optVal.split(":"); @@ -139,7 +163,7 @@ private String[] splitColonSeparated(String option, return cols; } - private int getNumThreads(String numThreadsStr) { + protected int getNumThreads(String numThreadsStr) { return parseInt(numThreadsStr, 1, Short.MAX_VALUE); } @@ -147,7 +171,7 @@ private int getNumThreads(String numThreadsStr) { * Apply column family options such as Bloom filters, compression, and data * block encoding. */ - private void applyColumnFamilyOptions(byte[] tableName, + protected void applyColumnFamilyOptions(byte[] tableName, byte[][] columnFamilies) throws IOException { HBaseAdmin admin = new HBaseAdmin(conf); HTableDescriptor tableDesc = admin.getTableDescriptor(tableName); @@ -155,6 +179,10 @@ private void applyColumnFamilyOptions(byte[] tableName, admin.disableTable(tableName); for (byte[] cf : columnFamilies) { HColumnDescriptor columnDesc = tableDesc.getFamily(cf); + boolean isNewCf = columnDesc == null; + if (isNewCf) { + columnDesc = new HColumnDescriptor(cf); + } if (bloomType != null) { columnDesc.setBloomFilterType(bloomType); } @@ -165,7 +193,14 @@ private void applyColumnFamilyOptions(byte[] tableName, columnDesc.setDataBlockEncoding(dataBlockEncodingAlgo); columnDesc.setEncodeOnDisk(!encodeInCacheOnly); } - admin.modifyColumn(tableName, columnDesc); + if (inMemoryCF) { + columnDesc.setInMemory(inMemoryCF); + } + if (isNewCf) { + admin.addColumn(tableName, columnDesc); + } else { + admin.modifyColumn(tableName, columnDesc); + } } LOG.info("Enabling table " + Bytes.toString(tableName)); admin.enableTable(tableName); @@ -178,6 +213,7 @@ protected void addOptions() { addOptWithArg(OPT_TABLE_NAME, "The name of the table to read or write"); addOptWithArg(OPT_WRITE, OPT_USAGE_LOAD); addOptWithArg(OPT_READ, OPT_USAGE_READ); + addOptNoArg(OPT_INIT_ONLY, "Initialize the test table only, don't do any loading"); addOptWithArg(OPT_BLOOM, OPT_USAGE_BLOOM); addOptWithArg(OPT_COMPRESSION, OPT_USAGE_COMPRESSION); addOptWithArg(OPT_DATA_BLOCK_ENCODING, OPT_DATA_BLOCK_ENCODING_USAGE); @@ -191,11 +227,19 @@ protected void addOptions() { addOptNoArg(OPT_MULTIPUT, "Whether to use multi-puts as opposed to " + "separate puts for every column in a row"); addOptNoArg(OPT_ENCODE_IN_CACHE_ONLY, OPT_ENCODE_IN_CACHE_ONLY_USAGE); + addOptNoArg(OPT_INMEMORY, OPT_USAGE_IN_MEMORY); - addRequiredOptWithArg(OPT_NUM_KEYS, "The number of keys to read/write"); + addOptWithArg(OPT_NUM_KEYS, "The number of keys to read/write"); addOptWithArg(OPT_START_KEY, "The first key to read/write " + "(a 0-based index). The default value is " + DEFAULT_START_KEY + "."); + addOptNoArg(OPT_SKIP_INIT, "Skip the initialization; assume test table " + + "already exists"); + + addOptWithArg(NUM_TABLES, + "A positive integer number. When a number n is speicfied, load test " + + "tool will load n table parallely. -tn parameter value becomes " + + "table name prefix. Each table name is in format _1..._n"); } @Override @@ -204,20 +248,35 @@ protected void processOptions(CommandLine cmd) { tableName = Bytes.toBytes(cmd.getOptionValue(OPT_TABLE_NAME, DEFAULT_TABLE_NAME)); - startKey = parseLong(cmd.getOptionValue(OPT_START_KEY, - String.valueOf(DEFAULT_START_KEY)), 0, Long.MAX_VALUE); - long numKeys = parseLong(cmd.getOptionValue(OPT_NUM_KEYS), 1, - Long.MAX_VALUE - startKey); - endKey = startKey + numKeys; isWrite = cmd.hasOption(OPT_WRITE); isRead = cmd.hasOption(OPT_READ); + isInitOnly = cmd.hasOption(OPT_INIT_ONLY); - if (!isWrite && !isRead) { + if (!isWrite && !isRead && !isInitOnly) { throw new IllegalArgumentException("Either -" + OPT_WRITE + " or " + "-" + OPT_READ + " has to be specified"); } + if (isInitOnly && (isRead || isWrite)) { + throw new IllegalArgumentException(OPT_INIT_ONLY + " cannot be specified with" + + " either -" + OPT_WRITE + " or -" + OPT_READ); + } + + if (!isInitOnly) { + if (!cmd.hasOption(OPT_NUM_KEYS)) { + throw new IllegalArgumentException(OPT_NUM_KEYS + " must be specified in " + + "read or write mode"); + } + startKey = parseLong(cmd.getOptionValue(OPT_START_KEY, + String.valueOf(DEFAULT_START_KEY)), 0, Long.MAX_VALUE); + long numKeys = parseLong(cmd.getOptionValue(OPT_NUM_KEYS), 1, + Long.MAX_VALUE - startKey); + endKey = startKey + numKeys; + isSkipInit = cmd.hasOption(OPT_SKIP_INIT); + System.out.println("Key range: [" + startKey + ".." + (endKey - 1) + "]"); + } + encodeInCacheOnly = cmd.hasOption(OPT_ENCODE_IN_CACHE_ONLY); parseColumnFamilyOptions(cmd); @@ -226,7 +285,7 @@ protected void processOptions(CommandLine cmd) { int colIndex = 0; minColsPerKey = 1; - maxColsPerKey = 2 * Long.parseLong(writeOpts[colIndex++]); + maxColsPerKey = 2 * Integer.parseInt(writeOpts[colIndex++]); int avgColDataSize = parseInt(writeOpts[colIndex++], 1, Integer.MAX_VALUE); minColDataSize = avgColDataSize / 2; @@ -266,11 +325,14 @@ protected void processOptions(CommandLine cmd) { System.out.println("Percent of keys to verify: " + verifyPercent); System.out.println("Reader threads: " + numReaderThreads); } - - System.out.println("Key range: [" + startKey + ".." + (endKey - 1) + "]"); + + numTables = 1; + if(cmd.hasOption(NUM_TABLES)) { + numTables = parseInt(cmd.getOptionValue(NUM_TABLES), 1, Short.MAX_VALUE); + } } - private void parseColumnFamilyOptions(CommandLine cmd) { + protected void parseColumnFamilyOptions(CommandLine cmd) { String dataBlockEncodingStr = cmd.getOptionValue(OPT_DATA_BLOCK_ENCODING); dataBlockEncodingAlgo = dataBlockEncodingStr == null ? null : DataBlockEncoding.valueOf(dataBlockEncodingStr); @@ -280,34 +342,56 @@ private void parseColumnFamilyOptions(CommandLine cmd) { } String compressStr = cmd.getOptionValue(OPT_COMPRESSION); - compressAlgo = compressStr == null ? null : + compressAlgo = compressStr == null ? Compression.Algorithm.NONE : Compression.Algorithm.valueOf(compressStr); String bloomStr = cmd.getOptionValue(OPT_BLOOM); bloomType = bloomStr == null ? null : StoreFile.BloomType.valueOf(bloomStr); + + inMemoryCF = cmd.hasOption(OPT_INMEMORY); + } + + public void initTestTable() throws IOException { + HBaseTestingUtility.createPreSplitLoadTestTable(conf, tableName, + COLUMN_FAMILY, compressAlgo, dataBlockEncodingAlgo); + applyColumnFamilyOptions(tableName, COLUMN_FAMILIES); } @Override - protected void doWork() throws IOException { + protected int doWork() throws IOException { + if (numTables > 1) { + return parallelLoadTables(); + } else { + return loadTable(); + } + } + + protected int loadTable() throws IOException { if (cmd.hasOption(OPT_ZK_QUORUM)) { conf.set(HConstants.ZOOKEEPER_QUORUM, cmd.getOptionValue(OPT_ZK_QUORUM)); } - HBaseTestingUtility.createPreSplitLoadTestTable(conf, tableName, - COLUMN_FAMILY, compressAlgo, dataBlockEncodingAlgo); - applyColumnFamilyOptions(tableName, COLUMN_FAMILIES); + if (isInitOnly) { + LOG.info("Initializing only; no reads or writes"); + initTestTable(); + return 0; + } + + if (!isSkipInit) { + initTestTable(); + } + + LoadTestDataGenerator dataGen = new MultiThreadedAction.DefaultDataGenerator( + minColDataSize, maxColDataSize, minColsPerKey, maxColsPerKey, COLUMN_FAMILY); if (isWrite) { - writerThreads = new MultiThreadedWriter(conf, tableName, COLUMN_FAMILY); + writerThreads = new MultiThreadedWriter(dataGen, conf, tableName); writerThreads.setMultiPut(isMultiPut); - writerThreads.setColumnsPerKey(minColsPerKey, maxColsPerKey); - writerThreads.setDataSize(minColDataSize, maxColDataSize); } if (isRead) { - readerThreads = new MultiThreadedReader(conf, tableName, COLUMN_FAMILY, - verifyPercent); + readerThreads = new MultiThreadedReader(dataGen, conf, tableName, verifyPercent); readerThreads.setMaxErrors(maxReadErrors); readerThreads.setKeyWindow(keyWindow); } @@ -335,10 +419,121 @@ protected void doWork() throws IOException { if (isRead) { readerThreads.waitForFinish(); } - } + boolean success = true; + if (isWrite) { + success = success && writerThreads.getNumWriteFailures() == 0; + } + if (isRead) { + success = success && readerThreads.getNumReadErrors() == 0 + && readerThreads.getNumReadFailures() == 0; + } + return success ? EXIT_SUCCESS : this.EXIT_FAILURE; + } + public static void main(String[] args) { new LoadTestTool().doStaticMain(args); } + /** + * When NUM_TABLES is specified, the function starts multiple worker threads + * which individually start a LoadTestTool instance to load a table. Each + * table name is in format _. For example, "-tn test -num_tables 2" + * , table names will be "test_1", "test_2" + * + * @throws IOException + */ + private int parallelLoadTables() + throws IOException { + // create new command args + String tableName = cmd.getOptionValue(OPT_TABLE_NAME, DEFAULT_TABLE_NAME); + String[] newArgs = null; + if (!cmd.hasOption(LoadTestTool.OPT_TABLE_NAME)) { + newArgs = new String[cmdLineArgs.length + 2]; + newArgs[0] = "-" + LoadTestTool.OPT_TABLE_NAME; + for (int i = 0; i < cmdLineArgs.length; i++) { + newArgs[i + 2] = cmdLineArgs[i]; + } + } else { + newArgs = cmdLineArgs; + } + + int tableNameValueIndex = -1; + for (int j = 0; j < newArgs.length; j++) { + if (newArgs[j].endsWith(OPT_TABLE_NAME)) { + tableNameValueIndex = j + 1; + } else if (newArgs[j].endsWith(NUM_TABLES)) { + // change NUM_TABLES to 1 so that each worker loads one table + newArgs[j + 1] = "1"; + } + } + + // starting to load multiple tables + List workers = new ArrayList(); + for (int i = 0; i < numTables; i++) { + String[] workerArgs = newArgs.clone(); + workerArgs[tableNameValueIndex] = tableName + "_" + (i+1); + WorkerThread worker = new WorkerThread(i, workerArgs); + workers.add(worker); + LOG.info(worker + " starting"); + worker.start(); + } + + // wait for all workers finish + LOG.info("Waiting for worker threads to finish"); + for (WorkerThread t : workers) { + try { + t.join(); + } catch (InterruptedException ie) { + IOException iie = new InterruptedIOException(); + iie.initCause(ie); + throw iie; + } + checkForErrors(); + } + + return EXIT_SUCCESS; + } + + // If an exception is thrown by one of worker threads, it will be + // stored here. + protected AtomicReference thrown = new AtomicReference(); + + private void workerThreadError(Throwable t) { + thrown.compareAndSet(null, t); + } + + /** + * Check for errors in the writer threads. If any is found, rethrow it. + */ + private void checkForErrors() throws IOException { + Throwable thrown = this.thrown.get(); + if (thrown == null) return; + if (thrown instanceof IOException) { + throw (IOException) thrown; + } else { + throw new RuntimeException(thrown); + } + } + + class WorkerThread extends Thread { + private String[] workerArgs; + + WorkerThread(int i, String[] args) { + super("WorkerThread-" + i); + workerArgs = args; + } + + public void run() { + try { + int ret = ToolRunner.run(HBaseConfiguration.create(), new LoadTestTool(), workerArgs); + if (ret != 0) { + throw new RuntimeException("LoadTestTool exit with non-zero return code."); + } + } catch (Exception ex) { + LOG.error("Error in worker thread", ex); + workerThreadError(ex); + } + } + } } diff --git a/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java b/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java index 967970d3d1f9..70a0e3d64711 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java +++ b/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java @@ -24,12 +24,16 @@ import java.util.concurrent.ConcurrentSkipListMap; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.hbase.HRegionInfo; import org.apache.hadoop.hbase.ServerName; import org.apache.hadoop.hbase.catalog.CatalogTracker; +import org.apache.hadoop.hbase.fs.HFileSystem; import org.apache.hadoop.hbase.ipc.RpcServer; import org.apache.hadoop.hbase.regionserver.CompactionRequestor; import org.apache.hadoop.hbase.regionserver.FlushRequester; import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.Leases; import org.apache.hadoop.hbase.regionserver.RegionServerAccounting; import org.apache.hadoop.hbase.regionserver.RegionServerServices; import org.apache.hadoop.hbase.regionserver.wal.HLog; @@ -45,6 +49,7 @@ public class MockRegionServerServices implements RegionServerServices { private boolean stopping = false; private final ConcurrentSkipListMap rit = new ConcurrentSkipListMap(Bytes.BYTES_COMPARATOR); + private HFileSystem hfs = null; @Override public boolean removeFromOnlineRegions(String encodedRegionName) { @@ -60,9 +65,6 @@ public List getOnlineRegions(byte[] tableName) throws IOException { return null; } - public void refreshRegion(HRegion hRegion) throws IOException { - } - @Override public void addToOnlineRegions(HRegion r) { this.regions.put(r.getRegionInfo().getEncodedName(), r); @@ -80,18 +82,18 @@ public boolean isStopping() { } @Override - public HLog getWAL() { + public HLog getWAL(HRegionInfo regionInfo) throws IOException { return null; } @Override - public RpcServer getRpcServer() { + public HLog getWAL() throws IOException { return null; } @Override - public ConcurrentSkipListMap getRegionsInTransitionInRS() { - return rit; + public RpcServer getRpcServer() { + return null; } @Override @@ -147,5 +149,30 @@ public boolean isStopped() { public boolean isAborted() { return false; } - + + @Override + public HFileSystem getFileSystem() { + return this.hfs; + } + + public void setFileSystem(FileSystem hfs) { + this.hfs = (HFileSystem)hfs; + } + + @Override + public Leases getLeases() { + return null; + } + + @Override + public boolean removeFromRegionsInTransition(HRegionInfo hri) { + // TODO Auto-generated method stub + return false; + } + + @Override + public boolean containsKeyInRegionsInTransition(HRegionInfo hri) { + // TODO Auto-generated method stub + return false; + } } diff --git a/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java b/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java index b312cca1d842..92eb11c86a72 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java +++ b/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java @@ -18,12 +18,16 @@ import java.io.IOException; import java.util.Collection; +import java.util.Map; +import java.util.Random; +import java.util.Set; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.util.StringUtils; /** @@ -34,7 +38,6 @@ public abstract class MultiThreadedAction { private static final Log LOG = LogFactory.getLog(MultiThreadedAction.class); protected final byte[] tableName; - protected final byte[] columnFamily; protected final Configuration conf; protected int numThreads = 1; @@ -51,8 +54,69 @@ public abstract class MultiThreadedAction { protected AtomicLong totalOpTimeMs = new AtomicLong(); protected boolean verbose = false; - protected int minDataSize = 256; - protected int maxDataSize = 1024; + protected LoadTestDataGenerator dataGenerator = null; + + /** + * Default implementation of LoadTestDataGenerator that uses LoadTestKVGenerator, fixed + * set of column families, and random number of columns in range. The table for it can + * be created manually or, for example, via + * {@link HBaseTestingUtility#createPreSplitLoadTestTable( + * org.apache.hadoop.hbase.Configuration, byte[], byte[], Algorithm, DataBlockEncoding)} + */ + public static class DefaultDataGenerator extends LoadTestDataGenerator { + private byte[][] columnFamilies = null; + private int minColumnsPerKey; + private int maxColumnsPerKey; + private final Random random = new Random(); + + public DefaultDataGenerator(int minValueSize, int maxValueSize, + int minColumnsPerKey, int maxColumnsPerKey, byte[]... columnFamilies) { + super(minValueSize, maxValueSize); + this.columnFamilies = columnFamilies; + this.minColumnsPerKey = minColumnsPerKey; + this.maxColumnsPerKey = maxColumnsPerKey; + } + + public DefaultDataGenerator(byte[]... columnFamilies) { + // Default values for tests that didn't care to provide theirs. + this(256, 1024, 1, 10, columnFamilies); + } + + @Override + public byte[] getDeterministicUniqueKey(long keyBase) { + return LoadTestKVGenerator.md5PrefixedKey(keyBase).getBytes(); + } + + @Override + public byte[][] getColumnFamilies() { + return columnFamilies; + } + + @Override + public byte[][] generateColumnsForCf(byte[] rowKey, byte[] cf) { + int numColumns = minColumnsPerKey + random.nextInt(maxColumnsPerKey - minColumnsPerKey + 1); + byte[][] columns = new byte[numColumns][]; + for (int i = 0; i < numColumns; ++i) { + columns[i] = Integer.toString(i).getBytes(); + } + return columns; + } + + @Override + public byte[] generateValue(byte[] rowKey, byte[] cf, byte[] column) { + return kvGenerator.generateRandomSizeValue(rowKey, cf, column); + } + + @Override + public boolean verify(byte[] rowKey, byte[] cf, byte[] column, byte[] value) { + return LoadTestKVGenerator.verify(value, rowKey, cf, column); + } + + @Override + public boolean verify(byte[] rowKey, byte[] cf, Set columnSet) { + return (columnSet.size() >= minColumnsPerKey) && (columnSet.size() <= maxColumnsPerKey); + } + } /** "R" or "W" */ private String actionLetter; @@ -62,11 +126,11 @@ public abstract class MultiThreadedAction { public static final int REPORTING_INTERVAL_MS = 5000; - public MultiThreadedAction(Configuration conf, byte[] tableName, - byte[] columnFamily, String actionLetter) { + public MultiThreadedAction(LoadTestDataGenerator dataGen, Configuration conf, byte[] tableName, + String actionLetter) { this.conf = conf; this.tableName = tableName; - this.columnFamily = columnFamily; + this.dataGenerator = dataGen; this.actionLetter = actionLetter; } @@ -165,17 +229,16 @@ private void printStreamingCounters(long numKeysDelta, } } - public void setDataSize(int minDataSize, int maxDataSize) { - this.minDataSize = minDataSize; - this.maxDataSize = maxDataSize; - } - public void waitForFinish() { while (numThreadsWorking.get() != 0) { Threads.sleepWithoutInterrupt(1000); } } + public boolean isDone() { + return (numThreadsWorking.get() == 0); + } + protected void startThreads(Collection threads) { numThreadsWorking.addAndGet(threads.size()); for (Thread thread : threads) { @@ -202,4 +265,86 @@ protected static void appendToStatus(StringBuilder sb, String desc, sb.append(v); } + protected static void appendToStatus(StringBuilder sb, String desc, + String v) { + sb.append(", "); + sb.append(desc); + sb.append("="); + sb.append(v); + } + + /** + * See {@link #verifyResultAgainstDataGenerator(Result, boolean, boolean)}. + * Does not verify cf/column integrity. + */ + public boolean verifyResultAgainstDataGenerator(Result result, boolean verifyValues) { + return verifyResultAgainstDataGenerator(result, verifyValues, false); + } + + /** + * Verifies the result from get or scan using the dataGenerator (that was presumably + * also used to generate said result). + * @param verifyValues verify that values in the result make sense for row/cf/column combination + * @param verifyCfAndColumnIntegrity verify that cf/column set in the result is complete. Note + * that to use this multiPut should be used, or verification + * has to happen after writes, otherwise there can be races. + * @return + */ + public boolean verifyResultAgainstDataGenerator(Result result, boolean verifyValues, + boolean verifyCfAndColumnIntegrity) { + String rowKeyStr = Bytes.toString(result.getRow()); + + // See if we have any data at all. + if (result.isEmpty()) { + LOG.error("No data returned for key = [" + rowKeyStr + "]"); + return false; + } + + if (!verifyValues && !verifyCfAndColumnIntegrity) { + return true; // as long as we have something, we are good. + } + + // See if we have all the CFs. + byte[][] expectedCfs = dataGenerator.getColumnFamilies(); + if (verifyCfAndColumnIntegrity && (expectedCfs.length != result.getMap().size())) { + LOG.error("Bad family count for [" + rowKeyStr + "]: " + result.getMap().size()); + return false; + } + + // Verify each column family from get in the result. + for (byte[] cf : result.getMap().keySet()) { + String cfStr = Bytes.toString(cf); + Map columnValues = result.getFamilyMap(cf); + if (columnValues == null) { + LOG.error("No data for family [" + cfStr + "] for [" + rowKeyStr + "]"); + return false; + } + // See if we have correct columns. + if (verifyCfAndColumnIntegrity + && !dataGenerator.verify(result.getRow(), cf, columnValues.keySet())) { + String colsStr = ""; + for (byte[] col : columnValues.keySet()) { + if (colsStr.length() > 0) { + colsStr += ", "; + } + colsStr += "[" + Bytes.toString(col) + "]"; + } + LOG.error("Bad columns for family [" + cfStr + "] for [" + rowKeyStr + "]: " + colsStr); + return false; + } + // See if values check out. + if (verifyValues) { + for (Map.Entry kv : columnValues.entrySet()) { + if (!dataGenerator.verify(result.getRow(), cf, kv.getKey(), kv.getValue())) { + LOG.error("Error checking data for key [" + rowKeyStr + "], column family [" + + cfStr + "], column [" + Bytes.toString(kv.getKey()) + "]; value of length " + + + kv.getValue().length); + return false; + } + } + } + } + return true; + } + } diff --git a/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java b/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java index a05a121bdb62..e42152f4b721 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java +++ b/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java @@ -18,7 +18,6 @@ import java.io.IOException; import java.util.HashSet; -import java.util.List; import java.util.Random; import java.util.Set; import java.util.concurrent.atomic.AtomicLong; @@ -27,14 +26,13 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HRegionLocation; -import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Result; /** Creates multiple threads that read and verify previously written data */ -public class MultiThreadedReader extends MultiThreadedAction -{ +public class MultiThreadedReader extends MultiThreadedAction { + private static final Log LOG = LogFactory.getLog(MultiThreadedReader.class); private Set readers = new HashSet(); @@ -71,9 +69,9 @@ public class MultiThreadedReader extends MultiThreadedAction private int maxErrors = DEFAULT_MAX_ERRORS; private int keyWindow = DEFAULT_KEY_WINDOW; - public MultiThreadedReader(Configuration conf, byte[] tableName, - byte[] columnFamily, double verifyPercent) { - super(conf, tableName, columnFamily, "R"); + public MultiThreadedReader(LoadTestDataGenerator dataGen, Configuration conf, + byte[] tableName, double verifyPercent) { + super(dataGen, conf, tableName, "R"); this.verifyPercent = verifyPercent; } @@ -222,14 +220,22 @@ private long getNextKeyToRead() { } private Get readKey(long keyToRead) { - Get get = new Get( - LoadTestKVGenerator.md5PrefixedKey(keyToRead).getBytes()); - get.addFamily(columnFamily); + Get get = new Get(dataGenerator.getDeterministicUniqueKey(keyToRead)); + String cfsString = ""; + byte[][] columnFamilies = dataGenerator.getColumnFamilies(); + for (byte[] cf : columnFamilies) { + get.addFamily(cf); + if (verbose) { + if (cfsString.length() > 0) { + cfsString += ", "; + } + cfsString += "[" + Bytes.toStringBinary(cf) + "]"; + } + } try { if (verbose) { - LOG.info("[" + readerId + "] " + "Querying key " + keyToRead - + ", cf " + Bytes.toStringBinary(columnFamily)); + LOG.info("[" + readerId + "] " + "Querying key " + keyToRead + ", cfs " + cfsString); } queryKey(get, random.nextInt(100) < verifyPercent); } catch (IOException e) { @@ -251,45 +257,37 @@ public void queryKey(Get get, boolean verify) throws IOException { numKeys.addAndGet(1); // if we got no data report error - if (result.isEmpty()) { - HRegionLocation hloc = table.getRegionLocation( - Bytes.toBytes(rowKey)); + if (!result.isEmpty()) { + if (verify) { + numKeysVerified.incrementAndGet(); + } + } else { + HRegionLocation hloc = table.getRegionLocation(Bytes.toBytes(rowKey)); LOG.info("Key = " + rowKey + ", RegionServer: " + hloc.getHostname()); - numReadErrors.addAndGet(1); - LOG.error("No data returned, tried to get actions for key = " - + rowKey + (writer == null ? "" : ", keys inserted by writer: " + - writer.numKeys.get() + ")")); - - if (numReadErrors.get() > maxErrors) { - LOG.error("Aborting readers -- found more than " + maxErrors - + " errors\n"); - aborted = true; - } } - if (result.getFamilyMap(columnFamily) != null) { - // increment number of columns read - numCols.addAndGet(result.getFamilyMap(columnFamily).size()); - - if (verify) { - // verify the result - List keyValues = result.list(); - for (KeyValue kv : keyValues) { - String qual = new String(kv.getQualifier()); - - // if something does not look right report it - if (!LoadTestKVGenerator.verify(rowKey, qual, kv.getValue())) { - numReadErrors.addAndGet(1); - LOG.error("Error checking data for key = " + rowKey - + ", actionId = " + qual); - } - } - numKeysVerified.addAndGet(1); + boolean isOk = verifyResultAgainstDataGenerator(result, verify); + long numErrorsAfterThis = 0; + if (isOk) { + long cols = 0; + // Count the columns for reporting purposes. + for (byte[] cf : result.getMap().keySet()) { + cols += result.getFamilyMap(cf).size(); + } + numCols.addAndGet(cols); + } else { + if (writer != null) { + LOG.error("At the time of failure, writer inserted " + writer.numKeys.get() + " keys"); } + numErrorsAfterThis = numReadErrors.incrementAndGet(); } - } + if (numErrorsAfterThis > maxErrors) { + LOG.error("Aborting readers -- found more than " + maxErrors + " errors"); + aborted = true; + } + } } public long getNumReadFailures() { diff --git a/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java b/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java index 4bfc2a959a13..ea372cc84050 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java +++ b/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java @@ -20,7 +20,6 @@ import java.util.HashSet; import java.util.PriorityQueue; import java.util.Queue; -import java.util.Random; import java.util.Set; import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.BlockingQueue; @@ -38,8 +37,6 @@ public class MultiThreadedWriter extends MultiThreadedAction { private static final Log LOG = LogFactory.getLog(MultiThreadedWriter.class); - private long minColumnsPerKey = 1; - private long maxColumnsPerKey = 10; private Set writers = new HashSet(); private boolean isMultiPut = false; @@ -50,8 +47,7 @@ public class MultiThreadedWriter extends MultiThreadedAction { * {@link #insertedUpToKey}, the maximum key in the contiguous range of keys * being inserted. This queue is supposed to stay small. */ - private BlockingQueue insertedKeys = - new ArrayBlockingQueue(10000); + private BlockingQueue insertedKeys = new ArrayBlockingQueue(10000); /** * This is the current key to be inserted by any thread. Each thread does an @@ -77,9 +73,9 @@ public class MultiThreadedWriter extends MultiThreadedAction { /** Enable this if used in conjunction with a concurrent reader. */ private boolean trackInsertedKeys; - public MultiThreadedWriter(Configuration conf, byte[] tableName, - byte[] columnFamily) { - super(conf, tableName, columnFamily, "W"); + public MultiThreadedWriter(LoadTestDataGenerator dataGen, Configuration conf, + byte[] tableName) { + super(dataGen, conf, tableName, "W"); } /** Use multi-puts vs. separate puts for every column in a row */ @@ -87,11 +83,6 @@ public void setMultiPut(boolean isMultiPut) { this.isMultiPut = isMultiPut; } - public void setColumnsPerKey(long minColumnsPerKey, long maxColumnsPerKey) { - this.minColumnsPerKey = minColumnsPerKey; - this.maxColumnsPerKey = maxColumnsPerKey; - } - @Override public void start(long startKey, long endKey, int numThreads) throws IOException { @@ -117,17 +108,9 @@ public void start(long startKey, long endKey, int numThreads) startThreads(writers); } - public static byte[] longToByteArrayKey(long rowKey) { - return LoadTestKVGenerator.md5PrefixedKey(rowKey).getBytes(); - } - private class HBaseWriterThread extends Thread { private final HTable table; - private final Random random = new Random(); - private final LoadTestKVGenerator dataGenerator = new LoadTestKVGenerator( - minDataSize, maxDataSize); - public HBaseWriterThread(int writerId) throws IOException { setName(getClass().getSimpleName() + "_" + writerId); table = new HTable(conf, tableName); @@ -135,20 +118,35 @@ public HBaseWriterThread(int writerId) throws IOException { public void run() { try { - long rowKey; - while ((rowKey = nextKeyToInsert.getAndIncrement()) < endKey) { - long numColumns = minColumnsPerKey + Math.abs(random.nextLong()) - % (maxColumnsPerKey - minColumnsPerKey); + long rowKeyBase; + byte[][] columnFamilies = dataGenerator.getColumnFamilies(); + while ((rowKeyBase = nextKeyToInsert.getAndIncrement()) < endKey) { + byte[] rowKey = dataGenerator.getDeterministicUniqueKey(rowKeyBase); + Put put = new Put(rowKey); numKeys.addAndGet(1); + int columnCount = 0; + for (byte[] cf : columnFamilies) { + byte[][] columns = dataGenerator.generateColumnsForCf(rowKey, cf); + for (byte[] column : columns) { + byte[] value = dataGenerator.generateValue(rowKey, cf, column); + put.add(cf, column, value); + ++columnCount; + if (!isMultiPut) { + insert(table, put, rowKeyBase); + numCols.addAndGet(1); + put = new Put(rowKey); + } + } + } if (isMultiPut) { - multiPutInsertKey(rowKey, 0, numColumns); - } else { - for (long col = 0; col < numColumns; ++col) { - insert(rowKey, col); + if (verbose) { + LOG.debug("Preparing put for key = [" + rowKey + "], " + columnCount + " columns"); } + insert(table, put, rowKeyBase); + numCols.addAndGet(columnCount); } if (trackInsertedKeys) { - insertedKeys.add(rowKey); + insertedKeys.add(rowKeyBase); } } } finally { @@ -160,55 +158,17 @@ public void run() { numThreadsWorking.decrementAndGet(); } } + } - public void insert(long rowKey, long col) { - Put put = new Put(longToByteArrayKey(rowKey)); - String colAsStr = String.valueOf(col); - put.add(columnFamily, Bytes.toBytes(colAsStr), - dataGenerator.generateRandomSizeValue(rowKey, colAsStr)); - try { - long start = System.currentTimeMillis(); - table.put(put); - numCols.addAndGet(1); - totalOpTimeMs.addAndGet(System.currentTimeMillis() - start); - } catch (IOException e) { - failedKeySet.add(rowKey); - LOG.error("Failed to insert: " + rowKey); - e.printStackTrace(); - } - } - - public void multiPutInsertKey(long rowKey, long startCol, long endCol) { - if (verbose) { - LOG.debug("Preparing put for key = " + rowKey + ", cols = [" - + startCol + ", " + endCol + ")"); - } - - if (startCol >= endCol) { - return; - } - - Put put = new Put(LoadTestKVGenerator.md5PrefixedKey( - rowKey).getBytes()); - byte[] columnQualifier; - byte[] value; - for (long i = startCol; i < endCol; ++i) { - String qualStr = String.valueOf(i); - columnQualifier = qualStr.getBytes(); - value = dataGenerator.generateRandomSizeValue(rowKey, qualStr); - put.add(columnFamily, columnQualifier, value); - } - - try { - long start = System.currentTimeMillis(); - table.put(put); - numCols.addAndGet(endCol - startCol); - totalOpTimeMs.addAndGet( - System.currentTimeMillis() - start); - } catch (IOException e) { - failedKeySet.add(rowKey); - e.printStackTrace(); - } + public void insert(HTable table, Put put, long keyBase) { + try { + long start = System.currentTimeMillis(); + table.put(put); + totalOpTimeMs.addAndGet(System.currentTimeMillis() - start); + } catch (IOException e) { + failedKeySet.add(keyBase); + LOG.error("Failed to insert: " + keyBase); + e.printStackTrace(); } } @@ -301,7 +261,7 @@ protected String progressInfo() { * key, which requires a blocking queue and a consumer thread. * @param enable whether to enable tracking the last inserted key */ - void setTrackInsertedKeys(boolean enable) { + public void setTrackInsertedKeys(boolean enable) { trackInsertedKeys = enable; } diff --git a/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java b/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java index 7b9f8b031d64..3b4b66c72571 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java +++ b/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java @@ -62,8 +62,8 @@ public class RestartMetaTest extends AbstractHBaseTool { private void loadData() throws IOException { long startKey = 0; long endKey = 100000; - long minColsPerKey = 5; - long maxColsPerKey = 15; + int minColsPerKey = 5; + int maxColsPerKey = 15; int minColDataSize = 256; int maxColDataSize = 256 * 3; int numThreads = 10; @@ -77,11 +77,10 @@ private void loadData() throws IOException { System.out.printf("Client Threads: %d\n", numThreads); // start the writers - MultiThreadedWriter writer = new MultiThreadedWriter(conf, TABLE_NAME, - LoadTestTool.COLUMN_FAMILY); + LoadTestDataGenerator dataGen = new MultiThreadedAction.DefaultDataGenerator( + minColDataSize, maxColDataSize, minColsPerKey, maxColsPerKey, LoadTestTool.COLUMN_FAMILY); + MultiThreadedWriter writer = new MultiThreadedWriter(dataGen, conf, TABLE_NAME); writer.setMultiPut(true); - writer.setColumnsPerKey(minColsPerKey, maxColsPerKey); - writer.setDataSize(minColDataSize, maxColDataSize); writer.start(startKey, endKey, numThreads); System.out.printf("Started loading data..."); writer.waitForFinish(); @@ -89,7 +88,7 @@ private void loadData() throws IOException { } @Override - protected void doWork() throws IOException { + protected int doWork() throws Exception { ProcessBasedLocalHBaseCluster hbaseCluster = new ProcessBasedLocalHBaseCluster(conf, hbaseHome, numRegionServers); @@ -130,6 +129,7 @@ protected void doWork() throws IOException { + Bytes.toStringBinary(result.getFamilyMap(HConstants.CATALOG_FAMILY) .get(HConstants.SERVER_QUALIFIER))); } + return 0; } @Override diff --git a/src/test/java/org/apache/hadoop/hbase/util/StoppableImplementation.java b/src/test/java/org/apache/hadoop/hbase/util/StoppableImplementation.java new file mode 100644 index 000000000000..51a22f3d2406 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/StoppableImplementation.java @@ -0,0 +1,40 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.util; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.hbase.Stoppable; + +/** + * A base implementation for a Stoppable service + */ +@InterfaceAudience.Private +public class StoppableImplementation implements Stoppable { + volatile boolean stopped = false; + + @Override + public void stop(String why) { + this.stopped = true; + } + + @Override + public boolean isStopped() { + return stopped; + } +} \ No newline at end of file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java b/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java index c55c4d4f87b7..979e33980a6f 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestByteBloomFilter.java b/src/test/java/org/apache/hadoop/hbase/util/TestByteBloomFilter.java index 2240018893fe..fd6bd8f0f488 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestByteBloomFilter.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestByteBloomFilter.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java b/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java index e7fea6c465f7..b49f696f1a04 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -27,6 +26,7 @@ import java.math.BigDecimal; import java.math.BigInteger; import java.util.Arrays; +import java.util.Random; import junit.framework.TestCase; import org.apache.hadoop.hbase.SmallTests; @@ -221,6 +221,31 @@ public void testBinarySearch() throws Exception { } } + public void testToStringBytesBinaryReversible() { + // let's run test with 1000 randomly generated byte arrays + Random rand = new Random(System.currentTimeMillis()); + byte[] randomBytes = new byte[1000]; + for (int i = 0; i < 1000; i++) { + rand.nextBytes(randomBytes); + verifyReversibleForBytes(randomBytes); + } + + + // some specific cases + verifyReversibleForBytes(new byte[] {}); + verifyReversibleForBytes(new byte[] {'\\', 'x', 'A', 'D'}); + verifyReversibleForBytes(new byte[] {'\\', 'x', 'A', 'D', '\\'}); + } + + private void verifyReversibleForBytes(byte[] originalBytes) { + String convertedString = Bytes.toStringBinary(originalBytes); + byte[] convertedBytes = Bytes.toBytesBinary(convertedString); + if (Bytes.compareTo(originalBytes, convertedBytes) != 0) { + fail("Not reversible for\nbyte[]: " + Arrays.toString(originalBytes) + + ",\nStringBinary: " + convertedString); + } + } + public void testStartsWith() { assertTrue(Bytes.startsWith(Bytes.toBytes("hello"), Bytes.toBytes("h"))); assertTrue(Bytes.startsWith(Bytes.toBytes("hello"), Bytes.toBytes(""))); @@ -290,6 +315,13 @@ public void testFixedSizeString() throws IOException { assertEquals("", Bytes.readStringFixedSize(dis, 9)); } + public void testToBytesBinaryTrailingBackslashes() throws Exception { + try { + Bytes.toBytesBinary("abc\\x00\\x01\\"); + } catch (StringIndexOutOfBoundsException ex) { + fail("Illegal string access: " + ex.getMessage()); + } + } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java b/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java index df783958b799..2e9bcb8d1ea8 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java b/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java new file mode 100644 index 000000000000..15533a6367db --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java @@ -0,0 +1,264 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.util; +// this is deliberately not in the o.a.h.h.regionserver package +// in order to make sure all required classes/method are available + +import java.io.IOException; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.NavigableSet; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HTable; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; +import org.apache.hadoop.hbase.coprocessor.ObserverContext; +import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment; +import org.apache.hadoop.hbase.regionserver.InternalScanner; +import org.apache.hadoop.hbase.regionserver.KeyValueScanner; +import org.apache.hadoop.hbase.regionserver.ScanType; +import org.apache.hadoop.hbase.regionserver.Store; +import org.apache.hadoop.hbase.regionserver.StoreScanner; +import org.apache.hadoop.hbase.regionserver.wal.WALEdit; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import static org.junit.Assert.*; + +@Category(MediumTests.class) +public class TestCoprocessorScanPolicy { + final Log LOG = LogFactory.getLog(getClass()); + protected final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static final byte[] F = Bytes.toBytes("fam"); + private static final byte[] Q = Bytes.toBytes("qual"); + private static final byte[] R = Bytes.toBytes("row"); + + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + Configuration conf = TEST_UTIL.getConfiguration(); + conf.setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, + ScanObserver.class.getName()); + TEST_UTIL.startMiniCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniCluster(); + } + + @Test + public void testBaseCases() throws Exception { + byte[] tableName = Bytes.toBytes("baseCases"); + HTable t = TEST_UTIL.createTable(tableName, F, 1); + // set the version override to 2 + Put p = new Put(R); + p.setAttribute("versions", new byte[]{}); + p.add(F, tableName, Bytes.toBytes(2)); + t.put(p); + + long now = EnvironmentEdgeManager.currentTimeMillis(); + + // insert 2 versions + p = new Put(R); + p.add(F, Q, now, Q); + t.put(p); + p = new Put(R); + p.add(F, Q, now+1, Q); + t.put(p); + Get g = new Get(R); + g.setMaxVersions(10); + Result r = t.get(g); + assertEquals(2, r.size()); + + TEST_UTIL.flush(tableName); + TEST_UTIL.compact(tableName, true); + + // both version are still visible even after a flush/compaction + g = new Get(R); + g.setMaxVersions(10); + r = t.get(g); + assertEquals(2, r.size()); + + // insert a 3rd version + p = new Put(R); + p.add(F, Q, now+2, Q); + t.put(p); + g = new Get(R); + g.setMaxVersions(10); + r = t.get(g); + // still only two version visible + assertEquals(2, r.size()); + + t.close(); + } + + @Test + public void testTTL() throws Exception { + byte[] tableName = Bytes.toBytes("testTTL"); + HTableDescriptor desc = new HTableDescriptor(tableName); + HColumnDescriptor hcd = new HColumnDescriptor(F) + .setMaxVersions(10) + .setTimeToLive(1); + desc.addFamily(hcd); + TEST_UTIL.getHBaseAdmin().createTable(desc); + HTable t = new HTable(new Configuration(TEST_UTIL.getConfiguration()), tableName); + long now = EnvironmentEdgeManager.currentTimeMillis(); + ManualEnvironmentEdge me = new ManualEnvironmentEdge(); + me.setValue(now); + EnvironmentEdgeManagerTestHelper.injectEdge(me); + // 2s in the past + long ts = now - 2000; + // Set the TTL override to 3s + Put p = new Put(R); + p.setAttribute("ttl", new byte[]{}); + p.add(F, tableName, Bytes.toBytes(3000L)); + t.put(p); + + p = new Put(R); + p.add(F, Q, ts, Q); + t.put(p); + p = new Put(R); + p.add(F, Q, ts+1, Q); + t.put(p); + + // these two should be expired but for the override + // (their ts was 2s in the past) + Get g = new Get(R); + g.setMaxVersions(10); + Result r = t.get(g); + // still there? + assertEquals(2, r.size()); + + TEST_UTIL.flush(tableName); + TEST_UTIL.compact(tableName, true); + + g = new Get(R); + g.setMaxVersions(10); + r = t.get(g); + // still there? + assertEquals(2, r.size()); + + // roll time forward 2s. + me.setValue(now + 2000); + // now verify that data eventually does expire + g = new Get(R); + g.setMaxVersions(10); + r = t.get(g); + // should be gone now + assertEquals(0, r.size()); + t.close(); + } + + public static class ScanObserver extends BaseRegionObserver { + private Map ttls = new HashMap(); + private Map versions = new HashMap(); + + // lame way to communicate with the coprocessor, + // since it is loaded by a different class loader + @Override + public void prePut(final ObserverContext c, final Put put, + final WALEdit edit, final boolean writeToWAL) throws IOException { + if (put.getAttribute("ttl") != null) { + KeyValue kv = put.getFamilyMap().values().iterator().next().get(0); + ttls.put(Bytes.toString(kv.getQualifier()), Bytes.toLong(kv.getValue())); + c.bypass(); + } else if (put.getAttribute("versions") != null) { + KeyValue kv = put.getFamilyMap().values().iterator().next().get(0); + versions.put(Bytes.toString(kv.getQualifier()), Bytes.toInt(kv.getValue())); + c.bypass(); + } + } + + @Override + public InternalScanner preFlushScannerOpen(final ObserverContext c, + Store store, KeyValueScanner memstoreScanner, InternalScanner s) throws IOException { + Long newTtl = ttls.get(store.getTableName()); + if (newTtl != null) { + System.out.println("PreFlush:" + newTtl); + } + Integer newVersions = versions.get(store.getTableName()); + Store.ScanInfo oldSI = store.getScanInfo(); + HColumnDescriptor family = store.getFamily(); + Store.ScanInfo scanInfo = new Store.ScanInfo(family.getName(), family.getMinVersions(), + newVersions == null ? family.getMaxVersions() : newVersions, + newTtl == null ? oldSI.getTtl() : newTtl, family.getKeepDeletedCells(), + oldSI.getTimeToPurgeDeletes(), oldSI.getComparator()); + Scan scan = new Scan(); + scan.setMaxVersions(newVersions == null ? oldSI.getMaxVersions() : newVersions); + return new StoreScanner(store, scanInfo, scan, Collections.singletonList(memstoreScanner), + ScanType.MINOR_COMPACT, store.getHRegion().getSmallestReadPoint(), + HConstants.OLDEST_TIMESTAMP); + } + + @Override + public InternalScanner preCompactScannerOpen(final ObserverContext c, + Store store, List scanners, ScanType scanType, + long earliestPutTs, InternalScanner s) throws IOException { + Long newTtl = ttls.get(store.getTableName()); + Integer newVersions = versions.get(store.getTableName()); + Store.ScanInfo oldSI = store.getScanInfo(); + HColumnDescriptor family = store.getFamily(); + Store.ScanInfo scanInfo = new Store.ScanInfo(family.getName(), family.getMinVersions(), + newVersions == null ? family.getMaxVersions() : newVersions, + newTtl == null ? oldSI.getTtl() : newTtl, family.getKeepDeletedCells(), + oldSI.getTimeToPurgeDeletes(), oldSI.getComparator()); + Scan scan = new Scan(); + scan.setMaxVersions(newVersions == null ? oldSI.getMaxVersions() : newVersions); + return new StoreScanner(store, scanInfo, scan, scanners, scanType, store.getHRegion() + .getSmallestReadPoint(), earliestPutTs); + } + + @Override + public KeyValueScanner preStoreScannerOpen( + final ObserverContext c, Store store, final Scan scan, + final NavigableSet targetCols, KeyValueScanner s) throws IOException { + Long newTtl = ttls.get(store.getTableName()); + Integer newVersions = versions.get(store.getTableName()); + Store.ScanInfo oldSI = store.getScanInfo(); + HColumnDescriptor family = store.getFamily(); + Store.ScanInfo scanInfo = new Store.ScanInfo(family.getName(), family.getMinVersions(), + newVersions == null ? family.getMaxVersions() : newVersions, + newTtl == null ? oldSI.getTtl() : newTtl, family.getKeepDeletedCells(), + oldSI.getTimeToPurgeDeletes(), oldSI.getComparator()); + return new StoreScanner(store, scanInfo, scan, targetCols); + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestDefaultEnvironmentEdge.java b/src/test/java/org/apache/hadoop/hbase/util/TestDefaultEnvironmentEdge.java index ea6903e8f48a..2a29c2b1295c 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestDefaultEnvironmentEdge.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestDefaultEnvironmentEdge.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java b/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java new file mode 100644 index 000000000000..de2f77ef7cc1 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java @@ -0,0 +1,121 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.util; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.fail; + +import java.io.File; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.SmallTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test TestDynamicClassLoader + */ +@Category(SmallTests.class) +public class TestDynamicClassLoader { + private static final Log LOG = LogFactory.getLog(TestDynamicClassLoader.class); + + private static final Configuration conf = HBaseConfiguration.create(); + + private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + static { + conf.set("hbase.dynamic.jars.dir", TEST_UTIL.getDataTestDir().toString()); + } + + @Test + public void testLoadClassFromLocalPath() throws Exception { + ClassLoader parent = TestDynamicClassLoader.class.getClassLoader(); + DynamicClassLoader classLoader = new DynamicClassLoader(conf, parent); + + String className = "TestLoadClassFromLocalPath"; + deleteClass(className); + try { + classLoader.loadClass(className); + fail("Should not be able to load class " + className); + } catch (ClassNotFoundException cnfe) { + // expected, move on + } + + try { + String folder = TEST_UTIL.getDataTestDir().toString(); + ClassLoaderTestHelper.buildJar(folder, className, null, localDirPath()); + classLoader.loadClass(className); + } catch (ClassNotFoundException cnfe) { + LOG.error("Should be able to load class " + className, cnfe); + fail(cnfe.getMessage()); + } + } + + @Test + public void testLoadClassFromAnotherPath() throws Exception { + ClassLoader parent = TestDynamicClassLoader.class.getClassLoader(); + DynamicClassLoader classLoader = new DynamicClassLoader(conf, parent); + + String className = "TestLoadClassFromAnotherPath"; + deleteClass(className); + try { + classLoader.loadClass(className); + fail("Should not be able to load class " + className); + } catch (ClassNotFoundException cnfe) { + // expected, move on + } + + try { + String folder = TEST_UTIL.getDataTestDir().toString(); + ClassLoaderTestHelper.buildJar(folder, className, null); + classLoader.loadClass(className); + } catch (ClassNotFoundException cnfe) { + LOG.error("Should be able to load class " + className, cnfe); + fail(cnfe.getMessage()); + } + } + + private String localDirPath() { + return conf.get("hbase.local.dir") + + File.separator + "jars" + File.separator; + } + + private void deleteClass(String className) throws Exception { + String jarFileName = className + ".jar"; + File file = new File(TEST_UTIL.getDataTestDir().toString(), jarFileName); + file.delete(); + assertFalse("Should be deleted: " + file.getPath(), file.exists()); + + file = new File(conf.get("hbase.dynamic.jars.dir"), jarFileName); + file.delete(); + assertFalse("Should be deleted: " + file.getPath(), file.exists()); + + file = new File(localDirPath(), jarFileName); + file.delete(); + assertFalse("Should be deleted: " + file.getPath(), file.exists()); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestEnvironmentEdgeManager.java b/src/test/java/org/apache/hadoop/hbase/util/TestEnvironmentEdgeManager.java index 59b2c2bc5a24..bce0b94ead98 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestEnvironmentEdgeManager.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestEnvironmentEdgeManager.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestFSHDFSUtils.java b/src/test/java/org/apache/hadoop/hbase/util/TestFSHDFSUtils.java new file mode 100644 index 000000000000..dbe0a09a34d2 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/TestFSHDFSUtils.java @@ -0,0 +1,156 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.util; + +import static org.junit.Assert.assertTrue; + +import java.io.IOException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hdfs.DistributedFileSystem; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +/** + * Test our recoverLease loop against mocked up filesystem. + */ +@Category(MediumTests.class) +public class TestFSHDFSUtils { + private static final Log LOG = LogFactory.getLog(TestFSHDFSUtils.class); + private static final HBaseTestingUtility HTU = new HBaseTestingUtility(); + static { + Configuration conf = HTU.getConfiguration(); + conf.setInt("hbase.lease.recovery.first.pause", 10); + conf.setInt("hbase.lease.recovery.pause", 10); + }; + private FSHDFSUtils fsHDFSUtils = new FSHDFSUtils(); + private static Path FILE = new Path(HTU.getDataTestDir(), "file.txt"); + long startTime = -1; + + @Before + public void setup() { + this.startTime = EnvironmentEdgeManager.currentTimeMillis(); + } + + /** + * Test recover lease eventually succeeding. + * @throws IOException + */ + @Test (timeout = 30000) + public void testRecoverLease() throws IOException { + HTU.getConfiguration().setInt("hbase.lease.recovery.dfs.timeout", 1000); + DistributedFileSystem dfs = Mockito.mock(DistributedFileSystem.class); + // Fail four times and pass on the fifth. + Mockito.when(dfs.recoverLease(FILE)). + thenReturn(false).thenReturn(false).thenReturn(false).thenReturn(false).thenReturn(true); + assertTrue(this.fsHDFSUtils.recoverDFSFileLease(dfs, FILE, HTU.getConfiguration())); + Mockito.verify(dfs, Mockito.times(5)).recoverLease(FILE); + // Make sure we waited at least hbase.lease.recovery.dfs.timeout * 3 (the first two + // invocations will happen pretty fast... the we fall into the longer wait loop). + assertTrue((EnvironmentEdgeManager.currentTimeMillis() - this.startTime) > + (3 * HTU.getConfiguration().getInt("hbase.lease.recovery.dfs.timeout", 61000))); + } + + /** + * Test that isFileClosed makes us recover lease faster. + * @throws IOException + */ + @Test (timeout = 30000) + public void testIsFileClosed() throws IOException { + // Make this time long so it is plain we broke out because of the isFileClosed invocation. + HTU.getConfiguration().setInt("hbase.lease.recovery.dfs.timeout", 100000); + IsFileClosedDistributedFileSystem dfs = Mockito.mock(IsFileClosedDistributedFileSystem.class); + // Now make it so we fail the first two times -- the two fast invocations, then we fall into + // the long loop during which we will call isFileClosed.... the next invocation should + // therefore return true if we are to break the loop. + Mockito.when(dfs.recoverLease(FILE)). + thenReturn(false).thenReturn(false).thenReturn(true); + Mockito.when(dfs.isFileClosed(FILE)).thenReturn(true); + assertTrue(this.fsHDFSUtils.recoverDFSFileLease(dfs, FILE, HTU.getConfiguration())); + Mockito.verify(dfs, Mockito.times(2)).recoverLease(FILE); + Mockito.verify(dfs, Mockito.times(1)).isFileClosed(FILE); + } + + @Test + public void testIsSameHdfs() throws IOException { + try { + Class dfsUtilClazz = Class.forName("org.apache.hadoop.hdfs.DFSUtil"); + dfsUtilClazz.getMethod("getNNServiceRpcAddresses", Configuration.class); + } catch (Exception e) { + LOG.info("Skip testIsSameHdfs test case because of the no-HA hadoop version."); + return; + } + + Configuration conf = HBaseConfiguration.create(); + Path srcPath = new Path("hdfs://localhost:8020/"); + Path desPath = new Path("hdfs://127.0.0.1/"); + FileSystem srcFs = srcPath.getFileSystem(conf); + FileSystem desFs = desPath.getFileSystem(conf); + + assertTrue(FSHDFSUtils.isSameHdfs(conf, srcFs, desFs)); + + desPath = new Path("hdfs://127.0.0.1:8070/"); + desFs = desPath.getFileSystem(conf); + assertTrue(!FSHDFSUtils.isSameHdfs(conf, srcFs, desFs)); + + desPath = new Path("hdfs://127.0.1.1:8020/"); + desFs = desPath.getFileSystem(conf); + assertTrue(!FSHDFSUtils.isSameHdfs(conf, srcFs, desFs)); + + conf.set("fs.defaultFS", "hdfs://haosong-hadoop"); + conf.set("dfs.nameservices", "haosong-hadoop"); + conf.set("dfs.federation.nameservices", "haosong-hadoop"); + conf.set("dfs.ha.namenodes.haosong-hadoop", "nn1,nn2"); + conf.set("dfs.client.failover.proxy.provider.haosong-hadoop", + "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"); + + conf.set("dfs.namenode.rpc-address.haosong-hadoop.nn1", "127.0.0.1:8020"); + conf.set("dfs.namenode.rpc-address.haosong-hadoop.nn2", "127.10.2.1:8000"); + desPath = new Path("/"); + desFs = desPath.getFileSystem(conf); + assertTrue(FSHDFSUtils.isSameHdfs(conf, srcFs, desFs)); + + conf.set("dfs.namenode.rpc-address.haosong-hadoop.nn1", "127.10.2.1:8020"); + conf.set("dfs.namenode.rpc-address.haosong-hadoop.nn2", "127.0.0.1:8000"); + desPath = new Path("/"); + desFs = desPath.getFileSystem(conf); + assertTrue(!FSHDFSUtils.isSameHdfs(conf, srcFs, desFs)); + } + + /** + * Version of DFS that has HDFS-4525 in it. + */ + class IsFileClosedDistributedFileSystem extends DistributedFileSystem { + /** + * Close status of a file. Copied over from HDFS-4525 + * @return true if file is already closed + **/ + public boolean isFileClosed(Path f) throws IOException{ + return false; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java b/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java index 0db4d425f73e..c5dda2f216e1 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java @@ -54,7 +54,7 @@ public void testRegexAgainstOldStyleTableInfo() { @Test public void testCreateAndUpdate() throws IOException { - Path testdir = UTIL.getDataTestDir(); + Path testdir = UTIL.getDataTestDir("testCreate"); HTableDescriptor htd = new HTableDescriptor("testCreate"); FileSystem fs = FileSystem.get(UTIL.getConfiguration()); assertTrue(FSTableDescriptors.createTableDescriptor(fs, testdir, htd)); @@ -252,6 +252,19 @@ public void testTableInfoFileStatusComparator() { } } + @Test + public void testReadingArchiveDirectoryFromFS() throws IOException { + FileSystem fs = FileSystem.get(UTIL.getConfiguration()); + try { + new FSTableDescriptors(fs, FSUtils.getRootDir(UTIL.getConfiguration())) + .get(HConstants.HFILE_ARCHIVE_DIRECTORY); + fail("Shouldn't be able to read a table descriptor for the archive directory."); + } catch (IOException e) { + LOG.debug("Correctly got error when reading a table descriptor from the archive directory: " + + e.getMessage()); + } + } + @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java b/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java index e2611e607d23..853ff032078b 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -19,14 +18,24 @@ */ package org.apache.hadoop.hbase.util; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; -import org.apache.hadoop.hbase.*; +import java.io.File; +import java.util.UUID; + +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataOutputStream; -import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HDFSBlocksDistribution; +import org.apache.hadoop.hbase.MediumTests; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -80,20 +89,26 @@ private void WriteDataToHDFS(FileSystem fs, Path file, int dataSize) // given the default replication factor is 3, the same as the number of // datanodes; the locality index for each host should be 100%, // or getWeight for each host should be the same as getUniqueBlocksWeights - FileStatus status = fs.getFileStatus(testFile); - HDFSBlocksDistribution blocksDistribution = - FSUtils.computeHDFSBlocksDistribution(fs, status, 0, status.getLen()); - long uniqueBlocksTotalWeight = - blocksDistribution.getUniqueBlocksTotalWeight(); - for (String host : hosts) { - long weight = blocksDistribution.getWeight(host); - assertTrue(uniqueBlocksTotalWeight == weight); - } - } finally { + final long maxTime = System.currentTimeMillis() + 2000; + boolean ok; + do { + ok = true; + FileStatus status = fs.getFileStatus(testFile); + HDFSBlocksDistribution blocksDistribution = + FSUtils.computeHDFSBlocksDistribution(fs, status, 0, status.getLen()); + long uniqueBlocksTotalWeight = + blocksDistribution.getUniqueBlocksTotalWeight(); + for (String host : hosts) { + long weight = blocksDistribution.getWeight(host); + ok = (ok && uniqueBlocksTotalWeight == weight); + } + } while (!ok && System.currentTimeMillis() < maxTime); + assertTrue(ok); + } finally { htu.shutdownMiniDFSCluster(); } - + try { // set up a cluster with 4 nodes String hosts[] = new String[] { "host1", "host2", "host3", "host4" }; @@ -108,16 +123,22 @@ private void WriteDataToHDFS(FileSystem fs, Path file, int dataSize) // given the default replication factor is 3, we will have total of 9 // replica of blocks; thus the host with the highest weight should have // weight == 3 * DEFAULT_BLOCK_SIZE - FileStatus status = fs.getFileStatus(testFile); - HDFSBlocksDistribution blocksDistribution = - FSUtils.computeHDFSBlocksDistribution(fs, status, 0, status.getLen()); - long uniqueBlocksTotalWeight = - blocksDistribution.getUniqueBlocksTotalWeight(); - - String tophost = blocksDistribution.getTopHosts().get(0); - long weight = blocksDistribution.getWeight(tophost); + final long maxTime = System.currentTimeMillis() + 2000; + long weight; + long uniqueBlocksTotalWeight; + do { + FileStatus status = fs.getFileStatus(testFile); + HDFSBlocksDistribution blocksDistribution = + FSUtils.computeHDFSBlocksDistribution(fs, status, 0, status.getLen()); + uniqueBlocksTotalWeight = blocksDistribution.getUniqueBlocksTotalWeight(); + + String tophost = blocksDistribution.getTopHosts().get(0); + weight = blocksDistribution.getWeight(tophost); + + // NameNode is informed asynchronously, so we may have a delay. See HBASE-6175 + } while (uniqueBlocksTotalWeight != weight && System.currentTimeMillis() < maxTime); assertTrue(uniqueBlocksTotalWeight == weight); - + } finally { htu.shutdownMiniDFSCluster(); } @@ -136,16 +157,79 @@ private void WriteDataToHDFS(FileSystem fs, Path file, int dataSize) // given the default replication factor is 3, we will have total of 3 // replica of blocks; thus there is one host without weight - FileStatus status = fs.getFileStatus(testFile); - HDFSBlocksDistribution blocksDistribution = - FSUtils.computeHDFSBlocksDistribution(fs, status, 0, status.getLen()); - assertTrue(blocksDistribution.getTopHosts().size() == 3); + final long maxTime = System.currentTimeMillis() + 2000; + HDFSBlocksDistribution blocksDistribution; + do { + FileStatus status = fs.getFileStatus(testFile); + blocksDistribution = FSUtils.computeHDFSBlocksDistribution(fs, status, 0, status.getLen()); + // NameNode is informed asynchronously, so we may have a delay. See HBASE-6175 + } + while (blocksDistribution.getTopHosts().size() != 3 && System.currentTimeMillis() < maxTime); + assertEquals("Wrong number of hosts distributing blocks.", 3, + blocksDistribution.getTopHosts().size()); } finally { htu.shutdownMiniDFSCluster(); } - + } + + @Test + public void testPermMask() throws Exception { + + Configuration conf = HBaseConfiguration.create(); + conf.setBoolean(HConstants.ENABLE_DATA_FILE_UMASK, true); + FileSystem fs = FileSystem.get(conf); + // first check that we don't crash if we don't have perms set + FsPermission defaultPerms = FSUtils.getFilePermissions(fs, conf, + HConstants.DATA_FILE_UMASK_KEY); + assertEquals(FsPermission.getDefault(), defaultPerms); + + conf.setStrings(HConstants.DATA_FILE_UMASK_KEY, "077"); + // now check that we get the right perms + FsPermission filePerm = FSUtils.getFilePermissions(fs, conf, + HConstants.DATA_FILE_UMASK_KEY); + assertEquals(new FsPermission("700"), filePerm); + + // then that the correct file is created + Path p = new Path("target" + File.separator + UUID.randomUUID().toString()); + try { + FSDataOutputStream out = FSUtils.create(fs, p, filePerm); + out.close(); + FileStatus stat = fs.getFileStatus(p); + assertEquals(new FsPermission("700"), stat.getPermission()); + // and then cleanup + } finally { + fs.delete(p, true); + } } + @Test + public void testDeleteAndExists() throws Exception { + Configuration conf = HBaseConfiguration.create(); + conf.setBoolean(HConstants.ENABLE_DATA_FILE_UMASK, true); + FileSystem fs = FileSystem.get(conf); + FsPermission perms = FSUtils.getFilePermissions(fs, conf, HConstants.DATA_FILE_UMASK_KEY); + // then that the correct file is created + String file = UUID.randomUUID().toString(); + Path p = new Path("temptarget" + File.separator + file); + Path p1 = new Path("temppath" + File.separator + file); + try { + FSDataOutputStream out = FSUtils.create(fs, p, perms); + out.close(); + assertTrue("The created file should be present", FSUtils.isExists(fs, p)); + // delete the file with recursion as false. Only the file will be deleted. + FSUtils.delete(fs, p, false); + // Create another file + FSDataOutputStream out1 = FSUtils.create(fs, p1, perms); + out1.close(); + // delete the file with recursion as false. Still the file only will be deleted + FSUtils.delete(fs, p1, true); + assertFalse("The created file should be present", FSUtils.isExists(fs, p1)); + // and then cleanup + } finally { + FSUtils.delete(fs, p, true); + FSUtils.delete(fs, p1, true); + } + } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java b/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java new file mode 100644 index 000000000000..c2c95b153365 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java @@ -0,0 +1,225 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.util; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.IOException; +import java.util.UUID; +import java.util.Set; +import java.util.HashSet; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataInputStream; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HDFSBlocksDistribution; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.regionserver.wal.HLog; +import org.apache.hadoop.hbase.util.MD5Hash; +import org.apache.hadoop.hbase.util.FSUtils; +import org.junit.*; +import org.junit.experimental.categories.Category; + +/** + * Test {@link FSUtils}. + */ +@Category(MediumTests.class) +public class TestFSVisitor { + final Log LOG = LogFactory.getLog(getClass()); + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + private final String TABLE_NAME = "testtb"; + + private Set tableFamilies; + private Set tableRegions; + private Set recoveredEdits; + private Set tableHFiles; + private Set regionServers; + private Set serverLogs; + + private FileSystem fs; + private Path tableDir; + private Path logsDir; + private Path rootDir; + + @Before + public void setUp() throws Exception { + fs = FileSystem.get(TEST_UTIL.getConfiguration()); + rootDir = TEST_UTIL.getDataTestDir("hbase"); + logsDir = new Path(rootDir, HConstants.HREGION_LOGDIR_NAME); + + tableFamilies = new HashSet(); + tableRegions = new HashSet(); + recoveredEdits = new HashSet(); + tableHFiles = new HashSet(); + regionServers = new HashSet(); + serverLogs = new HashSet(); + tableDir = createTableFiles(rootDir, TABLE_NAME, tableRegions, tableFamilies, tableHFiles); + createRecoverEdits(tableDir, tableRegions, recoveredEdits); + createLogs(logsDir, regionServers, serverLogs); + FSUtils.logFileSystemState(fs, rootDir, LOG); + } + + @After + public void tearDown() throws Exception { + fs.delete(rootDir); + } + + @Test + public void testVisitStoreFiles() throws IOException { + final Set regions = new HashSet(); + final Set families = new HashSet(); + final Set hfiles = new HashSet(); + FSVisitor.visitTableStoreFiles(fs, tableDir, new FSVisitor.StoreFileVisitor() { + public void storeFile(final String region, final String family, final String hfileName) + throws IOException { + regions.add(region); + families.add(family); + hfiles.add(hfileName); + } + }); + assertEquals(tableRegions, regions); + assertEquals(tableFamilies, families); + assertEquals(tableHFiles, hfiles); + } + + @Test + public void testVisitRecoveredEdits() throws IOException { + final Set regions = new HashSet(); + final Set edits = new HashSet(); + FSVisitor.visitTableRecoveredEdits(fs, tableDir, new FSVisitor.RecoveredEditsVisitor() { + public void recoveredEdits (final String region, final String logfile) + throws IOException { + regions.add(region); + edits.add(logfile); + } + }); + assertEquals(tableRegions, regions); + assertEquals(recoveredEdits, edits); + } + + @Test + public void testVisitLogFiles() throws IOException { + final Set servers = new HashSet(); + final Set logs = new HashSet(); + FSVisitor.visitLogFiles(fs, rootDir, new FSVisitor.LogFileVisitor() { + public void logFile (final String server, final String logfile) throws IOException { + servers.add(server); + logs.add(logfile); + } + }); + assertEquals(regionServers, servers); + assertEquals(serverLogs, logs); + } + + + /* + * |-testtb/ + * |----f1d3ff8443297732862df21dc4e57262/ + * |-------f1/ + * |----------d0be84935ba84b66b1e866752ec5d663 + * |----------9fc9d481718f4878b29aad0a597ecb94 + * |-------f2/ + * |----------4b0fe6068c564737946bcf4fd4ab8ae1 + */ + private Path createTableFiles(final Path rootDir, final String tableName, + final Set tableRegions, final Set tableFamilies, + final Set tableHFiles) throws IOException { + Path tableDir = new Path(rootDir, tableName); + for (int r = 0; r < 10; ++r) { + String regionName = MD5Hash.getMD5AsHex(Bytes.toBytes(r)); + tableRegions.add(regionName); + Path regionDir = new Path(tableDir, regionName); + for (int f = 0; f < 3; ++f) { + String familyName = "f" + f; + tableFamilies.add(familyName); + Path familyDir = new Path(regionDir, familyName); + fs.mkdirs(familyDir); + for (int h = 0; h < 5; ++h) { + String hfileName = UUID.randomUUID().toString().replaceAll("-", ""); + tableHFiles.add(hfileName); + fs.createNewFile(new Path(familyDir, hfileName)); + } + } + } + return tableDir; + } + + /* + * |-testtb/ + * |----f1d3ff8443297732862df21dc4e57262/ + * |-------recovered.edits/ + * |----------0000001351969633479 + * |----------0000001351969633481 + */ + private void createRecoverEdits(final Path tableDir, final Set tableRegions, + final Set recoverEdits) throws IOException { + for (String region: tableRegions) { + Path regionEditsDir = HLog.getRegionDirRecoveredEditsDir(new Path(tableDir, region)); + long seqId = System.currentTimeMillis(); + for (int i = 0; i < 3; ++i) { + String editName = String.format("%019d", seqId + i); + recoverEdits.add(editName); + FSDataOutputStream stream = fs.create(new Path(regionEditsDir, editName)); + stream.write(Bytes.toBytes("test")); + stream.close(); + } + } + } + + /* + * |-.logs/ + * |----server5,5,1351969633508/ + * |-------server5,5,1351969633508.0 + * |----server6,6,1351969633512/ + * |-------server6,6,1351969633512.0 + * |-------server6,6,1351969633512.3 + */ + private void createLogs(final Path logDir, final Set servers, + final Set logs) throws IOException { + for (int s = 0; s < 7; ++s) { + String server = String.format("server%d,%d,%d", s, s, System.currentTimeMillis()); + servers.add(server); + Path serverLogDir = new Path(logDir, server); + fs.mkdirs(serverLogDir); + for (int i = 0; i < 5; ++i) { + String logfile = server + '.' + i; + logs.add(logfile); + FSDataOutputStream stream = fs.create(new Path(serverLogDir, logfile)); + stream.write(Bytes.toBytes("test")); + stream.close(); + } + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java b/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java index 937781d56cac..ea7501a902d4 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -23,51 +22,104 @@ import static org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.assertNoErrors; import static org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Map.Entry; +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import org.apache.commons.io.IOUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.ClusterStatus; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HColumnDescriptor; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.HRegionLocation; +import org.apache.hadoop.hbase.HTableDescriptor; +import org.apache.hadoop.hbase.LargeTests; +import org.apache.hadoop.hbase.MiniHBaseCluster; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.catalog.MetaReader; import org.apache.hadoop.hbase.client.Delete; +import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HBaseAdmin; +import org.apache.hadoop.hbase.client.HConnection; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.executor.EventHandler.EventType; +import org.apache.hadoop.hbase.executor.RegionTransitionData; +import org.apache.hadoop.hbase.io.hfile.TestHFile; +import org.apache.hadoop.hbase.ipc.HRegionInterface; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.HRegionServer; +import org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction; +import org.apache.hadoop.hbase.util.HBaseFsck.ErrorReporter; import org.apache.hadoop.hbase.util.HBaseFsck.ErrorReporter.ERROR_CODE; +import org.apache.hadoop.hbase.util.HBaseFsck.HbckInfo; +import org.apache.hadoop.hbase.util.HBaseFsck.PrintingErrorReporter; +import org.apache.hadoop.hbase.util.HBaseFsck.TableInfo; +import org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker; +import org.apache.hadoop.hbase.util.hbck.HbckTestingUtil; +import org.apache.hadoop.hbase.zookeeper.ZKAssign; +import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher; import org.apache.zookeeper.KeeperException; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; +import org.junit.rules.TestName; + +import com.google.common.collect.Multimap; /** * This tests HBaseFsck's ability to detect reasons for inconsistent tables. */ -@Category(MediumTests.class) +@Category(LargeTests.class) public class TestHBaseFsck { - final Log LOG = LogFactory.getLog(getClass()); + final static Log LOG = LogFactory.getLog(TestHBaseFsck.class); private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); private final static Configuration conf = TEST_UTIL.getConfiguration(); - private final static byte[] FAM = Bytes.toBytes("fam"); + private final static String FAM_STR = "fam"; + private final static byte[] FAM = Bytes.toBytes(FAM_STR); + private final static int REGION_ONLINE_TIMEOUT = 800; // for the instance, reset every test run private HTable tbl; - private final static byte[][] splits= new byte[][] { Bytes.toBytes("A"), + private final static byte[][] SPLITS = new byte[][] { Bytes.toBytes("A"), Bytes.toBytes("B"), Bytes.toBytes("C") }; - + // one row per region. + private final static byte[][] ROWKEYS= new byte[][] { + Bytes.toBytes("00"), Bytes.toBytes("50"), Bytes.toBytes("A0"), Bytes.toBytes("A5"), + Bytes.toBytes("B0"), Bytes.toBytes("B5"), Bytes.toBytes("C0"), Bytes.toBytes("C5") }; + @BeforeClass public static void setUpBeforeClass() throws Exception { TEST_UTIL.getConfiguration().setBoolean("hbase.master.distributed.log.splitting", false); TEST_UTIL.startMiniCluster(3); + TEST_UTIL.setHDFSClientRetry(0); } @AfterClass @@ -117,8 +169,8 @@ public void testHBaseFsck() throws Exception { assertErrors(doFsck(conf, true), new ERROR_CODE[]{ ERROR_CODE.SERVER_DOES_NOT_MATCH_META}); - // fixing assignements require opening regions is not synchronous. To make - // the test pass consistentyl so for now we bake in some sleep to let it + // fixing assignments require opening regions is not synchronous. To make + // the test pass consistently so for now we bake in some sleep to let it // finish. 1s seems sufficient. Thread.sleep(1000); @@ -135,6 +187,9 @@ public void testHBaseFsck() throws Exception { meta.close(); } + /** + * Create a new region in META. + */ private HRegionInfo createRegion(Configuration conf, final HTableDescriptor htd, byte[] startKey, byte[] endKey) throws IOException { @@ -147,47 +202,102 @@ private HRegionInfo createRegion(Configuration conf, final HTableDescriptor return hri; } - public void dumpMeta(HTableDescriptor htd) throws IOException { - List metaRows = TEST_UTIL.getMetaTableRows(htd.getName()); + /** + * Debugging method to dump the contents of meta. + */ + private void dumpMeta(byte[] tableName) throws IOException { + List metaRows = TEST_UTIL.getMetaTableRows(tableName); for (byte[] row : metaRows) { LOG.info(Bytes.toString(row)); } } - private void deleteRegion(Configuration conf, final HTableDescriptor htd, - byte[] startKey, byte[] endKey) throws IOException { + /** + * This method is used to undeploy a region -- close it and attempt to + * remove its state from the Master. + */ + private void undeployRegion(HBaseAdmin admin, ServerName sn, + HRegionInfo hri) throws IOException, InterruptedException { + try { + HBaseFsckRepair.closeRegionSilentlyAndWait(admin, sn, hri); + admin.getMaster().offline(hri.getRegionName()); + } catch (IOException ioe) { + LOG.warn("Got exception when attempting to offline region " + + Bytes.toString(hri.getRegionName()), ioe); + } + } + /** + * Delete a region from assignments, meta, or completely from hdfs. + * @param unassign if true unassign region if assigned + * @param metaRow if true remove region's row from META + * @param hdfs if true remove region's dir in HDFS + */ + private void deleteRegion(Configuration conf, final HTableDescriptor htd, + byte[] startKey, byte[] endKey, boolean unassign, boolean metaRow, + boolean hdfs) throws IOException, InterruptedException { + deleteRegion(conf, htd, startKey, endKey, unassign, metaRow, hdfs, false); + } - LOG.info("Before delete:"); - dumpMeta(htd); + /** + * Delete a region from assignments, meta, or completely from hdfs. + * @param unassign if true unassign region if assigned + * @param metaRow if true remove region's row from META + * @param hdfs if true remove region's dir in HDFS + * @param regionInfoOnly if true remove a region dir's .regioninfo file + */ + private void deleteRegion(Configuration conf, final HTableDescriptor htd, + byte[] startKey, byte[] endKey, boolean unassign, boolean metaRow, + boolean hdfs, boolean regionInfoOnly) throws IOException, InterruptedException { + LOG.info("** Before delete:"); + dumpMeta(htd.getName()); - Map hris = tbl.getRegionsInfo(); - for (Entry e: hris.entrySet()) { + Map hris = tbl.getRegionLocations(); + for (Entry e: hris.entrySet()) { HRegionInfo hri = e.getKey(); - HServerAddress hsa = e.getValue(); - if (Bytes.compareTo(hri.getStartKey(), startKey) == 0 + ServerName hsa = e.getValue(); + if (Bytes.compareTo(hri.getStartKey(), startKey) == 0 && Bytes.compareTo(hri.getEndKey(), endKey) == 0) { LOG.info("RegionName: " +hri.getRegionNameAsString()); byte[] deleteRow = hri.getRegionName(); - TEST_UTIL.getHBaseAdmin().unassign(deleteRow, true); - LOG.info("deleting hdfs data: " + hri.toString() + hsa.toString()); - Path rootDir = new Path(conf.get(HConstants.HBASE_DIR)); - FileSystem fs = rootDir.getFileSystem(conf); - Path p = new Path(rootDir + "/" + htd.getNameAsString(), hri.getEncodedName()); - fs.delete(p, true); + if (unassign) { + LOG.info("Undeploying region " + hri + " from server " + hsa); + undeployRegion(new HBaseAdmin(conf), hsa, new HRegionInfo(hri)); + } + + if (regionInfoOnly) { + LOG.info("deleting hdfs .regioninfo data: " + hri.toString() + hsa.toString()); + Path rootDir = new Path(conf.get(HConstants.HBASE_DIR)); + FileSystem fs = rootDir.getFileSystem(conf); + Path p = new Path(rootDir + "/" + htd.getNameAsString(), hri.getEncodedName()); + Path hriPath = new Path(p, HRegion.REGIONINFO_FILE); + fs.delete(hriPath, true); + } + + if (hdfs) { + LOG.info("deleting hdfs data: " + hri.toString() + hsa.toString()); + Path rootDir = new Path(conf.get(HConstants.HBASE_DIR)); + FileSystem fs = rootDir.getFileSystem(conf); + Path p = new Path(rootDir + "/" + htd.getNameAsString(), hri.getEncodedName()); + HBaseFsck.debugLsr(conf, p); + boolean success = fs.delete(p, true); + LOG.info("Deleted " + p + " sucessfully? " + success); + HBaseFsck.debugLsr(conf, p); + } - HTable meta = new HTable(conf, HConstants.META_TABLE_NAME); - Delete delete = new Delete(deleteRow); - meta.delete(delete); + if (metaRow) { + HTable meta = new HTable(conf, HConstants.META_TABLE_NAME); + Delete delete = new Delete(deleteRow); + meta.delete(delete); + } } LOG.info(hri.toString() + hsa.toString()); } TEST_UTIL.getMetaTableRows(htd.getName()); - LOG.info("After delete:"); - dumpMeta(htd); - + LOG.info("*** After delete:"); + dumpMeta(htd.getName()); } /** @@ -201,11 +311,44 @@ HTable setupTable(String tablename) throws Exception { HTableDescriptor desc = new HTableDescriptor(tablename); HColumnDescriptor hcd = new HColumnDescriptor(Bytes.toString(FAM)); desc.addFamily(hcd); // If a table has no CF's it doesn't get checked - TEST_UTIL.getHBaseAdmin().createTable(desc, splits); + TEST_UTIL.getHBaseAdmin().createTable(desc, SPLITS); tbl = new HTable(TEST_UTIL.getConfiguration(), tablename); + + List puts = new ArrayList(); + for (byte[] row : ROWKEYS) { + Put p = new Put(row); + p.add(FAM, Bytes.toBytes("val"), row); + puts.add(p); + } + tbl.put(puts); + tbl.flushCommits(); + long endTime = System.currentTimeMillis() + 60000; + while (!TEST_UTIL.getHBaseAdmin().isTableEnabled(tablename)) { + try { + if (System.currentTimeMillis() > endTime) { + fail("Failed to enable table " + tablename + " after waiting for 60 sec"); + } + Thread.sleep(100); + } catch (InterruptedException e) { + e.printStackTrace(); + fail("Interrupted when waiting table " + tablename + " to be enabled"); + } + } return tbl; } + /** + * Counts the number of row to verify data loss or non-dataloss. + */ + int countRows() throws IOException { + Scan s = new Scan(); + ResultScanner rs = tbl.getScanner(s); + int i = 0; + while(rs.next() !=null) { + i++; + } + return i; + } /** * delete table in preparation for next test @@ -214,14 +357,23 @@ HTable setupTable(String tablename) throws Exception { * @throws IOException */ void deleteTable(String tablename) throws IOException { - HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + HBaseAdmin admin = new HBaseAdmin(conf); + admin.getConnection().clearRegionCache(); byte[] tbytes = Bytes.toBytes(tablename); - admin.disableTable(tbytes); + if (admin.isTableEnabled(tbytes)) { + admin.disableTableAsync(tbytes); + } + while (!admin.isTableDisabled(tbytes)) { + try { + Thread.sleep(250); + } catch (InterruptedException e) { + e.printStackTrace(); + fail("Interrupted when trying to disable table " + tablename); + } + } admin.deleteTable(tbytes); } - - /** * This creates a clean table and confirms that the table is clean. */ @@ -234,18 +386,90 @@ public void testHBaseFsckClean() throws Exception { assertNoErrors(hbck); setupTable(table); - + assertEquals(ROWKEYS.length, countRows()); + // We created 1 table, should be fine hbck = doFsck(conf, false); assertNoErrors(hbck); assertEquals(0, hbck.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); + } finally { + deleteTable(table); + } + } + + /** + * Test thread pooling in the case where there are more regions than threads + */ + @Test + public void testHbckThreadpooling() throws Exception { + String table = "tableDupeStartKey"; + try { + // Create table with 4 regions + setupTable(table); + + // limit number of threads to 1. + Configuration newconf = new Configuration(conf); + newconf.setInt("hbasefsck.numthreads", 1); + assertNoErrors(doFsck(newconf, false)); + + // We should pass without triggering a RejectedExecutionException + } finally { + deleteTable(table); + } + } + + @Test + public void testHbckFixOrphanTable() throws Exception { + String table = "tableInfo"; + FileSystem fs = null; + Path tableinfo = null; + try { + setupTable(table); + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + + Path hbaseTableDir = new Path(conf.get(HConstants.HBASE_DIR) + "/" + table ); + fs = hbaseTableDir.getFileSystem(conf); + FileStatus status = FSTableDescriptors.getTableInfoPath(fs, hbaseTableDir); + tableinfo = status.getPath(); + fs.rename(tableinfo, new Path("/.tableinfo")); + + //to report error if .tableinfo is missing. + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.NO_TABLEINFO_FILE }); + + // fix OrphanTable with default .tableinfo (htd not yet cached on master) + hbck = doFsck(conf, true); + assertNoErrors(hbck); + status = null; + status = FSTableDescriptors.getTableInfoPath(fs, hbaseTableDir); + assertNotNull(status); + + HTableDescriptor htd = admin.getTableDescriptor(table.getBytes()); + htd.setValue("NOT_DEFAULT", "true"); + admin.disableTable(table); + admin.modifyTable(table.getBytes(), htd); + admin.enableTable(table); + fs.delete(status.getPath(), true); + + // fix OrphanTable with cache + htd = admin.getTableDescriptor(table.getBytes()); // warms up cached htd on master + hbck = doFsck(conf, true); + assertNoErrors(hbck); + status = null; + status = FSTableDescriptors.getTableInfoPath(fs, hbaseTableDir); + assertNotNull(status); + htd = admin.getTableDescriptor(table.getBytes()); + assertEquals(htd.getValue("NOT_DEFAULT"), "true"); } finally { + fs.rename(new Path("/.tableinfo"), tableinfo); deleteTable(table); } } /** - * This creates a bad table with regions that have a duplicate start key + * This create and fixes a bad table with regions that have a duplicate + * start key */ @Test public void testDupeStartKey() throws Exception { @@ -253,6 +477,7 @@ public void testDupeStartKey() throws Exception { try { setupTable(table); assertNoErrors(doFsck(conf, false)); + assertEquals(ROWKEYS.length, countRows()); // Now let's mess it up, by adding a region with a duplicate startkey HRegionInfo hriDupe = createRegion(conf, tbl.getTableDescriptor(), @@ -265,13 +490,156 @@ public void testDupeStartKey() throws Exception { assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.DUPE_STARTKEYS, ERROR_CODE.DUPE_STARTKEYS}); assertEquals(2, hbck.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); // seems like the "bigger" region won. + + // fix the degenerate region. + doFsck(conf,true); + + // check that the degenerate region is gone and no data loss + HBaseFsck hbck2 = doFsck(conf,false); + assertNoErrors(hbck2); + assertEquals(0, hbck2.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); } finally { deleteTable(table); } } - + + /** + * Get region info from local cluster. + */ + Map> getDeployedHRIs(HBaseAdmin admin) + throws IOException { + ClusterStatus status = admin.getMaster().getClusterStatus(); + Collection regionServers = status.getServers(); + Map> mm = + new HashMap>(); + HConnection connection = admin.getConnection(); + for (ServerName hsi : regionServers) { + HRegionInterface server = + connection.getHRegionConnection(hsi.getHostname(), hsi.getPort()); + + // list all online regions from this region server + List regions = server.getOnlineRegions(); + List regionNames = new ArrayList(); + for (HRegionInfo hri : regions) { + regionNames.add(hri.getRegionNameAsString()); + } + mm.put(hsi, regionNames); + } + return mm; + } + + /** + * Returns the HSI a region info is on. + */ + ServerName findDeployedHSI(Map> mm, HRegionInfo hri) { + for (Map.Entry> e : mm.entrySet()) { + if (e.getValue().contains(hri.getRegionNameAsString())) { + return e.getKey(); + } + } + return null; + } + + /** + * This test makes sure that parallel instances of Hbck is disabled. + * + * @throws Exception + */ + @Test + public void testParallelHbck() throws Exception { + final ExecutorService service; + final Future hbck1,hbck2; + + class RunHbck implements Callable{ + boolean fail = true; + public HBaseFsck call(){ + try{ + return doFsck(conf, false); + } catch(Exception e){ + if (e.getMessage().contains("Duplicate hbck")) { + fail = false; + } else { + LOG.fatal("hbck failed.", e); + } + } + // If we reach here, then an exception was caught + if (fail) fail(); + return null; + } + } + service = Executors.newFixedThreadPool(2); + hbck1 = service.submit(new RunHbck()); + hbck2 = service.submit(new RunHbck()); + service.shutdown(); + //wait for 15 seconds, for both hbck calls finish + service.awaitTermination(15, TimeUnit.SECONDS); + HBaseFsck h1 = hbck1.get(); + HBaseFsck h2 = hbck2.get(); + // Make sure only one of the calls was successful + assert(h1 == null || h2 == null); + if (h1 != null) { + assert(h1.getRetCode() >= 0); + } + if (h2 != null) { + assert(h2.getRetCode() >= 0); + } + } + + /** + * This create and fixes a bad table with regions that have a duplicate + * start key + */ + @Test + public void testDupeRegion() throws Exception { + String table = "tableDupeRegion"; + try { + setupTable(table); + assertNoErrors(doFsck(conf, false)); + assertEquals(ROWKEYS.length, countRows()); + + // Now let's mess it up, by adding a region with a duplicate startkey + HRegionInfo hriDupe = createRegion(conf, tbl.getTableDescriptor(), + Bytes.toBytes("A"), Bytes.toBytes("B")); + + TEST_UTIL.getHBaseCluster().getMaster().assignRegion(hriDupe); + TEST_UTIL.getHBaseCluster().getMaster().getAssignmentManager() + .waitForAssignment(hriDupe); + + // Yikes! The assignment manager can't tell between diff between two + // different regions with the same start/endkeys since it doesn't + // differentiate on ts/regionId! We actually need to recheck + // deployments! + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + while (findDeployedHSI(getDeployedHRIs(admin), hriDupe) == null) { + Thread.sleep(250); + } + + LOG.debug("Finished assignment of dupe region"); + + // TODO why is dupe region different from dupe start keys? + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.DUPE_STARTKEYS, + ERROR_CODE.DUPE_STARTKEYS}); + assertEquals(2, hbck.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); // seems like the "bigger" region won. + + // fix the degenerate region. + doFsck(conf,true); + + // check that the degenerate region is gone and no data loss + HBaseFsck hbck2 = doFsck(conf,false); + assertNoErrors(hbck2); + assertEquals(0, hbck2.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); + } finally { + deleteTable(table); + } + } + /** - * This creates a bad table with regions that has startkey == endkey + * This creates and fixes a bad table with regions that has startkey == endkey */ @Test public void testDegenerateRegions() throws Exception { @@ -279,6 +647,7 @@ public void testDegenerateRegions() throws Exception { try { setupTable(table); assertNoErrors(doFsck(conf,false)); + assertEquals(ROWKEYS.length, countRows()); // Now let's mess it up, by adding a region with a duplicate startkey HRegionInfo hriDupe = createRegion(conf, tbl.getTableDescriptor(), @@ -291,19 +660,202 @@ public void testDegenerateRegions() throws Exception { assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.DEGENERATE_REGION, ERROR_CODE.DUPE_STARTKEYS, ERROR_CODE.DUPE_STARTKEYS}); assertEquals(2, hbck.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); + + // fix the degenerate region. + doFsck(conf,true); + + // check that the degenerate region is gone and no data loss + HBaseFsck hbck2 = doFsck(conf,false); + assertNoErrors(hbck2); + assertEquals(0, hbck2.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); } finally { deleteTable(table); } } /** - * This creates a bad table where a start key contained in another region. + * This creates and fixes a bad table where a region is completely contained + * by another region. + */ + @Test + public void testContainedRegionOverlap() throws Exception { + String table = "tableContainedRegionOverlap"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by creating an overlap in the metadata + HRegionInfo hriOverlap = createRegion(conf, tbl.getTableDescriptor(), + Bytes.toBytes("A2"), Bytes.toBytes("B")); + TEST_UTIL.getHBaseCluster().getMaster().assignRegion(hriOverlap); + TEST_UTIL.getHBaseCluster().getMaster().getAssignmentManager() + .waitForAssignment(hriOverlap); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.OVERLAP_IN_REGION_CHAIN }); + assertEquals(2, hbck.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); + + // fix the problem. + doFsck(conf, true); + + // verify that overlaps are fixed + HBaseFsck hbck2 = doFsck(conf,false); + assertNoErrors(hbck2); + assertEquals(0, hbck2.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); + } finally { + deleteTable(table); + } + } + + /** + * This creates and fixes a bad table where an overlap group of + * 3 regions. Set HBaseFsck.maxMerge to 2 to trigger sideline overlapped + * region. Mess around the meta data so that closeRegion/offlineRegion + * throws exceptions. + */ + @Test + public void testSidelineOverlapRegion() throws Exception { + String table = "testSidelineOverlapRegion"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by creating an overlap + MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + HMaster master = cluster.getMaster(); + HRegionInfo hriOverlap1 = createRegion(conf, tbl.getTableDescriptor(), + Bytes.toBytes("A"), Bytes.toBytes("AB")); + master.assignRegion(hriOverlap1); + master.getAssignmentManager().waitForAssignment(hriOverlap1); + HRegionInfo hriOverlap2 = createRegion(conf, tbl.getTableDescriptor(), + Bytes.toBytes("AB"), Bytes.toBytes("B")); + master.assignRegion(hriOverlap2); + master.getAssignmentManager().waitForAssignment(hriOverlap2); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] {ERROR_CODE.DUPE_STARTKEYS, + ERROR_CODE.DUPE_STARTKEYS, ERROR_CODE.OVERLAP_IN_REGION_CHAIN}); + assertEquals(3, hbck.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); + + // mess around the overlapped regions, to trigger NotServingRegionException + Multimap overlapGroups = hbck.getOverlapGroups(table); + ServerName serverName = null; + byte[] regionName = null; + for (HbckInfo hbi: overlapGroups.values()) { + if ("A".equals(Bytes.toString(hbi.getStartKey())) + && "B".equals(Bytes.toString(hbi.getEndKey()))) { + regionName = hbi.getRegionName(); + + // get an RS not serving the region to force bad assignment info in to META. + int k = cluster.getServerWith(regionName); + for (int i = 0; i < 3; i++) { + if (i != k) { + HRegionServer rs = cluster.getRegionServer(i); + serverName = rs.getServerName(); + break; + } + } + + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + HBaseFsckRepair.closeRegionSilentlyAndWait(admin, + cluster.getRegionServer(k).getServerName(), hbi.getHdfsHRI()); + admin.unassign(regionName, true); + break; + } + } + + assertNotNull(regionName); + assertNotNull(serverName); + HTable meta = new HTable(conf, HConstants.META_TABLE_NAME); + Put put = new Put(regionName); + put.add(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER, + Bytes.toBytes(serverName.getHostAndPort())); + meta.put(put); + + // fix the problem. + HBaseFsck fsck = new HBaseFsck(conf); + fsck.connect(); + fsck.setDisplayFullReport(); // i.e. -details + fsck.setTimeLag(0); + fsck.setFixAssignments(true); + fsck.setFixMeta(true); + fsck.setFixHdfsHoles(true); + fsck.setFixHdfsOverlaps(true); + fsck.setFixHdfsOrphans(true); + fsck.setFixVersionFile(true); + fsck.setSidelineBigOverlaps(true); + fsck.setMaxMerge(2); + fsck.onlineHbck(); + + // verify that overlaps are fixed, and there are less rows + // since one region is sidelined. + HBaseFsck hbck2 = doFsck(conf,false); + assertNoErrors(hbck2); + assertEquals(0, hbck2.getOverlapGroups(table).size()); + assertTrue(ROWKEYS.length > countRows()); + } finally { + deleteTable(table); + } + } + + /** + * This creates and fixes a bad table where a region is completely contained + * by another region, and there is a hole (sort of like a bad split) + */ + @Test + public void testOverlapAndOrphan() throws Exception { + String table = "tableOverlapAndOrphan"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by creating an overlap in the metadata + TEST_UTIL.getHBaseAdmin().disableTable(table); + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("A"), + Bytes.toBytes("B"), true, true, false, true); + TEST_UTIL.getHBaseAdmin().enableTable(table); + + HRegionInfo hriOverlap = createRegion(conf, tbl.getTableDescriptor(), + Bytes.toBytes("A2"), Bytes.toBytes("B")); + TEST_UTIL.getHBaseCluster().getMaster().assignRegion(hriOverlap); + TEST_UTIL.getHBaseCluster().getMaster().getAssignmentManager() + .waitForAssignment(hriOverlap); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.ORPHAN_HDFS_REGION, ERROR_CODE.NOT_IN_META_OR_DEPLOYED, + ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // fix the problem. + doFsck(conf, true); + + // verify that overlaps are fixed + HBaseFsck hbck2 = doFsck(conf,false); + assertNoErrors(hbck2); + assertEquals(0, hbck2.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); + } finally { + deleteTable(table); + } + } + + /** + * This creates and fixes a bad table where a region overlaps two regions -- + * a start key contained in another region and its end key is contained in + * yet another region. */ @Test public void testCoveredStartKey() throws Exception { String table = "tableCoveredStartKey"; try { setupTable(table); + assertEquals(ROWKEYS.length, countRows()); // Mess it up by creating an overlap in the metadata HRegionInfo hriOverlap = createRegion(conf, tbl.getTableDescriptor(), @@ -317,42 +869,1080 @@ public void testCoveredStartKey() throws Exception { ERROR_CODE.OVERLAP_IN_REGION_CHAIN, ERROR_CODE.OVERLAP_IN_REGION_CHAIN }); assertEquals(3, hbck.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); + + // fix the problem. + doFsck(conf, true); + + // verify that overlaps are fixed + HBaseFsck hbck2 = doFsck(conf, false); + assertErrors(hbck2, new ERROR_CODE[0]); + assertEquals(0, hbck2.getOverlapGroups(table).size()); + assertEquals(ROWKEYS.length, countRows()); + } finally { + deleteTable(table); + } + } + + /** + * This creates and fixes a bad table with a missing region -- hole in meta + * and data missing in the fs. + */ + @Test + public void testRegionHole() throws Exception { + String table = "tableRegionHole"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by leaving a hole in the assignment, meta, and hdfs data + TEST_UTIL.getHBaseAdmin().disableTable(table); + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("B"), + Bytes.toBytes("C"), true, true, true); + TEST_UTIL.getHBaseAdmin().enableTable(table); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.HOLE_IN_REGION_CHAIN}); + // holes are separate from overlap groups + assertEquals(0, hbck.getOverlapGroups(table).size()); + + // fix hole + doFsck(conf, true); + + // check that hole fixed + assertNoErrors(doFsck(conf,false)); + assertEquals(ROWKEYS.length - 2 , countRows()); // lost a region so lost a row } finally { deleteTable(table); } } /** - * This creates a bad table with a hole in meta. + * This creates and fixes a bad table with a missing region -- hole in meta + * and data present but .regioinfino missing (an orphan hdfs region)in the fs. */ @Test - public void testMetaHole() throws Exception { - String table = "tableMetaHole"; + public void testHDFSRegioninfoMissing() throws Exception { + String table = "tableHDFSRegioininfoMissing"; try { setupTable(table); + assertEquals(ROWKEYS.length, countRows()); // Mess it up by leaving a hole in the meta data - HRegionInfo hriHole = createRegion(conf, tbl.getTableDescriptor(), - Bytes.toBytes("D"), Bytes.toBytes("")); - TEST_UTIL.getHBaseCluster().getMaster().assignRegion(hriHole); - TEST_UTIL.getHBaseCluster().getMaster().getAssignmentManager() - .waitForAssignment(hriHole); + TEST_UTIL.getHBaseAdmin().disableTable(table); + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("B"), + Bytes.toBytes("C"), true, true, false, true); + TEST_UTIL.getHBaseAdmin().enableTable(table); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.ORPHAN_HDFS_REGION, + ERROR_CODE.NOT_IN_META_OR_DEPLOYED, + ERROR_CODE.HOLE_IN_REGION_CHAIN}); + // holes are separate from overlap groups + assertEquals(0, hbck.getOverlapGroups(table).size()); + + // fix hole + doFsck(conf, true); + + // check that hole fixed + assertNoErrors(doFsck(conf, false)); + assertEquals(ROWKEYS.length, countRows()); + } finally { + deleteTable(table); + } + } + + /** + * This creates and fixes a bad table with a region that is missing meta and + * not assigned to a region server. + */ + @Test + public void testNotInMetaOrDeployedHole() throws Exception { + String table = "tableNotInMetaOrDeployedHole"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + // Mess it up by leaving a hole in the meta data TEST_UTIL.getHBaseAdmin().disableTable(table); - deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("C"), Bytes.toBytes("")); + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("B"), + Bytes.toBytes("C"), true, true, false); // don't rm from fs TEST_UTIL.getHBaseAdmin().enableTable(table); HBaseFsck hbck = doFsck(conf, false); - assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.HOLE_IN_REGION_CHAIN }); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.NOT_IN_META_OR_DEPLOYED, ERROR_CODE.HOLE_IN_REGION_CHAIN}); // holes are separate from overlap groups assertEquals(0, hbck.getOverlapGroups(table).size()); + + // fix hole + assertErrors(doFsck(conf, true) , new ERROR_CODE[] { + ERROR_CODE.NOT_IN_META_OR_DEPLOYED, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // check that hole fixed + assertNoErrors(doFsck(conf,false)); + assertEquals(ROWKEYS.length, countRows()); } finally { deleteTable(table); } } - @org.junit.Rule - public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = - new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); -} + /** + * This creates fixes a bad table with a hole in meta. + */ + @Test + public void testNotInMetaHole() throws Exception { + String table = "tableNotInMetaHole"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by leaving a hole in the meta data + TEST_UTIL.getHBaseAdmin().disableTable(table); + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("B"), + Bytes.toBytes("C"), false, true, false); // don't rm from fs + TEST_UTIL.getHBaseAdmin().enableTable(table); + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.NOT_IN_META_OR_DEPLOYED, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + // holes are separate from overlap groups + assertEquals(0, hbck.getOverlapGroups(table).size()); + + // fix hole + assertErrors(doFsck(conf, true) , new ERROR_CODE[] { + ERROR_CODE.NOT_IN_META_OR_DEPLOYED, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // check that hole fixed + assertNoErrors(doFsck(conf,false)); + assertEquals(ROWKEYS.length, countRows()); + } finally { + deleteTable(table); + } + } + + /** + * This creates and fixes a bad table with a region that is in meta but has + * no deployment or data hdfs + */ + @Test + public void testNotInHdfs() throws Exception { + String table = "tableNotInHdfs"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // make sure data in regions, if in hlog only there is no data loss + TEST_UTIL.getHBaseAdmin().flush(table); + + // Mess it up by leaving a hole in the hdfs data + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("B"), + Bytes.toBytes("C"), false, false, true); // don't rm meta + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] {ERROR_CODE.NOT_IN_HDFS}); + // holes are separate from overlap groups + assertEquals(0, hbck.getOverlapGroups(table).size()); + + // fix hole + doFsck(conf, true); + + // check that hole fixed + assertNoErrors(doFsck(conf,false)); + assertEquals(ROWKEYS.length - 2, countRows()); + } finally { + deleteTable(table); + } + } + + /** + * This creates entries in META with no hdfs data. This should cleanly + * remove the table. + */ + @Test + public void testNoHdfsTable() throws Exception { + String table = "NoHdfsTable"; + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // make sure data in regions, if in hlog only there is no data loss + TEST_UTIL.getHBaseAdmin().flush(table); + + // Mess it up by leaving a giant hole in meta + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes(""), + Bytes.toBytes("A"), false, false, true); // don't rm meta + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("A"), + Bytes.toBytes("B"), false, false, true); // don't rm meta + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("B"), + Bytes.toBytes("C"), false, false, true); // don't rm meta + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("C"), + Bytes.toBytes(""), false, false, true); // don't rm meta + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] {ERROR_CODE.NOT_IN_HDFS, + ERROR_CODE.NOT_IN_HDFS, ERROR_CODE.NOT_IN_HDFS, + ERROR_CODE.NOT_IN_HDFS,}); + // holes are separate from overlap groups + assertEquals(0, hbck.getOverlapGroups(table).size()); + + // fix hole + doFsck(conf, true); // in 0.92+, meta entries auto create regiondirs + + // check that hole fixed + assertNoErrors(doFsck(conf,false)); + assertFalse("Table "+ table + " should have been deleted", + TEST_UTIL.getHBaseAdmin().tableExists(table)); + } + + /** + * when the hbase.version file missing, It is fix the fault. + */ + @Test + public void testNoVersionFile() throws Exception { + // delete the hbase.version file + Path rootDir = new Path(conf.get(HConstants.HBASE_DIR)); + FileSystem fs = rootDir.getFileSystem(conf); + Path versionFile = new Path(rootDir, HConstants.VERSION_FILE_NAME); + fs.delete(versionFile, true); + + // test + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.NO_VERSION_FILE }); + // fix hbase.version missing + doFsck(conf, true); + + // no version file fixed + assertNoErrors(doFsck(conf, false)); + } + + /** + * the region is not deployed when the table is disabled. + */ + @Test + public void testRegionShouldNotBeDeployed() throws Exception { + String table = "tableRegionShouldNotBeDeployed"; + try { + LOG.info("Starting testRegionShouldNotBeDeployed."); + MiniHBaseCluster cluster = TEST_UTIL.getHBaseCluster(); + assertTrue(cluster.waitForActiveAndReadyMaster()); + + // Create a ZKW to use in the test + ZooKeeperWatcher zkw = HBaseTestingUtility.getZooKeeperWatcher(TEST_UTIL); + + FileSystem filesystem = FileSystem.get(conf); + Path rootdir = filesystem.makeQualified(new Path(conf + .get(HConstants.HBASE_DIR))); + + byte[][] SPLIT_KEYS = new byte[][] { new byte[0], Bytes.toBytes("aaa"), + Bytes.toBytes("bbb"), Bytes.toBytes("ccc"), Bytes.toBytes("ddd") }; + HTableDescriptor htdDisabled = new HTableDescriptor(Bytes.toBytes(table)); + htdDisabled.addFamily(new HColumnDescriptor(FAM)); + + // Write the .tableinfo + FSTableDescriptors + .createTableDescriptor(filesystem, rootdir, htdDisabled); + List disabledRegions = TEST_UTIL.createMultiRegionsInMeta( + TEST_UTIL.getConfiguration(), htdDisabled, SPLIT_KEYS); + + // Let's just assign everything to first RS + HRegionServer hrs = cluster.getRegionServer(0); + ServerName serverName = hrs.getServerName(); + + // create region files. + TEST_UTIL.getHBaseAdmin().disableTable(table); + TEST_UTIL.getHBaseAdmin().enableTable(table); + + // Region of disable table was opened on RS + TEST_UTIL.getHBaseAdmin().disableTable(table); + HRegionInfo region = disabledRegions.remove(0); + ZKAssign.createNodeOffline(zkw, region, serverName); + hrs.openRegion(region); + + int iTimes = 0; + while (true) { + RegionTransitionData rtd = ZKAssign.getData(zkw, + region.getEncodedName()); + if (rtd != null && rtd.getEventType() == EventType.RS_ZK_REGION_OPENED) { + break; + } + Thread.sleep(100); + iTimes++; + if (iTimes >= REGION_ONLINE_TIMEOUT) { + break; + } + } + assertTrue(iTimes < REGION_ONLINE_TIMEOUT); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.SHOULD_NOT_BE_DEPLOYED }); + + // fix this fault + doFsck(conf, true); + + // check result + assertNoErrors(doFsck(conf, false)); + } finally { + TEST_UTIL.getHBaseAdmin().enableTable(table); + deleteTable(table); + } + } + + /** + * This creates two tables and mess both of them and fix them one by one + */ + @Test + public void testFixByTable() throws Exception { + String table1 = "testFixByTable1"; + String table2 = "testFixByTable2"; + try { + setupTable(table1); + // make sure data in regions, if in hlog only there is no data loss + TEST_UTIL.getHBaseAdmin().flush(table1); + // Mess them up by leaving a hole in the hdfs data + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("B"), + Bytes.toBytes("C"), false, false, true); // don't rm meta + + setupTable(table2); + // make sure data in regions, if in hlog only there is no data loss + TEST_UTIL.getHBaseAdmin().flush(table2); + // Mess them up by leaving a hole in the hdfs data + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("B"), + Bytes.toBytes("C"), false, false, true); // don't rm meta + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.NOT_IN_HDFS, ERROR_CODE.NOT_IN_HDFS}); + + // fix hole in table 1 + doFsck(conf, true, table1); + // check that hole in table 1 fixed + assertNoErrors(doFsck(conf, false, table1)); + // check that hole in table 2 still there + assertErrors(doFsck(conf, false, table2), + new ERROR_CODE[] {ERROR_CODE.NOT_IN_HDFS}); + + // fix hole in table 2 + doFsck(conf, true, table2); + // check that hole in both tables fixed + assertNoErrors(doFsck(conf, false)); + assertEquals(ROWKEYS.length - 2, countRows()); + } finally { + deleteTable(table1); + deleteTable(table2); + } + } + /** + * A split parent in meta, in hdfs, and not deployed + */ + @Test + public void testLingeringSplitParent() throws Exception { + String table = "testLingeringSplitParent"; + HTable meta = null; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // make sure data in regions, if in hlog only there is no data loss + TEST_UTIL.getHBaseAdmin().flush(table); + HRegionLocation location = tbl.getRegionLocation("B"); + + // Delete one region from meta, but not hdfs, unassign it. + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("B"), + Bytes.toBytes("C"), true, true, false); + + // Create a new meta entry to fake it as a split parent. + meta = new HTable(conf, HTableDescriptor.META_TABLEDESC.getName()); + HRegionInfo hri = location.getRegionInfo(); + + HRegionInfo a = new HRegionInfo(tbl.getTableName(), + Bytes.toBytes("B"), Bytes.toBytes("BM")); + HRegionInfo b = new HRegionInfo(tbl.getTableName(), + Bytes.toBytes("BM"), Bytes.toBytes("C")); + Put p = new Put(hri.getRegionName()); + hri.setOffline(true); + hri.setSplit(true); + p.add(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER, + Writables.getBytes(hri)); + p.add(HConstants.CATALOG_FAMILY, HConstants.SPLITA_QUALIFIER, + Writables.getBytes(a)); + p.add(HConstants.CATALOG_FAMILY, HConstants.SPLITB_QUALIFIER, + Writables.getBytes(b)); + meta.put(p); + meta.flushCommits(); + TEST_UTIL.getHBaseAdmin().flush(HConstants.META_TABLE_NAME); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.LINGERING_SPLIT_PARENT, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // regular repair cannot fix lingering split parent + hbck = doFsck(conf, true); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.LINGERING_SPLIT_PARENT, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + assertFalse(hbck.shouldRerun()); + hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.LINGERING_SPLIT_PARENT, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // fix lingering split parent + hbck = new HBaseFsck(conf); + hbck.connect(); + hbck.setDisplayFullReport(); // i.e. -details + hbck.setTimeLag(0); + hbck.setFixSplitParents(true); + hbck.onlineHbck(); + assertTrue(hbck.shouldRerun()); + + Get get = new Get(hri.getRegionName()); + Result result = meta.get(get); + assertTrue(result.getColumn(HConstants.CATALOG_FAMILY, + HConstants.SPLITA_QUALIFIER).isEmpty()); + assertTrue(result.getColumn(HConstants.CATALOG_FAMILY, + HConstants.SPLITB_QUALIFIER).isEmpty()); + TEST_UTIL.getHBaseAdmin().flush(HConstants.META_TABLE_NAME); + + // fix other issues + doFsck(conf, true); + + // check that all are fixed + assertNoErrors(doFsck(conf, false)); + assertEquals(ROWKEYS.length, countRows()); + } finally { + deleteTable(table); + IOUtils.closeQuietly(meta); + } + } + + /** + * Tests that LINGERING_SPLIT_PARENT is not erroneously reported for + * valid cases where the daughters are there. + */ + @Test + public void testValidLingeringSplitParent() throws Exception { + String table = "testLingeringSplitParent"; + HTable meta = null; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // make sure data in regions, if in hlog only there is no data loss + TEST_UTIL.getHBaseAdmin().flush(table); + HRegionLocation location = tbl.getRegionLocation("B"); + + meta = new HTable(conf, HTableDescriptor.META_TABLEDESC.getName()); + HRegionInfo hri = location.getRegionInfo(); + + // do a regular split + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + byte[] regionName = location.getRegionInfo().getRegionName(); + admin.split(location.getRegionInfo().getRegionName(), Bytes.toBytes("BM")); + TestEndToEndSplitTransaction.blockUntilRegionSplit( + TEST_UTIL.getConfiguration(), 60000, regionName, true); + + // TODO: fixHdfsHoles does not work against splits, since the parent dir lingers on + // for some time until children references are deleted. HBCK erroneously sees this as + // overlapping regions + HBaseFsck hbck = doFsck(conf, true, true, false, false, false, true, true, true, null); + assertErrors(hbck, new ERROR_CODE[] {}); //no LINGERING_SPLIT_PARENT reported + + // assert that the split META entry is still there. + Get get = new Get(hri.getRegionName()); + Result result = meta.get(get); + assertNotNull(result); + assertNotNull(MetaReader.parseCatalogResult(result).getFirst()); + + assertEquals(ROWKEYS.length, countRows()); + + // assert that we still have the split regions + assertEquals(tbl.getStartKeys().length, SPLITS.length + 1 + 1); //SPLITS + 1 is # regions pre-split. + assertNoErrors(doFsck(conf, false)); + } finally { + deleteTable(table); + IOUtils.closeQuietly(meta); + } + } + + /** + * Split crashed after write to META finished for the parent region, but + * failed to write daughters (pre HBASE-7721 codebase) + */ + @Test(timeout=75000) + public void testSplitDaughtersNotInMeta() throws Exception { + String table = "testSplitdaughtersNotInMeta"; + HTable meta = null; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // make sure data in regions, if in hlog only there is no data loss + TEST_UTIL.getHBaseAdmin().flush(table); + HRegionLocation location = tbl.getRegionLocation("B"); + + meta = new HTable(conf, HTableDescriptor.META_TABLEDESC.getName()); + HRegionInfo hri = location.getRegionInfo(); + + // do a regular split + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + byte[] regionName = location.getRegionInfo().getRegionName(); + admin.split(location.getRegionInfo().getRegionName(), Bytes.toBytes("BM")); + TestEndToEndSplitTransaction.blockUntilRegionSplit( + TEST_UTIL.getConfiguration(), 60000, regionName, true); + + PairOfSameType daughters = MetaReader.getDaughterRegions(meta.get(new Get(regionName))); + + // Delete daughter regions from meta, but not hdfs, unassign it. + Map hris = tbl.getRegionLocations(); + undeployRegion(admin, hris.get(daughters.getFirst()), daughters.getFirst()); + undeployRegion(admin, hris.get(daughters.getSecond()), daughters.getSecond()); + + meta.delete(new Delete(daughters.getFirst().getRegionName())); + meta.delete(new Delete(daughters.getSecond().getRegionName())); + meta.flushCommits(); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] {ERROR_CODE.NOT_IN_META_OR_DEPLOYED, + ERROR_CODE.NOT_IN_META_OR_DEPLOYED, ERROR_CODE.HOLE_IN_REGION_CHAIN}); //no LINGERING_SPLIT_PARENT + + // now fix it. The fix should not revert the region split, but add daughters to META + hbck = doFsck(conf, true, true, false, false, false, false, false, false, null); + assertErrors(hbck, new ERROR_CODE[] {ERROR_CODE.NOT_IN_META_OR_DEPLOYED, + ERROR_CODE.NOT_IN_META_OR_DEPLOYED, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // assert that the split META entry is still there. + Get get = new Get(hri.getRegionName()); + Result result = meta.get(get); + assertNotNull(result); + assertNotNull(MetaReader.parseCatalogResult(result).getFirst()); + + assertEquals(ROWKEYS.length, countRows()); + + // assert that we still have the split regions + assertEquals(tbl.getStartKeys().length, SPLITS.length + 1 + 1); //SPLITS + 1 is # regions pre-split. + assertNoErrors(doFsck(conf, false)); //should be fixed by now + } finally { + deleteTable(table); + IOUtils.closeQuietly(meta); + } + } + + /** + * This creates and fixes a bad table with a missing region which is the 1st region -- hole in + * meta and data missing in the fs. + */ + @Test + public void testMissingFirstRegion() throws Exception { + String table = "testMissingFirstRegion"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by leaving a hole in the assignment, meta, and hdfs data + TEST_UTIL.getHBaseAdmin().disableTable(table); + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes(""), Bytes.toBytes("A"), true, + true, true); + TEST_UTIL.getHBaseAdmin().enableTable(table); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.FIRST_REGION_STARTKEY_NOT_EMPTY }); + // fix hole + doFsck(conf, true); + // check that hole fixed + assertNoErrors(doFsck(conf, false)); + } finally { + deleteTable(table); + } + } + + /** + * This creates and fixes a bad table with missing last region -- hole in meta and data missing in + * the fs. + */ + @Test + public void testMissingLastRegion() throws Exception { + String table = "testMissingLastRegion"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by leaving a hole in the assignment, meta, and hdfs data + TEST_UTIL.getHBaseAdmin().disableTable(table); + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("C"), Bytes.toBytes(""), true, + true, true); + TEST_UTIL.getHBaseAdmin().enableTable(table); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.LAST_REGION_ENDKEY_NOT_EMPTY }); + // fix hole + doFsck(conf, true); + // check that hole fixed + assertNoErrors(doFsck(conf, false)); + } finally { + deleteTable(table); + } + } + + /** + * Test -noHdfsChecking option can detect and fix assignments issue. + */ + @Test + public void testFixAssignmentsAndNoHdfsChecking() throws Exception { + String table = "testFixAssignmentsAndNoHdfsChecking"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by closing a region + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("A"), + Bytes.toBytes("B"), true, false, false, false); + + // verify there is no other errors + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.NOT_DEPLOYED, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // verify that noHdfsChecking report the same errors + HBaseFsck fsck = new HBaseFsck(conf); + fsck.connect(); + fsck.setDisplayFullReport(); // i.e. -details + fsck.setTimeLag(0); + fsck.setCheckHdfs(false); + fsck.onlineHbck(); + assertErrors(fsck, new ERROR_CODE[] { + ERROR_CODE.NOT_DEPLOYED, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // verify that fixAssignments works fine with noHdfsChecking + fsck = new HBaseFsck(conf); + fsck.connect(); + fsck.setDisplayFullReport(); // i.e. -details + fsck.setTimeLag(0); + fsck.setCheckHdfs(false); + fsck.setFixAssignments(true); + fsck.onlineHbck(); + assertTrue(fsck.shouldRerun()); + fsck.onlineHbck(); + assertNoErrors(fsck); + + assertEquals(ROWKEYS.length, countRows()); + } finally { + deleteTable(table); + } + } + + /** + * Test -noHdfsChecking option can detect region is not in meta but deployed. + * However, it can not fix it without checking Hdfs because we need to get + * the region info from Hdfs in this case, then to patch the meta. + */ + @Test + public void testFixMetaNotWorkingWithNoHdfsChecking() throws Exception { + String table = "testFixMetaNotWorkingWithNoHdfsChecking"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by deleting a region from the metadata + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("A"), + Bytes.toBytes("B"), false, true, false, false); + + // verify there is no other errors + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.NOT_IN_META, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // verify that noHdfsChecking report the same errors + HBaseFsck fsck = new HBaseFsck(conf); + fsck.connect(); + fsck.setDisplayFullReport(); // i.e. -details + fsck.setTimeLag(0); + fsck.setCheckHdfs(false); + fsck.onlineHbck(); + assertErrors(fsck, new ERROR_CODE[] { + ERROR_CODE.NOT_IN_META, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // verify that fixMeta doesn't work with noHdfsChecking + fsck = new HBaseFsck(conf); + fsck.connect(); + fsck.setDisplayFullReport(); // i.e. -details + fsck.setTimeLag(0); + fsck.setCheckHdfs(false); + fsck.setFixAssignments(true); + fsck.setFixMeta(true); + fsck.onlineHbck(); + assertFalse(fsck.shouldRerun()); + assertErrors(fsck, new ERROR_CODE[] { + ERROR_CODE.NOT_IN_META, ERROR_CODE.HOLE_IN_REGION_CHAIN}); + } finally { + deleteTable(table); + } + } + + /** + * Test -fixHdfsHoles doesn't work with -noHdfsChecking option, + * and -noHdfsChecking can't detect orphan Hdfs region. + */ + @Test + public void testFixHdfsHolesNotWorkingWithNoHdfsChecking() throws Exception { + String table = "testFixHdfsHolesNotWorkingWithNoHdfsChecking"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by creating an overlap in the metadata + TEST_UTIL.getHBaseAdmin().disableTable(table); + deleteRegion(conf, tbl.getTableDescriptor(), Bytes.toBytes("A"), + Bytes.toBytes("B"), true, true, false, true); + TEST_UTIL.getHBaseAdmin().enableTable(table); + + HRegionInfo hriOverlap = createRegion(conf, tbl.getTableDescriptor(), + Bytes.toBytes("A2"), Bytes.toBytes("B")); + TEST_UTIL.getHBaseCluster().getMaster().assignRegion(hriOverlap); + TEST_UTIL.getHBaseCluster().getMaster().getAssignmentManager() + .waitForAssignment(hriOverlap); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { + ERROR_CODE.ORPHAN_HDFS_REGION, ERROR_CODE.NOT_IN_META_OR_DEPLOYED, + ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // verify that noHdfsChecking can't detect ORPHAN_HDFS_REGION + HBaseFsck fsck = new HBaseFsck(conf); + fsck.connect(); + fsck.setDisplayFullReport(); // i.e. -details + fsck.setTimeLag(0); + fsck.setCheckHdfs(false); + fsck.onlineHbck(); + assertErrors(fsck, new ERROR_CODE[] { + ERROR_CODE.HOLE_IN_REGION_CHAIN}); + + // verify that fixHdfsHoles doesn't work with noHdfsChecking + fsck = new HBaseFsck(conf); + fsck.connect(); + fsck.setDisplayFullReport(); // i.e. -details + fsck.setTimeLag(0); + fsck.setCheckHdfs(false); + fsck.setFixHdfsHoles(true); + fsck.setFixHdfsOverlaps(true); + fsck.setFixHdfsOrphans(true); + fsck.onlineHbck(); + assertFalse(fsck.shouldRerun()); + assertErrors(fsck, new ERROR_CODE[] { + ERROR_CODE.HOLE_IN_REGION_CHAIN}); + } finally { + if (TEST_UTIL.getHBaseAdmin().isTableDisabled(table)) { + TEST_UTIL.getHBaseAdmin().enableTable(table); + } + deleteTable(table); + } + } + + /** + * We don't have an easy way to verify that a flush completed, so we loop until we find a + * legitimate hfile and return it. + * @param fs + * @param table + * @return Path of a flushed hfile. + * @throws IOException + */ + Path getFlushedHFile(FileSystem fs, String table) throws IOException { + Path tableDir= FSUtils.getTablePath(FSUtils.getRootDir(conf), table); + Path regionDir = FSUtils.getRegionDirs(fs, tableDir).get(0); + Path famDir = new Path(regionDir, FAM_STR); + + // keep doing this until we get a legit hfile + while (true) { + FileStatus[] hfFss = fs.listStatus(famDir); + if (hfFss.length == 0) { + continue; + } + for (FileStatus hfs : hfFss) { + if (!hfs.isDir()) { + return hfs.getPath(); + } + } + } + } + + /** + * This creates a table and then corrupts an hfile. Hbck should quarantine the file. + */ + @Test(timeout=180000) + public void testQuarantineCorruptHFile() throws Exception { + String table = name.getMethodName(); + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + TEST_UTIL.getHBaseAdmin().flush(table); // flush is async. + + FileSystem fs = FileSystem.get(conf); + Path hfile = getFlushedHFile(fs, table); + + // Mess it up by leaving a hole in the assignment, meta, and hdfs data + TEST_UTIL.getHBaseAdmin().disableTable(table); + + // create new corrupt file called deadbeef (valid hfile name) + Path corrupt = new Path(hfile.getParent(), "deadbeef"); + TestHFile.truncateFile(fs, hfile, corrupt); + LOG.info("Created corrupted file " + corrupt); + HBaseFsck.debugLsr(conf, FSUtils.getRootDir(conf)); + + // we cannot enable here because enable never finished due to the corrupt region. + HBaseFsck res = HbckTestingUtil.doHFileQuarantine(conf, table); + assertEquals(res.getRetCode(), 0); + HFileCorruptionChecker hfcc = res.getHFilecorruptionChecker(); + assertEquals(hfcc.getHFilesChecked(), 5); + assertEquals(hfcc.getCorrupted().size(), 1); + assertEquals(hfcc.getFailures().size(), 0); + assertEquals(hfcc.getQuarantined().size(), 1); + assertEquals(hfcc.getMissing().size(), 0); + + // Its been fixed, verify that we can enable. + TEST_UTIL.getHBaseAdmin().enableTable(table); + } finally { + deleteTable(table); + } + } + + /** + * Test that use this should have a timeout, because this method could potentially wait forever. + */ + private void doQuarantineTest(String table, HBaseFsck hbck, int check, int corrupt, int fail, + int quar, int missing) throws Exception { + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + TEST_UTIL.getHBaseAdmin().flush(table); // flush is async. + + // Mess it up by leaving a hole in the assignment, meta, and hdfs data + TEST_UTIL.getHBaseAdmin().disableTable(table); + + String[] args = {"-sidelineCorruptHFiles", "-repairHoles", "-ignorePreCheckPermission", table}; + ExecutorService exec = new ScheduledThreadPoolExecutor(10); + HBaseFsck res = hbck.exec(exec, args); + + HFileCorruptionChecker hfcc = res.getHFilecorruptionChecker(); + assertEquals(hfcc.getHFilesChecked(), check); + assertEquals(hfcc.getCorrupted().size(), corrupt); + assertEquals(hfcc.getFailures().size(), fail); + assertEquals(hfcc.getQuarantined().size(), quar); + assertEquals(hfcc.getMissing().size(), missing); + + // its been fixed, verify that we can enable + HBaseAdmin admin = TEST_UTIL.getHBaseAdmin(); + admin.enableTableAsync(table); + while (!admin.isTableEnabled(table)) { + try { + Thread.sleep(250); + } catch (InterruptedException e) { + e.printStackTrace(); + fail("Interrupted when trying to enable table " + table); + } + } + } finally { + deleteTable(table); + } + } + + /** + * This creates a table and simulates the race situation where a concurrent compaction or split + * has removed an hfile after the corruption checker learned about it. + */ + @Test(timeout=180000) + public void testQuarantineMissingHFile() throws Exception { + String table = name.getMethodName(); + ExecutorService exec = new ScheduledThreadPoolExecutor(10); + // inject a fault in the hfcc created. + final FileSystem fs = FileSystem.get(conf); + HBaseFsck hbck = new HBaseFsck(conf, exec) { + public HFileCorruptionChecker createHFileCorruptionChecker(boolean sidelineCorruptHFiles) throws IOException { + return new HFileCorruptionChecker(conf, executor, sidelineCorruptHFiles) { + boolean attemptedFirstHFile = false; + protected void checkHFile(Path p) throws IOException { + if (!attemptedFirstHFile) { + attemptedFirstHFile = true; + assertTrue(fs.delete(p, true)); // make sure delete happened. + } + super.checkHFile(p); + } + }; + } + }; + doQuarantineTest(table, hbck, 4, 0, 0, 0, 1); // 4 attempted, but 1 missing. + } + + /** + * This creates a table and simulates the race situation where a concurrent compaction or split + * has removed an colfam dir before the corruption checker got to it. + */ + @Test(timeout=180000) + public void testQuarantineMissingFamdir() throws Exception { + String table = name.getMethodName(); + ExecutorService exec = new ScheduledThreadPoolExecutor(10); + // inject a fault in the hfcc created. + final FileSystem fs = FileSystem.get(conf); + HBaseFsck hbck = new HBaseFsck(conf, exec) { + public HFileCorruptionChecker createHFileCorruptionChecker(boolean sidelineCorruptHFiles) throws IOException { + return new HFileCorruptionChecker(conf, executor, sidelineCorruptHFiles) { + boolean attemptedFirstFamDir = false; + protected void checkColFamDir(Path p) throws IOException { + if (!attemptedFirstFamDir) { + attemptedFirstFamDir = true; + assertTrue(fs.delete(p, true)); // make sure delete happened. + } + super.checkColFamDir(p); + } + }; + } + }; + doQuarantineTest(table, hbck, 3, 0, 0, 0, 1); + } + + /** + * This creates a table and simulates the race situation where a concurrent compaction or split + * has removed a region dir before the corruption checker got to it. + */ + @Test(timeout=180000) + public void testQuarantineMissingRegionDir() throws Exception { + String table = name.getMethodName(); + ExecutorService exec = new ScheduledThreadPoolExecutor(10); + // inject a fault in the hfcc created. + final FileSystem fs = FileSystem.get(conf); + HBaseFsck hbck = new HBaseFsck(conf, exec) { + public HFileCorruptionChecker createHFileCorruptionChecker(boolean sidelineCorruptHFiles) throws IOException { + return new HFileCorruptionChecker(conf, executor, sidelineCorruptHFiles) { + boolean attemptedFirstRegionDir = false; + protected void checkRegionDir(Path p) throws IOException { + if (!attemptedFirstRegionDir) { + attemptedFirstRegionDir = true; + assertTrue(fs.delete(p, true)); // make sure delete happened. + } + super.checkRegionDir(p); + } + }; + } + }; + doQuarantineTest(table, hbck, 3, 0, 0, 0, 1); + } + + /** + * Test fixing lingering reference file. + */ + @Test + public void testLingeringReferenceFile() throws Exception { + String table = "testLingeringReferenceFile"; + try { + setupTable(table); + assertEquals(ROWKEYS.length, countRows()); + + // Mess it up by creating a fake reference file + FileSystem fs = FileSystem.get(conf); + Path tableDir= FSUtils.getTablePath(FSUtils.getRootDir(conf), table); + Path regionDir = FSUtils.getRegionDirs(fs, tableDir).get(0); + Path famDir = new Path(regionDir, FAM_STR); + Path fakeReferenceFile = new Path(famDir, "fbce357483ceea.12144538"); + fs.create(fakeReferenceFile); + + HBaseFsck hbck = doFsck(conf, false); + assertErrors(hbck, new ERROR_CODE[] { ERROR_CODE.LINGERING_REFERENCE_HFILE }); + // fix reference file + doFsck(conf, true); + // check that reference file fixed + assertNoErrors(doFsck(conf, false)); + } finally { + deleteTable(table); + } + } + + /** + * Test pluggable error reporter. It can be plugged in + * from system property or configuration. + */ + @Test + public void testErrorReporter() throws Exception { + try { + MockErrorReporter.calledCount = 0; + doFsck(conf, false); + assertEquals(MockErrorReporter.calledCount, 0); + + conf.set("hbasefsck.errorreporter", MockErrorReporter.class.getName()); + doFsck(conf, false); + assertTrue(MockErrorReporter.calledCount > 20); + } finally { + conf.set("hbasefsck.errorreporter", + PrintingErrorReporter.class.getName()); + MockErrorReporter.calledCount = 0; + } + } + + static class MockErrorReporter implements ErrorReporter { + static int calledCount = 0; + + public void clear() { + calledCount++; + } + + public void report(String message) { + calledCount++; + } + + public void reportError(String message) { + calledCount++; + } + + public void reportError(ERROR_CODE errorCode, String message) { + calledCount++; + } + + public void reportError(ERROR_CODE errorCode, String message, TableInfo table) { + calledCount++; + } + + public void reportError(ERROR_CODE errorCode, + String message, TableInfo table, HbckInfo info) { + calledCount++; + } + + public void reportError(ERROR_CODE errorCode, String message, + TableInfo table, HbckInfo info1, HbckInfo info2) { + calledCount++; + } + + public int summarize() { + return ++calledCount; + } + + public void detail(String details) { + calledCount++; + } + + public ArrayList getErrorList() { + calledCount++; + return new ArrayList(); + } + + public void progress() { + calledCount++; + } + + public void print(String message) { + calledCount++; + } + + public void resetErrors() { + calledCount++; + } + + public boolean tableHasErrors(TableInfo table) { + calledCount++; + return false; + } + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); + + @org.junit.Rule + public TestName name = new TestName(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java b/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java index 0599da13eb45..a5096b933563 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -96,14 +95,6 @@ public void testAbsEndKey() { assertTrue(HBaseFsck.cmp.compare(hi2, hi1) > 0); } - @Test - public void testTiebreaker() { - HbckInfo hi1 = genHbckInfo(table, keyA, keyC, 0); - HbckInfo hi2 = genHbckInfo(table, keyA, keyC, 1); - assertTrue(HBaseFsck.cmp.compare(hi1, hi2) < 0); - assertTrue(HBaseFsck.cmp.compare(hi2, hi1) > 0); - } - @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java b/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java new file mode 100644 index 000000000000..82f96d912e46 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java @@ -0,0 +1,78 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.util; + +import static org.junit.Assert.*; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.SmallTests; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.mockito.Mockito; + +/** + * Test that the utility works as expected + */ +@Category(SmallTests.class) +public class TestHFileArchiveUtil { + + @Test + public void testGetTableArchivePath() { + assertNotNull(HFileArchiveUtil.getTableArchivePath(new Path("table"))); + assertNotNull(HFileArchiveUtil.getTableArchivePath(new Path("root", new Path("table")))); + } + + @Test + public void testGetArchivePath() throws Exception { + Configuration conf = new Configuration(); + FSUtils.setRootDir(conf, new Path("root")); + assertNotNull(HFileArchiveUtil.getArchivePath(conf)); + } + + @Test + public void testRegionArchiveDir() { + Configuration conf = null; + Path tableDir = new Path("table"); + Path regionDir = new Path("region"); + assertNotNull(HFileArchiveUtil.getRegionArchiveDir(conf, tableDir, regionDir)); + } + + @Test + public void testGetStoreArchivePath(){ + byte[] family = Bytes.toBytes("Family"); + Path tabledir = new Path("table"); + HRegionInfo region = new HRegionInfo(Bytes.toBytes("table")); + Configuration conf = null; + assertNotNull(HFileArchiveUtil.getStoreArchivePath(conf, region, tabledir, family)); + conf = new Configuration(); + assertNotNull(HFileArchiveUtil.getStoreArchivePath(conf, region, tabledir, family)); + + // do a little mocking of a region to get the same results + HRegion mockRegion = Mockito.mock(HRegion.class); + Mockito.when(mockRegion.getRegionInfo()).thenReturn(region); + Mockito.when(mockRegion.getTableDir()).thenReturn(tabledir); + + assertNotNull(HFileArchiveUtil.getStoreArchivePath(null, mockRegion, family)); + conf = new Configuration(); + assertNotNull(HFileArchiveUtil.getStoreArchivePath(conf, mockRegion, family)); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestIdLock.java b/src/test/java/org/apache/hadoop/hbase/util/TestIdLock.java index 478bfbdc54b6..ca2f39ce5f48 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestIdLock.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestIdLock.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,6 +19,8 @@ package org.apache.hadoop.hbase.util; +import static org.junit.Assert.assertTrue; + import java.util.Map; import java.util.Random; import java.util.concurrent.Callable; @@ -28,12 +29,10 @@ import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; - -import static org.junit.Assert.*; - import org.apache.hadoop.hbase.MediumTests; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -106,6 +105,7 @@ public void testMultipleClients() throws Exception { idLock.assertMapEmpty(); } finally { exec.shutdown(); + exec.awaitTermination(5000, TimeUnit.MILLISECONDS); } } diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestIncrementingEnvironmentEdge.java b/src/test/java/org/apache/hadoop/hbase/util/TestIncrementingEnvironmentEdge.java index f4fbce953f4b..a3d7eb8be971 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestIncrementingEnvironmentEdge.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestIncrementingEnvironmentEdge.java @@ -1,5 +1,4 @@ /* - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestKeying.java b/src/test/java/org/apache/hadoop/hbase/util/TestKeying.java index 88f5821f9f49..ebf1ccffd6b2 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestKeying.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestKeying.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java b/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java index 43aad3b40ada..f6408b85316e 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java @@ -39,8 +39,8 @@ public class TestLoadTestKVGenerator { @Test public void testValueLength() { for (int i = 0; i < 1000; ++i) { - byte[] v = gen.generateRandomSizeValue(i, - String.valueOf(rand.nextInt())); + byte[] v = gen.generateRandomSizeValue(Integer.toString(i).getBytes(), + String.valueOf(rand.nextInt()).getBytes()); assertTrue(MIN_LEN <= v.length); assertTrue(v.length <= MAX_LEN); } @@ -50,12 +50,12 @@ public void testValueLength() { public void testVerification() { for (int i = 0; i < 1000; ++i) { for (int qualIndex = 0; qualIndex < 20; ++qualIndex) { - String qual = String.valueOf(qualIndex); - byte[] v = gen.generateRandomSizeValue(i, qual); - String rowKey = LoadTestKVGenerator.md5PrefixedKey(i); - assertTrue(LoadTestKVGenerator.verify(rowKey, qual, v)); + byte[] qual = String.valueOf(qualIndex).getBytes(); + byte[] rowKey = LoadTestKVGenerator.md5PrefixedKey(i).getBytes(); + byte[] v = gen.generateRandomSizeValue(rowKey, qual); + assertTrue(LoadTestKVGenerator.verify(v, rowKey, qual)); v[0]++; - assertFalse(LoadTestKVGenerator.verify(rowKey, qual, v)); + assertFalse(LoadTestKVGenerator.verify(v, rowKey, qual)); } } } @@ -64,7 +64,7 @@ public void testVerification() { public void testCorrectAndUniqueKeys() { Set keys = new HashSet(); for (int i = 0; i < 1000; ++i) { - String k = LoadTestKVGenerator.md5PrefixedKey(i); + String k = gen.md5PrefixedKey(i); assertFalse(keys.contains(k)); assertTrue(k.endsWith("-" + i)); keys.add(k); diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java b/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java index a9948c7fe0ac..ef7bbeaf8301 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java @@ -1,5 +1,4 @@ /** - * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java b/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java index 465dc95292d5..ff619b8c3c14 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java @@ -1,5 +1,4 @@ /** - * Copyright 2008 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -169,6 +168,13 @@ public void setUp() throws Exception { @Override public void tearDown() throws Exception { super.tearDown(); + for (int i = 0; i < sourceRegions.length; i++) { + HRegion r = regions[i]; + if (r != null) { + r.close(); + r.getLog().closeAndDelete(); + } + } TEST_UTIL.shutdownMiniCluster(); } diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java b/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java index d24648c014d6..64c67d3a169b 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java @@ -29,6 +29,7 @@ import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.LargeTests; import org.apache.hadoop.hbase.TableNotFoundException; import org.apache.hadoop.hbase.client.HBaseAdmin; @@ -57,7 +58,7 @@ public class TestMiniClusterLoadSequential { protected static final byte[] CF = Bytes.toBytes("load_test_cf"); protected static final int NUM_THREADS = 8; protected static final int NUM_RS = 2; - protected static final int TIMEOUT_MS = 120000; + protected static final int TIMEOUT_MS = 180000; protected static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); @@ -102,6 +103,19 @@ public void tearDown() throws Exception { TEST_UTIL.shutdownMiniCluster(); } + protected MultiThreadedReader prepareReaderThreads(LoadTestDataGenerator dataGen, + Configuration conf, byte[] tableName, double verifyPercent) { + MultiThreadedReader reader = new MultiThreadedReader(dataGen, conf, tableName, verifyPercent); + return reader; + } + + protected MultiThreadedWriter prepareWriterThreads(LoadTestDataGenerator dataGen, + Configuration conf, byte[] tableName) { + MultiThreadedWriter writer = new MultiThreadedWriter(dataGen, conf, tableName); + writer.setMultiPut(isMultiPut); + return writer; + } + @Test(timeout=TIMEOUT_MS) public void loadTest() throws Exception { prepareForLoadTest(); @@ -120,6 +134,12 @@ protected void runLoadTestOnExistingTable() throws IOException { assertEquals(numKeys, readerThreads.getNumKeysVerified()); } + protected void createPreSplitLoadTestTable(HTableDescriptor htd, HColumnDescriptor hcd) + throws IOException { + HBaseTestingUtility.createPreSplitLoadTestTable(conf, htd, hcd); + TEST_UTIL.waitUntilAllRegionsAssigned(htd.getName()); + } + protected void prepareForLoadTest() throws IOException { LOG.info("Starting load test: dataBlockEncoding=" + dataBlockEncoding + ", isMultiPut=" + isMultiPut); @@ -131,14 +151,15 @@ protected void prepareForLoadTest() throws IOException { } admin.close(); - int numRegions = HBaseTestingUtility.createPreSplitLoadTestTable(conf, - TABLE, CF, compression, dataBlockEncoding); - - TEST_UTIL.waitUntilAllRegionsAssigned(numRegions); + HTableDescriptor htd = new HTableDescriptor(TABLE); + HColumnDescriptor hcd = new HColumnDescriptor(CF) + .setCompressionType(compression) + .setDataBlockEncoding(dataBlockEncoding); + createPreSplitLoadTestTable(htd, hcd); - writerThreads = new MultiThreadedWriter(conf, TABLE, CF); - writerThreads.setMultiPut(isMultiPut); - readerThreads = new MultiThreadedReader(conf, TABLE, CF, 100); + LoadTestDataGenerator dataGen = new MultiThreadedAction.DefaultDataGenerator(CF); + writerThreads = prepareWriterThreads(dataGen, conf, TABLE); + readerThreads = prepareReaderThreads(dataGen, conf, TABLE, 100); } protected int numKeys() { diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java b/src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java index bb4304c6afdd..ca639db9c1c5 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java b/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java index ac3b2250f56a..afef0ff9d4f5 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -20,10 +19,14 @@ package org.apache.hadoop.hbase.util; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import java.util.ArrayList; import java.util.Collection; import java.util.Comparator; +import java.util.List; import java.util.SortedSet; +import java.util.UUID; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -32,25 +35,26 @@ import com.google.common.collect.ComparisonChain; import com.google.common.collect.Multimap; + import org.junit.experimental.categories.Category; @Category(SmallTests.class) public class TestRegionSplitCalculator { - final static Log LOG = LogFactory.getLog(TestRegionSplitCalculator.class); + private static final Log LOG = LogFactory.getLog(TestRegionSplitCalculator.class); /** * This is range uses a user specified start and end keys. It also has an - * extra time based tiebreaker so that different ranges with the same - * start/end key pair count as different regions. + * extra tiebreaker so that different ranges with the same start/end key pair + * count as different regions. */ static class SimpleRange implements KeyRange { byte[] start, end; - long tiebreaker; + UUID tiebreaker; SimpleRange(byte[] start, byte[] end) { this.start = start; this.end = end; - this.tiebreaker = System.nanoTime(); + this.tiebreaker = UUID.randomUUID(); } @Override @@ -105,9 +109,9 @@ String dump(SortedSet splits, Multimap regions) { // we display this way because the last end key should be displayed as well. StringBuilder sb = new StringBuilder(); for (byte[] k : splits) { - sb.append(Bytes.toString(k) + ":\t"); + sb.append(Bytes.toString(k)).append(":\t"); for (SimpleRange r : regions.get(k)) { - sb.append(r.toString() + "\t"); + sb.append(r.toString()).append("\t"); } sb.append("\n"); } @@ -144,7 +148,7 @@ public void testSplitCalculatorNoEdge() { LOG.info("Empty"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions); - assertEquals(res, ""); + assertEquals("", res); } @Test @@ -158,7 +162,7 @@ public void testSplitCalculatorSingleEdge() { LOG.info("Single edge"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 1, 0); - assertEquals(res, "A:\t[A, B]\t\n" + "B:\t\n"); + assertEquals("A:\t[A, B]\t\n" + "B:\t\n", res); } @Test @@ -172,7 +176,7 @@ public void testSplitCalculatorDegenerateEdge() { LOG.info("Single empty edge"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 1); - assertEquals(res, "A:\t[A, A]\t\n"); + assertEquals("A:\t[A, A]\t\n", res); } @Test @@ -190,8 +194,8 @@ public void testSplitCalculatorCoverSplit() { LOG.info("AC covers AB, BC"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 2, 2, 0); - assertEquals(res, "A:\t[A, B]\t[A, C]\t\n" + "B:\t[A, C]\t[B, C]\t\n" - + "C:\t\n"); + assertEquals("A:\t[A, B]\t[A, C]\t\n" + "B:\t[A, C]\t[B, C]\t\n" + + "C:\t\n", res); } @Test @@ -209,8 +213,8 @@ public void testSplitCalculatorOverEndpoint() { LOG.info("AB, BD covers BC"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 1, 2, 1, 0); - assertEquals(res, "A:\t[A, B]\t\n" + "B:\t[B, C]\t[B, D]\t\n" - + "C:\t[B, D]\t\n" + "D:\t\n"); + assertEquals("A:\t[A, B]\t\n" + "B:\t[B, C]\t[B, D]\t\n" + + "C:\t[B, D]\t\n" + "D:\t\n", res); } @Test @@ -228,8 +232,8 @@ public void testSplitCalculatorHoles() { LOG.info("Hole between C and E"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 1, 1, 0, 1, 0); - assertEquals(res, "A:\t[A, B]\t\n" + "B:\t[B, C]\t\n" + "C:\t\n" - + "E:\t[E, F]\t\n" + "F:\t\n"); + assertEquals("A:\t[A, B]\t\n" + "B:\t[B, C]\t\n" + "C:\t\n" + + "E:\t[E, F]\t\n" + "F:\t\n", res); } @Test @@ -245,8 +249,8 @@ public void testSplitCalculatorOverreach() { LOG.info("AC and BD overlap but share no start/end keys"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 1, 2, 1, 0); - assertEquals(res, "A:\t[A, C]\t\n" + "B:\t[A, C]\t[B, D]\t\n" - + "C:\t[B, D]\t\n" + "D:\t\n"); + assertEquals("A:\t[A, C]\t\n" + "B:\t[A, C]\t[B, D]\t\n" + + "C:\t[B, D]\t\n" + "D:\t\n", res); } @Test @@ -262,7 +266,7 @@ public void testSplitCalculatorFloor() { LOG.info("AC and AB overlap in the beginning"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 2, 1, 0); - assertEquals(res, "A:\t[A, B]\t[A, C]\t\n" + "B:\t[A, C]\t\n" + "C:\t\n"); + assertEquals("A:\t[A, B]\t[A, C]\t\n" + "B:\t[A, C]\t\n" + "C:\t\n", res); } @Test @@ -278,13 +282,15 @@ public void testSplitCalculatorCeil() { LOG.info("AC and BC overlap in the end"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 1, 2, 0); - assertEquals(res, "A:\t[A, C]\t\n" + "B:\t[A, C]\t[B, C]\t\n" + "C:\t\n"); + assertEquals("A:\t[A, C]\t\n" + "B:\t[A, C]\t[B, C]\t\n" + "C:\t\n", res); } @Test public void testSplitCalculatorEq() { SimpleRange a = new SimpleRange(Bytes.toBytes("A"), Bytes.toBytes("C")); SimpleRange b = new SimpleRange(Bytes.toBytes("A"), Bytes.toBytes("C")); + + LOG.info(a.tiebreaker + " - " + b.tiebreaker); RegionSplitCalculator sc = new RegionSplitCalculator( cmp); sc.add(a); @@ -294,7 +300,7 @@ public void testSplitCalculatorEq() { LOG.info("AC and AC overlap completely"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 2, 0); - assertEquals(res, "A:\t[A, C]\t[A, C]\t\n" + "C:\t\n"); + assertEquals("A:\t[A, C]\t[A, C]\t\n" + "C:\t\n", res); } @Test @@ -308,7 +314,7 @@ public void testSplitCalculatorBackwards() { LOG.info("CA is backwards"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions); // expect nothing - assertEquals(res, ""); + assertEquals("", res); } @Test @@ -328,11 +334,11 @@ public void testComplex() { LOG.info("Something fairly complex"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 3, 3, 3, 1, 2, 0, 1, 0, 1, 0); - assertEquals(res, "A:\t[A, Am]\t[A, B]\t[A, C]\t\n" + assertEquals("A:\t[A, Am]\t[A, B]\t[A, C]\t\n" + "Am:\t[A, B]\t[A, C]\t[Am, C]\t\n" + "B:\t[A, C]\t[Am, C]\t[B, E]\t\n" + "C:\t[B, E]\t\n" + "D:\t[B, E]\t[D, E]\t\n" + "E:\t\n" + "F:\t[F, G]\t\n" + "G:\t\n" - + "H:\t[H, I]\t\n" + "I:\t\n"); + + "H:\t[H, I]\t\n" + "I:\t\n", res); } @Test @@ -347,8 +353,44 @@ public void testBeginEndMarker() { LOG.info("Special cases -- empty"); String res = dump(sc.getSplits(), regions); checkDepths(sc.getSplits(), regions, 1, 1, 1, 0); - assertEquals(res, ":\t[, A]\t\n" + "A:\t[A, B]\t\n" + "B:\t[B, ]\t\n" - + "null:\t\n"); + assertEquals(":\t[, A]\t\n" + "A:\t[A, B]\t\n" + "B:\t[B, ]\t\n" + + "null:\t\n", res); + } + + @Test + public void testBigRanges() { + SimpleRange ai = new SimpleRange(Bytes.toBytes("A"), Bytes.toBytes("I")); + SimpleRange ae = new SimpleRange(Bytes.toBytes("A"), Bytes.toBytes("E")); + SimpleRange ac = new SimpleRange(Bytes.toBytes("A"), Bytes.toBytes("C")); + + Collection bigOverlap = new ArrayList(); + bigOverlap.add(new SimpleRange(Bytes.toBytes("A"), Bytes.toBytes("E"))); + bigOverlap.add(new SimpleRange(Bytes.toBytes("A"), Bytes.toBytes("C"))); + bigOverlap.add(new SimpleRange(Bytes.toBytes("A"), Bytes.toBytes("B"))); + bigOverlap.add(new SimpleRange(Bytes.toBytes("B"), Bytes.toBytes("C"))); + bigOverlap.add(new SimpleRange(Bytes.toBytes("E"), Bytes.toBytes("H"))); + bigOverlap.add(ai); + bigOverlap.add(ae); + bigOverlap.add(ac); + + // Expect 1 range to be returned: ai + List bigRanges = RegionSplitCalculator.findBigRanges(bigOverlap, 1); + assertEquals(1, bigRanges.size()); + assertEquals(ai, bigRanges.get(0)); + + // Expect 3 ranges to be returned: ai, ae and ac + bigRanges = RegionSplitCalculator.findBigRanges(bigOverlap, 3); + assertEquals(3, bigRanges.size()); + assertEquals(ai, bigRanges.get(0)); + + SimpleRange r1 = bigRanges.get(1); + SimpleRange r2 = bigRanges.get(2); + assertEquals("A", Bytes.toString(r1.start)); + assertEquals("A", Bytes.toString(r2.start)); + String r1e = Bytes.toString(r1.end); + String r2e = Bytes.toString(r2.end); + assertTrue((r1e.equals("C") && r2e.equals("E")) + || (r1e.equals("E") && r2e.equals("C"))); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java b/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java index 2a72c715787b..6f616e6dbc4c 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java @@ -1,5 +1,4 @@ /** - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestRootPath.java b/src/test/java/org/apache/hadoop/hbase/util/TestRootPath.java index c596018c331e..6d27552de05f 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestRootPath.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestRootPath.java @@ -1,5 +1,4 @@ /** - * Copyright 2008 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestSizeBasedThrottler.java b/src/test/java/org/apache/hadoop/hbase/util/TestSizeBasedThrottler.java new file mode 100644 index 000000000000..180ebae4150a --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/util/TestSizeBasedThrottler.java @@ -0,0 +1,135 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.util; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.fail; + +import java.util.ArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.hadoop.hbase.MediumTests; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * This tests some race conditions that can happen + * occasionally, but not every time. + */ +@Category(MediumTests.class) +public class TestSizeBasedThrottler { + + private static final int REPEATS = 100; + + private Thread makeThread(final SizeBasedThrottler throttler, + final AtomicBoolean failed, final int delta, + final int limit, final CountDownLatch latch) { + + Thread ret = new Thread(new Runnable() { + + @Override + public void run() { + try { + latch.await(); + if (throttler.increase(delta) > limit) { + failed.set(true); + } + throttler.decrease(delta); + } catch (Exception e) { + failed.set(true); + } + } + }); + + ret.start(); + return ret; + } + + private void runGenericTest(int threshold, int delta, int maxValueAllowed, + int numberOfThreads, long timeout) { + SizeBasedThrottler throttler = new SizeBasedThrottler(threshold); + AtomicBoolean failed = new AtomicBoolean(false); + + ArrayList threads = new ArrayList(numberOfThreads); + CountDownLatch latch = new CountDownLatch(1); + long timeElapsed = 0; + + for (int i = 0; i < numberOfThreads; ++i) { + threads.add(makeThread(throttler, failed, delta, maxValueAllowed, latch)); + } + + latch.countDown(); + for (Thread t : threads) { + try { + long beforeJoin = System.currentTimeMillis(); + t.join(timeout - timeElapsed); + timeElapsed += System.currentTimeMillis() - beforeJoin; + if (t.isAlive() || timeElapsed >= timeout) { + fail("Timeout reached."); + } + } catch (InterruptedException e) { + fail("Got InterruptedException"); + } + } + + assertFalse(failed.get()); + } + + @Test + public void testSmallIncreases(){ + for (int i = 0; i < REPEATS; ++i) { + runGenericTest( + 10, // threshold + 1, // delta + 15, // fail if throttler's value + // exceeds 15 + 1000, // use 1000 threads + 5000 // wait for 5 sec + ); + } + } + + @Test + public void testBigIncreases() { + for (int i = 0; i < REPEATS; ++i) { + runGenericTest( + 1, // threshold + 2, // delta + 4, // fail if throttler's value + // exceeds 4 + 1000, // use 1000 threads + 5000 // wait for 5 sec + ); + } + } + + @Test + public void testIncreasesEqualToThreshold(){ + for (int i = 0; i < REPEATS; ++i) { + runGenericTest( + 1, // threshold + 1, // delta + 2, // fail if throttler's value + // exceeds 2 + 1000, // use 1000 threads + 5000 // wait for 5 sec + ); + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestSortedCopyOnWriteSet.java b/src/test/java/org/apache/hadoop/hbase/util/TestSortedCopyOnWriteSet.java index 29e025abc127..49f3cd6d94db 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestSortedCopyOnWriteSet.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestSortedCopyOnWriteSet.java @@ -1,5 +1,4 @@ /* - * Copyright 2011 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/util/TestThreads.java b/src/test/java/org/apache/hadoop/hbase/util/TestThreads.java index 3bd39afb1295..6c249c746d5e 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/TestThreads.java +++ b/src/test/java/org/apache/hadoop/hbase/util/TestThreads.java @@ -35,7 +35,7 @@ public class TestThreads { private volatile boolean wasInterrupted; - @Test(timeout=6000) + @Test(timeout=60000) public void testSleepWithoutInterrupt() throws InterruptedException { Thread sleeper = new Thread(new Runnable() { @Override diff --git a/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java b/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java index dbb97f8f2bd3..99f4f9b8a1c8 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java +++ b/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java @@ -19,27 +19,68 @@ import static org.junit.Assert.assertEquals; +import java.util.ArrayList; import java.util.Arrays; import java.util.List; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.ScheduledThreadPoolExecutor; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.util.HBaseFsck; import org.apache.hadoop.hbase.util.HBaseFsck.ErrorReporter.ERROR_CODE; public class HbckTestingUtil { - public static HBaseFsck doFsck(Configuration conf, boolean fix) throws Exception { - HBaseFsck fsck = new HBaseFsck(conf); + private static ExecutorService exec = new ScheduledThreadPoolExecutor(10); + public static HBaseFsck doFsck( + Configuration conf, boolean fix) throws Exception { + return doFsck(conf, fix, null); + } + + public static HBaseFsck doFsck( + Configuration conf, boolean fix, String table) throws Exception { + return doFsck(conf, fix, fix, fix, fix,fix, fix, fix, fix, table); + } + + public static HBaseFsck doFsck(Configuration conf, boolean fixAssignments, + boolean fixMeta, boolean fixHdfsHoles, boolean fixHdfsOverlaps, + boolean fixHdfsOrphans, boolean fixTableOrphans, boolean fixVersionFile, + boolean fixReferenceFiles, String table) throws Exception { + HBaseFsck fsck = new HBaseFsck(conf, exec); fsck.connect(); - fsck.displayFullReport(); // i.e. -details + fsck.setDisplayFullReport(); // i.e. -details fsck.setTimeLag(0); - fsck.setFixErrors(fix); - fsck.doWork(); + fsck.setFixAssignments(fixAssignments); + fsck.setFixMeta(fixMeta); + fsck.setFixHdfsHoles(fixHdfsHoles); + fsck.setFixHdfsOverlaps(fixHdfsOverlaps); + fsck.setFixHdfsOrphans(fixHdfsOrphans); + fsck.setFixTableOrphans(fixTableOrphans); + fsck.setFixVersionFile(fixVersionFile); + fsck.setFixReferenceFiles(fixReferenceFiles); + if (table != null) { + fsck.includeTable(table); + } + fsck.onlineHbck(); return fsck; } + /** + * Runs hbck with the -sidelineCorruptHFiles option + * @param conf + * @param table table constraint + * @return + * @throws Exception + */ + public static HBaseFsck doHFileQuarantine(Configuration conf, String table) throws Exception { + String[] args = {"-sidelineCorruptHFiles", "-ignorePreCheckPermission", table}; + HBaseFsck hbck = new HBaseFsck(conf, exec); + hbck.exec(exec, args); + return hbck; + } + public static void assertNoErrors(HBaseFsck fsck) throws Exception { List errs = fsck.getErrors().getErrorList(); - assertEquals(0, errs.size()); + assertEquals(new ArrayList(), errs); } public static void assertErrors(HBaseFsck fsck, ERROR_CODE[] expectedErrors) { diff --git a/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java b/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java index 201d38cbf6df..078f1f68aa5c 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java +++ b/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java @@ -102,7 +102,7 @@ public void setUpBefore() throws Exception { @After public void tearDownAfter() throws Exception { TEST_UTIL.shutdownMiniCluster(); - HConnectionManager.deleteConnection(conf, true); + HConnectionManager.deleteConnection(conf); } /** diff --git a/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java b/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java index 2b4cac86cb76..4bdc6454f58e 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java +++ b/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java @@ -57,11 +57,11 @@ public void testMetaRebuild() throws Exception { // shutdown the minicluster TEST_UTIL.shutdownMiniHBaseCluster(); TEST_UTIL.shutdownMiniZKCluster(); - HConnectionManager.deleteConnection(conf, false); + HConnectionManager.deleteConnection(conf); // rebuild meta table from scratch HBaseFsck fsck = new HBaseFsck(conf); - assertTrue(fsck.rebuildMeta()); + assertTrue(fsck.rebuildMeta(false)); // bring up the minicluster TEST_UTIL.startMiniZKCluster(); // tables seem enabled by default diff --git a/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java b/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java index ebbeeada0d36..d5274986696b 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java +++ b/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java @@ -21,6 +21,7 @@ import static org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; import java.util.Arrays; @@ -64,7 +65,7 @@ public void testMetaRebuildHoleFail() throws Exception { // attempt to rebuild meta table from scratch HBaseFsck fsck = new HBaseFsck(conf); - assertFalse(fsck.rebuildMeta()); + assertFalse(fsck.rebuildMeta(false)); // bring up the minicluster TEST_UTIL.startMiniZKCluster(); // tables seem enabled by default diff --git a/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java b/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java index b17554819a98..67d7c83afed1 100644 --- a/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java +++ b/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java @@ -69,7 +69,7 @@ public void testMetaRebuildOverlapFail() throws Exception { // attempt to rebuild meta table from scratch HBaseFsck fsck = new HBaseFsck(conf); - assertFalse(fsck.rebuildMeta()); + assertFalse(fsck.rebuildMeta(false)); Multimap problems = fsck.getOverlapGroups(table); assertEquals(1, problems.keySet().size()); diff --git a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java index 764ac1984f10..3e1c89412856 100644 --- a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java +++ b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java @@ -1,5 +1,4 @@ /** - * Copyright 2009 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestRecoverableZooKeeper.java b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestRecoverableZooKeeper.java new file mode 100644 index 000000000000..a69388469744 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestRecoverableZooKeeper.java @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.zookeeper; + +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.lang.reflect.Field; +import java.util.Properties; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.Abortable; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.Watcher; +import org.apache.zookeeper.ZooDefs.Ids; +import org.apache.zookeeper.ZooKeeper; +import org.apache.zookeeper.data.Stat; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(MediumTests.class) +public class TestRecoverableZooKeeper { + + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + Abortable abortable = new Abortable() { + @Override + public void abort(String why, Throwable e) { + + } + + @Override + public boolean isAborted() { + return false; + } + }; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.startMiniZKCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniZKCluster(); + } + + @Test + public void testSetDataVersionMismatchInLoop() throws Exception { + String znode = "/hbase/unassigned/9af7cfc9b15910a0b3d714bf40a3248f"; + Configuration conf = TEST_UTIL.getConfiguration(); + Properties properties = ZKConfig.makeZKProps(conf); + ZooKeeperWatcher zkw = new ZooKeeperWatcher(conf, "testSetDataVersionMismatchInLoop", + abortable, true); + String ensemble = ZKConfig.getZKQuorumServersString(properties); + RecoverableZooKeeper rzk = ZKUtil.connect(conf, ensemble, zkw); + rzk.create(znode, new byte[0], Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); + rzk.setData(znode, "OPENING".getBytes(), 0); + Field zkField = RecoverableZooKeeper.class.getDeclaredField("zk"); + zkField.setAccessible(true); + int timeout = conf.getInt(HConstants.ZK_SESSION_TIMEOUT, HConstants.DEFAULT_ZK_SESSION_TIMEOUT); + ZookeeperStub zkStub = new ZookeeperStub(ensemble, timeout, zkw); + zkStub.setThrowExceptionInNumOperations(1); + zkField.set(rzk, zkStub); + byte[] opened = "OPENED".getBytes(); + rzk.setData(znode, opened, 1); + byte[] data = rzk.getData(znode, false, new Stat()); + assertTrue(Bytes.equals(opened, data)); + } + + class ZookeeperStub extends ZooKeeper { + + private int throwExceptionInNumOperations; + + public ZookeeperStub(String connectString, int sessionTimeout, Watcher watcher) + throws IOException { + super(connectString, sessionTimeout, watcher); + } + + public void setThrowExceptionInNumOperations(int throwExceptionInNumOperations) { + this.throwExceptionInNumOperations = throwExceptionInNumOperations; + } + + private void checkThrowKeeperException() throws KeeperException { + if (throwExceptionInNumOperations == 1) { + throwExceptionInNumOperations = 0; + throw new KeeperException.ConnectionLossException(); + } + if (throwExceptionInNumOperations > 0) + throwExceptionInNumOperations--; + } + + @Override + public Stat setData(String path, byte[] data, int version) throws KeeperException, + InterruptedException { + Stat stat = super.setData(path, data, version); + checkThrowKeeperException(); + return stat; + } + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java index 43046809e257..8df742cc9e1d 100644 --- a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java +++ b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java @@ -204,8 +204,8 @@ public void testLeaderSelection() throws Exception { private MockLeader getCurrentLeader() throws Exception { MockLeader currentLeader = null; outer: - // wait up to 2 secs for initial leader - for (int i = 0; i < 20; i++) { + // wait up to 10 secs for initial leader + for (int i = 0; i < 100; i++) { for (int j = 0; j < CANDIDATES.length; j++) { if (CANDIDATES[j].isMaster()) { // should only be one leader diff --git a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java new file mode 100644 index 000000000000..dd00372a94c5 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java @@ -0,0 +1,291 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.zookeeper; + + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.util.LinkedList; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.Abortable; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.MediumTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.zookeeper.ZKUtil.ZKUtilOp; +import org.apache.zookeeper.KeeperException; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +/** + * Test ZooKeeper multi-update functionality + */ +@Category(MediumTests.class) +public class TestZKMulti { + private static final Log LOG = LogFactory.getLog(TestZKMulti.class); + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + private static ZooKeeperWatcher zkw = null; + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.startMiniZKCluster(); + Configuration conf = TEST_UTIL.getConfiguration(); + conf.setBoolean("hbase.zookeeper.useMulti", true); + Abortable abortable = new Abortable() { + @Override + public void abort(String why, Throwable e) { + LOG.info(why, e); + } + + @Override + public boolean isAborted() { + return false; + } + }; + zkw = new ZooKeeperWatcher(conf, + "TestZKMulti", abortable, true); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniZKCluster(); + } + + @Test + public void testSimpleMulti() throws Exception { + // null multi + ZKUtil.multiOrSequential(zkw, null, false); + + // empty multi + ZKUtil.multiOrSequential(zkw, new LinkedList(), false); + + // single create + String path = ZKUtil.joinZNode(zkw.baseZNode, "testSimpleMulti"); + LinkedList singleCreate = new LinkedList(); + singleCreate.add(ZKUtilOp.createAndFailSilent(path, new byte[0])); + ZKUtil.multiOrSequential(zkw, singleCreate, false); + assertTrue(ZKUtil.checkExists(zkw, path) != -1); + + // single setdata + LinkedList singleSetData = new LinkedList(); + byte [] data = Bytes.toBytes("foobar"); + singleSetData.add(ZKUtilOp.setData(path, data)); + ZKUtil.multiOrSequential(zkw, singleSetData, false); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path), data)); + + // single delete + LinkedList singleDelete = new LinkedList(); + singleDelete.add(ZKUtilOp.deleteNodeFailSilent(path)); + ZKUtil.multiOrSequential(zkw, singleDelete, false); + assertTrue(ZKUtil.checkExists(zkw, path) == -1); + } + + @Test + public void testComplexMulti() throws Exception { + String path1 = ZKUtil.joinZNode(zkw.baseZNode, "testComplexMulti1"); + String path2 = ZKUtil.joinZNode(zkw.baseZNode, "testComplexMulti2"); + String path3 = ZKUtil.joinZNode(zkw.baseZNode, "testComplexMulti3"); + String path4 = ZKUtil.joinZNode(zkw.baseZNode, "testComplexMulti4"); + String path5 = ZKUtil.joinZNode(zkw.baseZNode, "testComplexMulti5"); + String path6 = ZKUtil.joinZNode(zkw.baseZNode, "testComplexMulti6"); + // create 4 nodes that we'll setData on or delete later + LinkedList create4Nodes = new LinkedList(); + create4Nodes.add(ZKUtilOp.createAndFailSilent(path1, Bytes.toBytes(path1))); + create4Nodes.add(ZKUtilOp.createAndFailSilent(path2, Bytes.toBytes(path2))); + create4Nodes.add(ZKUtilOp.createAndFailSilent(path3, Bytes.toBytes(path3))); + create4Nodes.add(ZKUtilOp.createAndFailSilent(path4, Bytes.toBytes(path4))); + ZKUtil.multiOrSequential(zkw, create4Nodes, false); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path1), Bytes.toBytes(path1))); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path2), Bytes.toBytes(path2))); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path3), Bytes.toBytes(path3))); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path4), Bytes.toBytes(path4))); + + // do multiple of each operation (setData, delete, create) + LinkedList ops = new LinkedList(); + // setData + ops.add(ZKUtilOp.setData(path1, Bytes.add(Bytes.toBytes(path1), Bytes.toBytes(path1)))); + ops.add(ZKUtilOp.setData(path2, Bytes.add(Bytes.toBytes(path2), Bytes.toBytes(path2)))); + // delete + ops.add(ZKUtilOp.deleteNodeFailSilent(path3)); + ops.add(ZKUtilOp.deleteNodeFailSilent(path4)); + // create + ops.add(ZKUtilOp.createAndFailSilent(path5, Bytes.toBytes(path5))); + ops.add(ZKUtilOp.createAndFailSilent(path6, Bytes.toBytes(path6))); + ZKUtil.multiOrSequential(zkw, ops, false); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path1), + Bytes.add(Bytes.toBytes(path1), Bytes.toBytes(path1)))); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path2), + Bytes.add(Bytes.toBytes(path2), Bytes.toBytes(path2)))); + assertTrue(ZKUtil.checkExists(zkw, path3) == -1); + assertTrue(ZKUtil.checkExists(zkw, path4) == -1); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path5), Bytes.toBytes(path5))); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path6), Bytes.toBytes(path6))); + } + + @Test + public void testSingleFailure() throws Exception { + // try to delete a node that doesn't exist + boolean caughtNoNode = false; + String path = ZKUtil.joinZNode(zkw.baseZNode, "testSingleFailureZ"); + LinkedList ops = new LinkedList(); + ops.add(ZKUtilOp.deleteNodeFailSilent(path)); + try { + ZKUtil.multiOrSequential(zkw, ops, false); + } catch (KeeperException.NoNodeException nne) { + caughtNoNode = true; + } + assertTrue(caughtNoNode); + + // try to setData on a node that doesn't exist + caughtNoNode = false; + ops = new LinkedList(); + ops.add(ZKUtilOp.setData(path, Bytes.toBytes(path))); + try { + ZKUtil.multiOrSequential(zkw, ops, false); + } catch (KeeperException.NoNodeException nne) { + caughtNoNode = true; + } + assertTrue(caughtNoNode); + + // try to create on a node that already exists + boolean caughtNodeExists = false; + ops = new LinkedList(); + ops.add(ZKUtilOp.createAndFailSilent(path, Bytes.toBytes(path))); + ZKUtil.multiOrSequential(zkw, ops, false); + try { + ZKUtil.multiOrSequential(zkw, ops, false); + } catch (KeeperException.NodeExistsException nee) { + caughtNodeExists = true; + } + assertTrue(caughtNodeExists); + } + + @Test + public void testSingleFailureInMulti() throws Exception { + // try a multi where all but one operation succeeds + String pathA = ZKUtil.joinZNode(zkw.baseZNode, "testSingleFailureInMultiA"); + String pathB = ZKUtil.joinZNode(zkw.baseZNode, "testSingleFailureInMultiB"); + String pathC = ZKUtil.joinZNode(zkw.baseZNode, "testSingleFailureInMultiC"); + LinkedList ops = new LinkedList(); + ops.add(ZKUtilOp.createAndFailSilent(pathA, Bytes.toBytes(pathA))); + ops.add(ZKUtilOp.createAndFailSilent(pathB, Bytes.toBytes(pathB))); + ops.add(ZKUtilOp.deleteNodeFailSilent(pathC)); + boolean caughtNoNode = false; + try { + ZKUtil.multiOrSequential(zkw, ops, false); + } catch (KeeperException.NoNodeException nne) { + caughtNoNode = true; + } + assertTrue(caughtNoNode); + // assert that none of the operations succeeded + assertTrue(ZKUtil.checkExists(zkw, pathA) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathB) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathC) == -1); + } + + @Test + public void testMultiFailure() throws Exception { + String pathX = ZKUtil.joinZNode(zkw.baseZNode, "testMultiFailureX"); + String pathY = ZKUtil.joinZNode(zkw.baseZNode, "testMultiFailureY"); + String pathZ = ZKUtil.joinZNode(zkw.baseZNode, "testMultiFailureZ"); + // create X that we will use to fail create later + LinkedList ops = new LinkedList(); + ops.add(ZKUtilOp.createAndFailSilent(pathX, Bytes.toBytes(pathX))); + ZKUtil.multiOrSequential(zkw, ops, false); + + // fail one of each create ,setData, delete + String pathV = ZKUtil.joinZNode(zkw.baseZNode, "testMultiFailureV"); + String pathW = ZKUtil.joinZNode(zkw.baseZNode, "testMultiFailureW"); + ops = new LinkedList(); + ops.add(ZKUtilOp.createAndFailSilent(pathX, Bytes.toBytes(pathX))); // fail -- already exists + ops.add(ZKUtilOp.setData(pathY, Bytes.toBytes(pathY))); // fail -- doesn't exist + ops.add(ZKUtilOp.deleteNodeFailSilent(pathZ)); // fail -- doesn't exist + ops.add(ZKUtilOp.createAndFailSilent(pathX, Bytes.toBytes(pathV))); // pass + ops.add(ZKUtilOp.createAndFailSilent(pathX, Bytes.toBytes(pathW))); // pass + boolean caughtNodeExists = false; + try { + ZKUtil.multiOrSequential(zkw, ops, false); + } catch (KeeperException.NodeExistsException nee) { + // check first operation that fails throws exception + caughtNodeExists = true; + } + assertTrue(caughtNodeExists); + // check that no modifications were made + assertFalse(ZKUtil.checkExists(zkw, pathX) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathY) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathZ) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathW) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathV) == -1); + + // test that with multiple failures, throws an exception corresponding to first failure in list + ops = new LinkedList(); + ops.add(ZKUtilOp.setData(pathY, Bytes.toBytes(pathY))); // fail -- doesn't exist + ops.add(ZKUtilOp.createAndFailSilent(pathX, Bytes.toBytes(pathX))); // fail -- exists + boolean caughtNoNode = false; + try { + ZKUtil.multiOrSequential(zkw, ops, false); + } catch (KeeperException.NoNodeException nne) { + // check first operation that fails throws exception + caughtNoNode = true; + } + assertTrue(caughtNoNode); + // check that no modifications were made + assertFalse(ZKUtil.checkExists(zkw, pathX) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathY) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathZ) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathW) == -1); + assertTrue(ZKUtil.checkExists(zkw, pathV) == -1); + } + + @Test + public void testRunSequentialOnMultiFailure() throws Exception { + String path1 = ZKUtil.joinZNode(zkw.baseZNode, "runSequential1"); + String path2 = ZKUtil.joinZNode(zkw.baseZNode, "runSequential2"); + String path3 = ZKUtil.joinZNode(zkw.baseZNode, "runSequential3"); + String path4 = ZKUtil.joinZNode(zkw.baseZNode, "runSequential4"); + + // create some nodes that we will use later + LinkedList ops = new LinkedList(); + ops.add(ZKUtilOp.createAndFailSilent(path1, Bytes.toBytes(path1))); + ops.add(ZKUtilOp.createAndFailSilent(path2, Bytes.toBytes(path2))); + ZKUtil.multiOrSequential(zkw, ops, false); + + // test that, even with operations that fail, the ones that would pass will pass + // with runSequentialOnMultiFailure + ops = new LinkedList(); + ops.add(ZKUtilOp.setData(path1, Bytes.add(Bytes.toBytes(path1), Bytes.toBytes(path1)))); // pass + ops.add(ZKUtilOp.deleteNodeFailSilent(path2)); // pass + ops.add(ZKUtilOp.deleteNodeFailSilent(path3)); // fail -- node doesn't exist + ops.add(ZKUtilOp.createAndFailSilent(path4, + Bytes.add(Bytes.toBytes(path4), Bytes.toBytes(path4)))); // pass + ZKUtil.multiOrSequential(zkw, ops, true); + assertTrue(Bytes.equals(ZKUtil.getData(zkw, path1), + Bytes.add(Bytes.toBytes(path1), Bytes.toBytes(path1)))); + assertTrue(ZKUtil.checkExists(zkw, path2) == -1); + assertTrue(ZKUtil.checkExists(zkw, path3) == -1); + assertFalse(ZKUtil.checkExists(zkw, path4) == -1); + } +} diff --git a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java index 3de482bfc778..cc5cdf2512da 100644 --- a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java +++ b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -27,16 +26,20 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.zookeeper.ZKTable.TableState; import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.data.Stat; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; +import org.mockito.Mockito; @Category(MediumTests.class) public class TestZKTable { - private static final Log LOG = LogFactory.getLog(TestZooKeeperNodeTracker.class); + private static final Log LOG = LogFactory.getLog(TestZKTable.class); private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); @BeforeClass @@ -49,52 +52,247 @@ public static void tearDownAfterClass() throws Exception { TEST_UTIL.shutdownMiniZKCluster(); } + Abortable abortable = new Abortable() { + @Override + public void abort(String why, Throwable e) { + LOG.info(why, e); + } + + @Override + public boolean isAborted() { + return false; + } + }; + @Test public void testTableStates() throws ZooKeeperConnectionException, IOException, KeeperException { final String name = "testDisabled"; - Abortable abortable = new Abortable() { - @Override - public void abort(String why, Throwable e) { - LOG.info(why, e); - } - - @Override - public boolean isAborted() { - return false; - } - - }; + ZooKeeperWatcher zkw = new ZooKeeperWatcher(TEST_UTIL.getConfiguration(), name, abortable, true); ZKTable zkt = new ZKTable(zkw); - assertTrue(zkt.isEnabledTable(name)); + assertFalse(zkt.isEnabledTable(name)); assertFalse(zkt.isDisablingTable(name)); assertFalse(zkt.isDisabledTable(name)); assertFalse(zkt.isEnablingTable(name)); assertFalse(zkt.isDisablingOrDisabledTable(name)); assertFalse(zkt.isDisabledOrEnablingTable(name)); + assertFalse(zkt.isTablePresent(name)); zkt.setDisablingTable(name); assertTrue(zkt.isDisablingTable(name)); assertTrue(zkt.isDisablingOrDisabledTable(name)); assertFalse(zkt.getDisabledTables().contains(name)); + assertTrue(zkt.isTablePresent(name)); zkt.setDisabledTable(name); assertTrue(zkt.isDisabledTable(name)); assertTrue(zkt.isDisablingOrDisabledTable(name)); assertFalse(zkt.isDisablingTable(name)); assertTrue(zkt.getDisabledTables().contains(name)); + assertTrue(zkt.isTablePresent(name)); zkt.setEnablingTable(name); assertTrue(zkt.isEnablingTable(name)); assertTrue(zkt.isDisabledOrEnablingTable(name)); assertFalse(zkt.isDisabledTable(name)); assertFalse(zkt.getDisabledTables().contains(name)); + assertTrue(zkt.isTablePresent(name)); zkt.setEnabledTable(name); assertTrue(zkt.isEnabledTable(name)); assertFalse(zkt.isEnablingTable(name)); + assertTrue(zkt.isTablePresent(name)); + zkt.setDeletedTable(name); + assertFalse(zkt.isEnabledTable(name)); + assertFalse(zkt.isDisablingTable(name)); + assertFalse(zkt.isDisabledTable(name)); + assertFalse(zkt.isEnablingTable(name)); + assertFalse(zkt.isDisablingOrDisabledTable(name)); + assertFalse(zkt.isDisabledOrEnablingTable(name)); + assertFalse(zkt.isTablePresent(name)); + } + + private void runTest9294CompatibilityTest(String tableName, Configuration conf) + throws Exception { + ZooKeeperWatcher zkw = new ZooKeeperWatcher(conf, + tableName, abortable, true); + ZKTable zkt = new ZKTable(zkw); + zkt.setEnabledTable(tableName); + // check that current/0.94 format table has proper ENABLED format + assertTrue( + ZKTableReadOnly.getTableState(zkw, zkw.masterTableZNode, tableName) == TableState.ENABLED); + // check that 0.92 format table is null, as expected by 0.92.0/0.92.1 clients + assertTrue(ZKTableReadOnly.getTableState(zkw, zkw.masterTableZNode92, tableName) == null); + } + + /** + * Test that ZK table writes table state in formats expected by 0.92 and 0.94 clients + */ + @Test + public void test9294Compatibility() throws Exception { + // without useMulti + String tableName = "test9294Compatibility"; + runTest9294CompatibilityTest(tableName, TEST_UTIL.getConfiguration()); + + // with useMulti + tableName = "test9294CompatibilityWithMulti"; + Configuration conf = HBaseConfiguration.create(TEST_UTIL.getConfiguration()); + conf.setBoolean(HConstants.ZOOKEEPER_USEMULTI, true); + runTest9294CompatibilityTest(tableName, conf); + } + + /** + * RecoverableZookeeper that throws a KeeperException after throwExceptionInNumOperations + */ + class ThrowingRecoverableZookeeper extends RecoverableZooKeeper { + private ZooKeeperWatcher zkw; + private int throwExceptionInNumOperations; + + public ThrowingRecoverableZookeeper(ZooKeeperWatcher zkw) throws Exception { + super(ZKConfig.getZKQuorumServersString(TEST_UTIL.getConfiguration()), + HConstants.DEFAULT_ZK_SESSION_TIMEOUT, zkw, 3, 1000); + this.zkw = zkw; + this.throwExceptionInNumOperations = 0; // indicate not to throw an exception + } + + public void setThrowExceptionInNumOperations(int throwExceptionInNumOperations) { + this.throwExceptionInNumOperations = throwExceptionInNumOperations; + } + + private void checkThrowKeeperException() throws KeeperException { + if (throwExceptionInNumOperations == 1) { + throwExceptionInNumOperations = 0; + throw new KeeperException.DataInconsistencyException(); + } + if(throwExceptionInNumOperations > 0) throwExceptionInNumOperations--; + } + + public Stat setData(String path, byte[] data, int version) + throws KeeperException, InterruptedException { + checkThrowKeeperException(); + return zkw.getRecoverableZooKeeper().setData(path, data, version); + } + + public void delete(String path, int version) + throws InterruptedException, KeeperException { + checkThrowKeeperException(); + zkw.getRecoverableZooKeeper().delete(path, version); + } + } + /** + * Because two ZooKeeper nodes are written for each table state transition + * {@link ZooKeeperWatcher#masterTableZNode} and {@link ZooKeeperWatcher#masterTableZNode92} + * it is possible that we fail in between the two operations and are left with + * inconsistent state (when hbase.zookeeper.useMulti is false). + * Check that we can get back to a consistent state by retrying the operation. + */ + @Test + public void testDisableTableRetry() throws Exception { + final String tableName = "testDisableTableRetry"; + + Configuration conf = TEST_UTIL.getConfiguration(); + // test only relevant if useMulti is false + conf.setBoolean(HConstants.ZOOKEEPER_USEMULTI, false); + ZooKeeperWatcher zkw = new ZooKeeperWatcher(conf, + tableName, abortable, true); + ThrowingRecoverableZookeeper throwing = new ThrowingRecoverableZookeeper(zkw); + ZooKeeperWatcher spyZookeeperWatcher = Mockito.spy(zkw); + Mockito.doReturn(throwing).when(spyZookeeperWatcher).getRecoverableZooKeeper(); + + ZKTable zkt = new ZKTable(spyZookeeperWatcher); + zkt.setEnabledTable(tableName); + assertTrue(zkt.isEnabledOrDisablingTable(tableName)); + boolean caughtExpectedException = false; + try { + // throw an exception on the second ZK operation, which means the first will succeed. + throwing.setThrowExceptionInNumOperations(2); + zkt.setDisabledTable(tableName); + } catch (KeeperException ke) { + caughtExpectedException = true; + } + assertTrue(caughtExpectedException); + assertFalse(zkt.isDisabledTable(tableName)); + // try again, ensure table is disabled + zkt.setDisabledTable(tableName); + // ensure disabled from master perspective + assertTrue(zkt.isDisabledTable(tableName)); + // ensure disabled from client perspective + assertTrue(ZKTableReadOnly.isDisabledTable(zkw, tableName)); + } + + /** + * Same as above, but with enableTable + */ + @Test + public void testEnableTableRetry() throws Exception { + final String tableName = "testEnableTableRetry"; + + Configuration conf = TEST_UTIL.getConfiguration(); + // test only relevant if useMulti is false + conf.setBoolean(HConstants.ZOOKEEPER_USEMULTI, false); + ZooKeeperWatcher zkw = new ZooKeeperWatcher(conf, + tableName, abortable, true); + ThrowingRecoverableZookeeper throwing = new ThrowingRecoverableZookeeper(zkw); + ZooKeeperWatcher spyZookeeperWatcher = Mockito.spy(zkw); + Mockito.doReturn(throwing).when(spyZookeeperWatcher).getRecoverableZooKeeper(); + + ZKTable zkt = new ZKTable(spyZookeeperWatcher); + zkt.setDisabledTable(tableName); + assertTrue(zkt.isDisabledTable(tableName)); + boolean caughtExpectedException = false; + try { + // throw an exception on the second ZK operation, which means the first will succeed. + throwing.throwExceptionInNumOperations = 2; + zkt.setEnabledTable(tableName); + } catch (KeeperException ke) { + caughtExpectedException = true; + } + assertTrue(caughtExpectedException); + assertFalse(zkt.isEnabledTable(tableName)); + // try again, ensure table is enabled + zkt.setEnabledTable(tableName); + // ensure enabled from master perspective + assertTrue(zkt.isEnabledTable(tableName)); + // ensure enabled from client perspective + assertTrue(ZKTableReadOnly.isEnabledTable(zkw, tableName)); + } + + /** + * Same as above, but with deleteTable + */ + @Test + public void testDeleteTableRetry() throws Exception { + final String tableName = "testEnableTableRetry"; + + Configuration conf = TEST_UTIL.getConfiguration(); + // test only relevant if useMulti is false + conf.setBoolean(HConstants.ZOOKEEPER_USEMULTI, false); + ZooKeeperWatcher zkw = new ZooKeeperWatcher(conf, + tableName, abortable, true); + ThrowingRecoverableZookeeper throwing = new ThrowingRecoverableZookeeper(zkw); + ZooKeeperWatcher spyZookeeperWatcher = Mockito.spy(zkw); + Mockito.doReturn(throwing).when(spyZookeeperWatcher).getRecoverableZooKeeper(); + + ZKTable zkt = new ZKTable(spyZookeeperWatcher); + zkt.setDisabledTable(tableName); + assertTrue(zkt.isDisabledTable(tableName)); + boolean caughtExpectedException = false; + try { + // throw an exception on the second ZK operation, which means the first will succeed. + throwing.setThrowExceptionInNumOperations(2); + zkt.setDeletedTable(tableName); + } catch (KeeperException ke) { + caughtExpectedException = true; + } + assertTrue(caughtExpectedException); + assertTrue(zkt.isTablePresent(tableName)); + // try again, ensure table is deleted + zkt.setDeletedTable(tableName); + // ensure deleted from master perspective + assertFalse(zkt.isTablePresent(tableName)); + // ensure deleted from client perspective + assertFalse(ZKTableReadOnly.getDisabledTables(zkw).contains(tableName)); } @org.junit.Rule public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); } - diff --git a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java new file mode 100644 index 000000000000..36baf6c71250 --- /dev/null +++ b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java @@ -0,0 +1,123 @@ +/** + * Copyright The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.zookeeper; + + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.*; +import org.apache.hadoop.hbase.zookeeper.ZKTableReadOnly; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +@Category(MediumTests.class) +public class TestZKTableReadOnly { + private static final Log LOG = LogFactory.getLog(TestZooKeeperNodeTracker.class); + private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); + + @BeforeClass + public static void setUpBeforeClass() throws Exception { + TEST_UTIL.startMiniZKCluster(); + } + + @AfterClass + public static void tearDownAfterClass() throws Exception { + TEST_UTIL.shutdownMiniZKCluster(); + } + + Abortable abortable = new Abortable() { + @Override + public void abort(String why, Throwable e) { + LOG.info(why, e); + } + + @Override + public boolean isAborted() { + return false; + } + }; + + private boolean enableAndCheckEnabled(ZooKeeperWatcher zkw, String tableName) throws Exception { + // set the table to enabled, as that is the only state that differs + // between the two formats + ZKTable zkt = new ZKTable(zkw); + zkt.setEnabledTable(tableName); + return ZKTableReadOnly.isEnabledTable(zkw, tableName); + } + + private void runClientCompatiblityWith92ZNodeTest(String tableName, Configuration conf) + throws Exception { + ZooKeeperWatcher zkw = new ZooKeeperWatcher(conf, + tableName, abortable, true); + assertTrue(enableAndCheckEnabled(zkw, tableName)); + } + /** + * Test that client ZK reader can handle the 0.92 table format znode. + */ + @Test + public void testClientCompatibilityWith92ZNode() throws Exception { + // test without useMulti + String tableName = "testClientCompatibilityWith92ZNode"; + // Set the client to read from the 0.92 table znode format + Configuration conf = HBaseConfiguration.create(TEST_UTIL.getConfiguration()); + String znode92 = conf.get("zookeeper.znode.masterTableEnableDisable92", "table92"); + conf.set("zookeeper.znode.clientTableEnableDisable", znode92); + runClientCompatiblityWith92ZNodeTest(tableName, conf); + + // test with useMulti + tableName = "testClientCompatibilityWith92ZNodeUseMulti"; + conf.setBoolean(HConstants.ZOOKEEPER_USEMULTI, true); + runClientCompatiblityWith92ZNodeTest(tableName, conf); + } + + private void runClientCompatibilityWith94ZNodeTest(String tableName, Configuration conf) + throws Exception { + ZooKeeperWatcher zkw = new ZooKeeperWatcher(TEST_UTIL.getConfiguration(), + tableName, abortable, true); + assertTrue(enableAndCheckEnabled(zkw, tableName)); + } + + /** + * Test that client ZK reader can handle the current (0.94) table format znode. + */ + @Test + public void testClientCompatibilityWith94ZNode() throws Exception { + String tableName = "testClientCompatibilityWith94ZNode"; + + // without useMulti + runClientCompatibilityWith94ZNodeTest(tableName, TEST_UTIL.getConfiguration()); + + // with useMulti + tableName = "testClientCompatiblityWith94ZNodeUseMulti"; + Configuration conf = HBaseConfiguration.create(TEST_UTIL.getConfiguration()); + conf.setBoolean(HConstants.ZOOKEEPER_USEMULTI, true); + runClientCompatibilityWith94ZNodeTest(tableName, conf); + } + + @org.junit.Rule + public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = + new org.apache.hadoop.hbase.ResourceCheckerJUnitRule(); +} diff --git a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java index 8ab6011e6135..fba1cafa65f9 100644 --- a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java +++ b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java @@ -86,7 +86,6 @@ public static void setUpBeforeClass() throws Exception { zkw = new ZooKeeperWatcher( new Configuration(TEST_UTIL.getConfiguration()), TestZooKeeper.class.getName(), null); - ZKUtil.waitForZKConnectionIfAuthenticating(zkw); } /** diff --git a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperMainServerArg.java b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperMainServerArg.java index 96073ce46158..cd880526c1eb 100644 --- a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperMainServerArg.java +++ b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperMainServerArg.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file @@ -40,7 +39,8 @@ public class TestZooKeeperMainServerArg { c.set("hbase.zookeeper.quorum", "example.com"); assertEquals("example.com:" + port, parser.parse(c)); c.set("hbase.zookeeper.quorum", "example1.com,example2.com,example3.com"); - assertTrue(port, parser.parse(c).matches("example[1-3]\\.com:" + port)); + assertTrue(port, + parser.parse(c).matches("(example[1-3]\\.com,){2}example[1-3]\\.com:" + port)); } @org.junit.Rule diff --git a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java index 370738930f77..fec4cabc204a 100644 --- a/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java +++ b/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java @@ -1,5 +1,4 @@ /** - * Copyright 2010 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file diff --git a/src/test/resources/META-INF/LICENSE b/src/test/resources/META-INF/LICENSE new file mode 100644 index 000000000000..d64569567334 --- /dev/null +++ b/src/test/resources/META-INF/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/src/test/resources/hbase-site.xml b/src/test/resources/hbase-site.xml index 84b561265927..61dd018c5c3d 100644 --- a/src/test/resources/hbase-site.xml +++ b/src/test/resources/hbase-site.xml @@ -2,7 +2,6 @@