* This is an integration test borrowed from goraci, written by Keith Turner, * which is in turn inspired by the Accumulo test called continous ingest (ci). * The original source code can be found here: - * https://github.com/keith-turner/goraci - * https://github.com/enis/goraci/ - * + *
* Apache Accumulo [0] has a simple test suite that verifies that data is not * lost at scale. This test suite is called continuous ingest. This test runs * many ingest clients that continually create linked lists containing 25 * million nodes. At some point the clients are stopped and a map reduce job is - * run to ensure no linked list has a hole. A hole indicates data was lost.·· - * + * run to ensure no linked list has a hole. A hole indicates data was lost. + *
+ ** The nodes in the linked list are random. This causes each linked list to * spread across the table. Therefore if one part of a table loses data, then it * will be detected by references in another part of the table. - * - * THE ANATOMY OF THE TEST + *
+ *+ *
+ *
* The key is that nodes only reference flushed nodes. Therefore a node should * never reference a missing node, even if the ingest client is killed at any * point in time. - * + *
+ ** When running this test suite w/ Accumulo there is a script running in * parallel called the Aggitator that randomly and continuously kills server - * processes.·· The outcome was that many data loss bugs were found in Accumulo - * by doing this.· This test suite can also help find bugs that impact uptime - * and stability when· run for days or weeks.·· - * - * This test suite consists the following· - a few Java programs· - a little - * helper script to run the java programs - a maven script to build it.·· - * + * processes. The outcome was that many data loss bugs were found in Accumulo + * by doing this. This test suite can also help find bugs that impact uptime + * and stability when run for days or weeks. + *
+ *+ * This test suite consists the following + *
* When generating data, its best to have each map task generate a multiple of * 25 million. The reason for this is that circular linked list are generated * every 25M. Not generating a multiple in 25M will result in some nodes in the * linked list not having references. The loss of an unreferenced node can not * be detected. - * - * - * Below is a description of the Java programs - * - * Generator - A map only job that generates data. As stated previously,·its best to generate data - * in multiples of 25M. An option is also available to allow concurrent walkers to select and walk - * random flushed loops during this phase. - * - * Verify - A map reduce job that looks for holes. Look at the counts after running. REFERENCED and - * UNREFERENCED are· ok, any UNDEFINED counts are bad. Do not run at the· same - * time as the Generator. - * - * Walker - A standalone program that start following a linked list· and emits timing info.·· - * - * Print - A standalone program that prints nodes in the linked list - * - * Delete - A standalone program that deletes a single node + *
+ *+ *
* ex: + *
* ./hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList * loop 2 1 100000 /temp 1 1000 50 1 0 - * + *+ * */ @Category(IntegrationTests.class) public class IntegrationTestBigLinkedList extends IntegrationTestBase { @@ -1865,7 +1895,7 @@ private void printCommands() { System.err.println(" walker " + "Standalone program that starts following a linked list & emits timing info."); System.err.println(" print Standalone program that prints nodes in the linked list."); - System.err.println(" delete Standalone program that deletes a·single node."); + System.err.println(" delete Standalone program that deletes a single node."); System.err.println(" loop Program to Loop through Generator and Verify steps"); System.err.println(" clean Program to clean all left over detritus."); System.err.println(" search Search for missing keys."); diff --git a/hbase-metrics-api/pom.xml b/hbase-metrics-api/pom.xml index 6fa44fec8701..86d53f9193ac 100644 --- a/hbase-metrics-api/pom.xml +++ b/hbase-metrics-api/pom.xml @@ -23,7 +23,7 @@
+ * http://www.apache.org/licenses/LICENSE-2.0 + *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.hbase.master.balancer; + +import java.io.BufferedReader; +import java.io.FileReader; +import java.io.IOException; +import java.io.InputStreamReader; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.regex.Pattern; +import java.util.regex.PatternSyntaxException; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.ServerName; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * This is an optional Cost function designed to allow region count skew across RegionServers. A + * rule file is loaded from the local FS or HDFS before balancing. It contains lines of rules. A + * rule is composed of a regexp for hostname, and a limit. For example, we could have: + *
+ * * rs[0-9] 200 * rs1[0-9] 50 + *
+ * RegionServers with hostname matching the first rules will have a limit of 200, and the others 50. + * If there's no match, a default is set. The costFunction is trying to fill all RegionServers + * linearly, meaning that if the global usage is at 50%, then all RegionServers should hold half of + * their capacity in terms of regions. In order to use this CostFunction, you need to set the + * following options: + *| Region Server | Start KeyEnd KeyLocality | Requests | ReplicaID |
|---|
| Region Server | Start KeyEnd KeyLocality | Requests |
|---|
| Name | +Region Server | +ReadRequests | +WriteRequests | +StorefileSize | +Num.Storefiles | +MemSize | +Locality | +Start Key | +End Key | + <% + if (withReplica) { + %> +ReplicaID | + <% + } + %> +
|---|---|---|---|---|---|---|---|---|---|---|
| <%= escapeXml(meta.getRegionNameAsString()) %> | -<%= metaLocationString %> | -<%= readReq%> | -<%= writeReq%> | -<%= fileSize%> | -<%= fileCount%> | -<%= memSize%> | -<%= locality%> | -<%= escapeXml(Bytes.toString(meta.getStartKey())) %> | -<%= escapeXml(Bytes.toString(meta.getEndKey())) %> | -<% - if (withReplica) { -%> -<%= meta.getReplicaId() %> | -<% - } -%> -
| Name | +Region Server | +Num. Compacting KVs | +Num. Compacted KVs | +Remaining KVs | +Compaction Progress | +
|---|---|---|---|---|---|
| <%= escapeXml(meta.getRegionNameAsString()) %> | +<%= metaLocationString %> | +<%= String.format("%,1d", compactingKVs)%> | +<%= String.format("%,1d", compactedKVs)%> | +<%= String.format("%,1d", compactingKVs - compactedKVs)%> | +<%= compactionProgress%> | +
| Name(<%= String.format("%,1d", regions.size())%>) | -Region Server | -ReadRequests (<%= String.format("%,1d", totalReadReq)%>) |
-WriteRequests (<%= String.format("%,1d", totalWriteReq)%>) |
-StorefileSize (<%= StringUtils.byteDesc(totalSize*1024l*1024)%>) |
-Num.Storefiles (<%= String.format("%,1d", totalStoreFileCount)%>) |
-MemSize (<%= StringUtils.byteDesc(totalMemSize*1024l*1024)%>) |
-Locality | -Start Key | -End Key | -<% - if (withReplica) { -%> -ReplicaID | -<% - } -%> -
|---|
| Name(<%= String.format("%,1d", regions.size())%>) | +Region Server | +ReadRequests (<%= String.format("%,1d", totalReadReq)%>) |
+ WriteRequests (<%= String.format("%,1d", totalWriteReq)%>) |
+ StorefileSize (<%= StringUtils.byteDesc(totalSize*1024l*1024)%>) |
+ Num.Storefiles (<%= String.format("%,1d", totalStoreFileCount)%>) |
+ MemSize (<%= StringUtils.byteDesc(totalMemSize*1024l*1024)%>) |
+ Locality | +Start Key | +End Key | + <% + if (withReplica) { + %> +ReplicaID | + <% + } + %> +|
|---|---|---|---|---|---|---|---|---|---|---|---|
| <%= escapeXml(Bytes.toStringBinary(regionInfo.getRegionName())) %> | + <% + if (urlRegionServer != null) { + %> ++ <%= StringEscapeUtils.escapeHtml(addr.getHostname().toString()) + ":" + master.getRegionServerInfoPort(addr) %> + | + <% + } else { + %> +not deployed | + <% + } + %> +<%= readReq%> | +<%= writeReq%> | +<%= regionSize%> | +<%= fileCount%> | +<%= memSize%> | +<%= locality%> | +<%= escapeXml(Bytes.toStringBinary(regionInfo.getStartKey()))%> | +<%= escapeXml(Bytes.toStringBinary(regionInfo.getEndKey()))%> | + <% + if (withReplica) { + %> +<%= regionInfo.getReplicaId() %> | + <% + } + %> +
This table has <%= numRegions %> regions in total, in order to improve the page load time, + only <%= numRegionsRendered %> regions are displayed here, click + here to see all regions.
+ <% } %> +| Name(<%= String.format("%,1d", regions.size())%>) | +Region Server | +Num. Compacting KVs (<%= String.format("%,1d", totalCompactingKVs)%>) |
+ Num. Compacted KVs (<%= String.format("%,1d", totalCompactedKVs)%>) |
+ Remaining KVs (<%= String.format("%,1d", totalCompactingKVs - totalCompactedKVs)%>) |
+ Compaction Progress (<%= percentDone %>) |
+ ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| <%= escapeXml(Bytes.toStringBinary(regionInfo.getRegionName())) %> | - <% - if (urlRegionServer != null) { - %> -- <%= StringEscapeUtils.escapeHtml(addr.getHostname().toString()) + ":" + master.getRegionServerInfoPort(addr) %> - | - <% - } else { - %> -not deployed | - <% - } - %> -<%= readReq%> | -<%= writeReq%> | -<%= regionSize%> | -<%= fileCount%> | -<%= memSize%> | -<%= locality%> | -<%= escapeXml(Bytes.toStringBinary(regionInfo.getStartKey()))%> | -<%= escapeXml(Bytes.toStringBinary(regionInfo.getEndKey()))%> | - <% - if (withReplica) { - %> -<%= regionInfo.getReplicaId() %> | - <% - } - %> -
This table has <%= numRegions %> regions in total, in order to improve the page load time, - only <%= numRegionsRendered %> regions are displayed here, click - here to see all regions.
-<% } %> + if (addr != null) { + // This port might be wrong if RS actually ended up using something else. + urlRegionServer = + "//" + URLEncoder.encode(addr.getHostname()) + ":" + master.getRegionServerInfoPort(addr) + "/"; + } + if (numRegionsRendered < numRegionsToRender) { + numRegionsRendered++; + %> +This table has <%= numRegions %> regions in total, in order to improve the page load time, + only <%= numRegionsRendered %> regions are displayed here, click + here to see all regions.
+ <% } %> + + +