-
Notifications
You must be signed in to change notification settings - Fork 525
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add compact api for rocksdb/cassandra/hbase backend #1378
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1378 +/- ##
============================================
- Coverage 63.52% 60.77% -2.75%
+ Complexity 5971 677 -5294
============================================
Files 402 423 +21
Lines 32736 35048 +2312
Branches 4559 4967 +408
============================================
+ Hits 20794 21302 +508
- Misses 9819 11599 +1780
- Partials 2123 2147 +24
Continue to review full report at Codecov.
|
fix codecov missing some files coverage by jacococli: expect: java -jar $TRAVIS_DIR/jacococli.jar report hugegraph-test/target/jacoco-it.exec \
--classfiles hugegraph-api/target/classes/com/baidu/hugegraph --xml report.xml
[INFO] Loading execution data file /home/travis/build/hugegraph/hugegraph/hugegraph-test/target/jacoco-it.exec.
[INFO] Analyzing 168 classes.
$TRAVIS_DIR/stop-server.sh
VERSION=`mvn help:evaluate -Dexpression=project.version -q -DforceStdout`
mvn help:evaluate -Dexpression=project.version -q -DforceStdout
BASE_DIR=hugegraph-$VERSION
BIN=$BASE_DIR/bin
$BIN/stop-hugegraph.sh
no crontab for travis
The HugeGraphServer monitor has been closed
Killing HugeGraphServer(pid 6826)...OK
The command "$TRAVIS_DIR/run-api-test.sh" exited with 0. error: java -jar $TRAVIS_DIR/jacococli.jar report hugegraph-test/target/jacoco-it.exec \
--classfiles hugegraph-dist/target/classes/com/baidu/hugegraph \
--classfiles hugegraph-api/target/classes/com/baidu/hugegraph \
--classfiles hugegraph-core/target/classes/com/baidu/hugegraph \
--classfiles hugegraph-*/target/classes/com/baidu/hugegraph \
--xml report.xml
[INFO] Loading execution data file /home/travis/build/hugegraph/hugegraph/hugegraph-test/target/jacoco-it.exec.
[INFO] Loading execution data file /home/travis/build/hugegraph/hugegraph/hugegraph-cassandra/target/classes/com/baidu/hugegraph.
Exception in thread "main" java.io.FileNotFoundException: hugegraph-cassandra/target/classes/com/baidu/hugegraph (Is a directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.jacoco.cli.internal.core.tools.ExecFileLoader.load(ExecFileLoader.java:71)
at org.jacoco.cli.internal.commands.Report.loadExecutionData(Report.java:98)
at org.jacoco.cli.internal.commands.Report.execute(Report.java:82)
at org.jacoco.cli.internal.Main.execute(Main.java:89)
at org.jacoco.cli.internal.Main.main(Main.java:104)
The command "$TRAVIS_DIR/run-api-test.sh" exited with 1. error java -jar $TRAVIS_DIR/jacococli.jar report hugegraph-test/target/jacoco-it.exec \
--classfiles hugegraph-*/target/classes/com/baidu/hugegraph \
--xml report.xml ok: java -jar $TRAVIS_DIR/jacococli.jar report hugegraph-test/target/jacoco-it.exec \
--classfiles hugegraph-dist/target/classes/com/baidu/hugegraph \
--classfiles hugegraph-api/target/classes/com/baidu/hugegraph \
--classfiles hugegraph-core/target/classes/com/baidu/hugegraph \
--classfiles hugegraph-$BACKEND/target/classes/com/baidu/hugegraph \
--xml report.xml |
39d2a13
to
5af4600
Compare
rocksdb metrics response demo{
"hugegraph": {
"backend": "rocksdb",
"nodes": 1,
"cluster_id": "local",
"servers": {
"local": {
"mem_unit": "MB",
"disk_unit": "GB",
"mem_used": 0.02,
"mem_used_readable": "21.36 KB",
"disk_usage": 0.0,
"disk_usage_readable": "4.24 KB",
"block_cache_usage": 4.138946533203125E-4,
"block_cache_pinned_usage": 0.0,
"block_cache_capacity": 224.0,
"estimate_table_readers_mem": 0.00164794921875,
"size_all_mem_tables": 0.018798828125,
"cur_size_all_mem_tables": 0.018798828125,
"estimate_live_data_size": 8.087605237960815E-6,
"total_sst_files_size": 8.087605237960815E-6,
"live_sst_files_size": 8.087605237960815E-6,
"estimate_pending_compaction_bytes": 0.0,
"estimate_num_keys": 0,
"num_entries_active_mem_table": 0,
"num_entries_imm_mem_tables": 0,
"num_deletes_active_mem_table": 0,
"num_deletes_imm_mem_tables": 0,
"num_running_flushes": 0,
"mem_table_flush_pending": 0,
"num_running_compactions": 0,
"compaction_pending": 0,
"num_immutable_mem_table": 0,
"num_snapshots": 0,
"oldest_snapshot_time": 0,
"num_live_versions": 28,
"current_super_version_number": 28
}
}
},
"hugegraph1": {
"backend": "rocksdb",
"nodes": 1,
"cluster_id": "local",
"servers": {
"local": {
"mem_unit": "MB",
"disk_unit": "GB",
"mem_used": 0.02,
"mem_used_readable": "21.28 KB",
"disk_usage": 0.01,
"disk_usage_readable": "6.16 KB",
"block_cache_usage": 0.0,
"block_cache_pinned_usage": 0.0,
"block_cache_capacity": 224.0,
"estimate_table_readers_mem": 0.001979827880859375,
"size_all_mem_tables": 0.018798828125,
"cur_size_all_mem_tables": 0.018798828125,
"estimate_live_data_size": 1.1742115020751953E-5,
"total_sst_files_size": 1.1742115020751953E-5,
"live_sst_files_size": 1.1742115020751953E-5,
"estimate_pending_compaction_bytes": 0.0,
"estimate_num_keys": 76,
"num_entries_active_mem_table": 0,
"num_entries_imm_mem_tables": 0,
"num_deletes_active_mem_table": 0,
"num_deletes_imm_mem_tables": 0,
"num_running_flushes": 0,
"mem_table_flush_pending": 0,
"num_running_compactions": 0,
"compaction_pending": 0,
"num_immutable_mem_table": 0,
"num_snapshots": 0,
"oldest_snapshot_time": 0,
"num_live_versions": 28,
"current_super_version_number": 28
}
}
}
} cassandra metrics response demo{
"hugegraph": {
"backend": "cassandra",
"cluster_id": "cluster3",
"nodes": 1,
"servers": {
"127.0.0.1": {
"mem_max": 2008.0,
"mem_commited": 2008.0,
"mem_used": 283.49,
"mem_used_readable": "283.49 MB",
"mem_unit": "MB",
"disk_usage": 0.0,
"disk_usage_readable": "336.33 KB",
"disk_usage_details": {
"127.0.0.1": "336.33 KiB"
},
"disk_unit": "GB",
"uptime": 28660002,
"uptime_readable": "7h57m40s",
"time_unit": "ms",
"estimated_partition_count": 46,
"dropped_mutations": 0,
"pending_flushes": 0,
"key_cache_hit_rate": 0.0,
"bloom_filter_false_ratio": 0.0,
"write_latency_hugegraph": {
"count": 4759,
"min": 1.598,
"mean": 38.33763092783505,
"max": 2346.799,
"stddev": 69.59220960583126,
"p50": 20.501,
"p75": 51.012,
"p95": 105.778,
"p98": 152.321,
"p99": 182.785,
"p999": 545.791,
"duration_unit": "microseconds",
"mean_rate": 23.72839801212326,
"m15_rate": 4.315772554871052,
"m5_rate": 8.642530967558088,
"m1_rate": 4.352671868667263,
"rate_unit": "events/second"
},
"read_latency_hugegraph": {
"count": 1824,
"min": 6.867,
"mean": 66.95636651234568,
"max": 654.9490000000001,
"stddev": 42.035327213360866,
"p50": 73.45700000000001,
"p75": 88.148,
"p95": 105.778,
"p98": 152.321,
"p99": 219.342,
"p999": 454.826,
"duration_unit": "microseconds",
"mean_rate": 9.093885890514578,
"m15_rate": 1.671099039695186,
"m5_rate": 3.4302851445173115,
"m1_rate": 2.434428704287022,
"rate_unit": "events/second"
},
"write_latency_*": {
"count": 53110,
"min": 1.332,
"mean": 59.94491580531794,
"max": 52066.354,
"stddev": 565.3993395762526,
"p50": 24.601,
"p75": 42.51,
"p95": 182.785,
"p98": 263.21,
"p99": 379.022,
"p999": 2816.159,
"duration_unit": "microseconds",
"mean_rate": 1.8540166185077438,
"m15_rate": 11.445035834804784,
"m5_rate": 20.077709526278344,
"m1_rate": 7.122562045494221,
"rate_unit": "events/second"
},
"read_latency_*": {
"count": 50100,
"min": 5.723,
"mean": 76.88411553189447,
"max": 7007.506,
"stddev": 134.90827030363585,
"p50": 51.012,
"p75": 88.148,
"p95": 219.342,
"p98": 454.826,
"p99": 545.791,
"p999": 1629.722,
"duration_unit": "microseconds",
"mean_rate": 1.7489400792173229,
"m15_rate": 17.105168541542277,
"m5_rate": 29.483216207638772,
"m1_rate": 7.962272561892672,
"rate_unit": "events/second"
},
"key_cache_size": 10024,
"key_cache_entries": 109,
"row_cache_size": 0,
"row_cache_entries": 0,
"counter_cache_size": 664,
"counter_cache_entries": 6,
"compaction_completed_tasks": 12470,
"compaction_pending_tasks": 0,
"compaction_bytes_compacted": 744450763,
"live_nodes": ["127.0.0.1"],
"joining_nodes": [],
"moving_nodes": [],
"leaving_nodes": [],
"unreachable_nodes": [],
"keyspaces": ["system_distributed", "system", "hugegraph1", "system_schema", "hugegraph", "system_auth", "system_traces"],
"num_tables": 90,
"exception_count": 0
}
}
}
} scylladb metrics response demo{
"mem_max": 247.5,
"mem_commited": 21.31,
"mem_used": 13.41,
"mem_used_readable": "13.41 MB",
"disk_usage": 0.0,
"disk_usage_readable": "2.05 MB",
"disk_usage_details": {
"127.0.0.1": "2.11 MB"
},
"disk_unit": "GB",
"uptime": 859397,
"uptime_readable": "14 m19s",
"time_unit": "ms",
"estimated_partition_count": 0,
"dropped_mutations": 0,
"pending_flushes": 0,
"bloom_filter_false_ratio": 0.0,
"key_cache_size": 0,
"key_cache_entries": 0,
"row_cache_size": 2651,
"row_cache_entries": 2651,
"counter_cache_size": 0,
"counter_cache_entries": 0,
"compaction_completed_tasks": 3399,
"compaction_pending_tasks": 0,
"live_nodes": ["127.0.0.1"],
"joining_nodes": [],
"moving_nodes": [],
"leaving_nodes": [],
"unreachable_nodes": [],
"keyspaces": ["system", "system_auth", "hugegraph", "system_schema", "system_distributed", "system_traces"],
"num_tables": 82,
"exception_count": 0
} hbase metrics response demo{
"hugegraph": {
"backend": "hbase",
"cluster_id": "0d0009fa-f8a6-4a96-a7db-f8daa1403a38",
"master_name": "192.168.31.166:16000",
"average_load": 34.0,
"hbase_version": "2.0.2",
"region_count": 34,
"leaving_servers": [],
"nodes": 1,
"region_servers": ["192.168.31.166:16020"],
"servers": {
"192.168.31.166:16020": {
"mem_max": 2014.0,
"mem_used": 91.0,
"mem_used_readable": "91.0 MB",
"mem_unit": "MB",
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"request_count": 34051,
"request_count_per_second": 7,
"regions": {
"hbase:meta,,1": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 9.765625E-4,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 2,
"store_file_count": 2,
"request_count": 19391,
"write_request_count": 1314,
"read_request_count": 18077,
"filtered_read_request_count": 1962,
"completed_sequence_id": 1330,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hbase:namespace,,1615017265140.8bec14dd3c761e19d5f6dc5083a8b782.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 3904,
"write_request_count": 5,
"read_request_count": 3899,
"filtered_read_request_count": 8,
"completed_sequence_id": 17,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:c,,1615020727652.542ada66f9632317f0e6d7f963373977.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 231,
"write_request_count": 60,
"read_request_count": 171,
"filtered_read_request_count": 0,
"completed_sequence_id": 64,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:el,,1615020724738.d77de2e14e1553fdf16ee78e6770f557.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 5,
"write_request_count": 3,
"read_request_count": 2,
"filtered_read_request_count": 0,
"completed_sequence_id": 7,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_ai,,1615020713987.ad9d91e9ef40b7967c57f1752453683b.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_di,,1615020716155.b3bc7ff1e9d18b9de409eb9f4d126851.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_ei,,1615020719056.9908f57970344ad3f3c2b9faffa85c9a.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 16,
"write_request_count": 16,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_fi,,1615020721731.fcd5ba5cbbe6bec0b5bbe74a4bf0c084.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_hi,,1615020722451.4708b789814b884710e36ffcdd640019.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_ie,,1615020715436.a758d04cec173fec3be9b11493dedbb7.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 16,
"write_request_count": 16,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_ii,,1615020719780.c4e5293948f792f2404dd1e41d35e5bb.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_li,,1615020718334.290865b8b88d425f62259f19eecc86a9.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_oe,,1615020717601.977cd3f759c13c200f9d8312b340cfb9.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 32,
"write_request_count": 16,
"read_request_count": 16,
"filtered_read_request_count": 64,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_si,,1615020723186.401d794c8f614ec79038570fc5fa98f2.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_ui,,1615020721011.ba3b8f7271b10b12052de9c384881469.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_v,,1615020716875.3d8e867ab54d9e39630c2d4d2359830e.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 60,
"write_request_count": 6,
"read_request_count": 54,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:g_vi,,1615020714711.5719024c84ebb226459d24889cd5cb77.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 6,
"write_request_count": 6,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:il,,1615020725466.86e59caff4d6882f971edec19ebdd43e.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 4,
"write_request_count": 1,
"read_request_count": 3,
"filtered_read_request_count": 0,
"completed_sequence_id": 5,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:m_si,,1615020726914.103eb3c5b71cd260707fd7235eaa21b3.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 74,
"write_request_count": 60,
"read_request_count": 14,
"filtered_read_request_count": 29,
"completed_sequence_id": 64,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:pk,,1615020726186.ac30f8a07a9d0c3db034debf4e27e99c.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 110,
"write_request_count": 49,
"read_request_count": 61,
"filtered_read_request_count": 0,
"completed_sequence_id": 53,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_ai,,1615020704439.373ad974288b4d1164af456605a9bac8.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_di,,1615020706607.2a3e00cce8c1ba9cde335bad21ca7d8b.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_ei,,1615020709511.0dca37f4b1e6656d1dd2835c33071378.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_fi,,1615020711714.503eb104ef299b020f1c0c2f2b656cac.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_hi,,1615020712433.35febf3aa9deb4368d5c581f5d0b3992.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_ie,,1615020705883.93e9462ea9db6b3ac2bc0c524c081e1b.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_ii,,1615020710257.d193113f7009aafeac74495ecec3537d.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_li,,1615020708779.f793628c32fd8a8645f0c439de59e2f9.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_oe,,1615020708059.d9fb614cbedcac72847e5b1f574064c0.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_si,,1615020713163.161bc267c57877cd280ce1bcc3d67a04.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_ui,,1615020710988.a1a19c8319168c2e42b18ad06366c0b2.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 0,
"write_request_count": 0,
"read_request_count": 0,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_v,,1615020707336.5cf1eae2010260a193a80a850406c55b.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 2029,
"write_request_count": 677,
"read_request_count": 1352,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:s_vi,,1615020705161.6cde238e400efd7ac67e3d583d30e6e0.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 1353,
"write_request_count": 677,
"read_request_count": 676,
"filtered_read_request_count": 0,
"completed_sequence_id": -1,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
},
"hugegraph:vl,,1615020724020.60d99aab92a266c3adf6252b354348e9.": {
"disk_usage": 0.0,
"disk_usage_readable": "0 B",
"disk_unit": "GB",
"index_store_size": 0.0,
"mem_store_size": 0.0,
"bloom_filter_size": 0.0,
"size_unit": "MB",
"store_count": 1,
"store_file_count": 1,
"request_count": 18,
"write_request_count": 7,
"read_request_count": 11,
"filtered_read_request_count": 0,
"completed_sequence_id": 11,
"data_locality": 0.0,
"compacted_cell_count": 0,
"compacting_cell_count": 0
}
}
}
}
}
} |
644a816
to
9d14c36
Compare
@@ -119,6 +120,7 @@ | |||
|
|||
/* rocksdb */ | |||
RocksDBSessionsTest.class, | |||
RocksDBSessionTest.class, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also add UnitUtilTest
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed it and to import from common module
try { | ||
// Waits while compaction is performed on the background threads | ||
// rocksdb().flush(new FlushOptions()) | ||
rocksdb().compactRange(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
compactRange will wait compaction finished?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, it may cost a long time, we can start async task for it through gremlin-job.
Scylla CI error: RROR] Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 52.566 s <<< FAILURE! - in com.baidu.hugegraph.api.ApiTestSuite
[ERROR] testMetricsBackend(com.baidu.hugegraph.api.MetricsApiTest) Time elapsed: 0.698 s <<< FAILURE!
java.lang.AssertionError: Expect contains key 'key_cache_hit_rate' in {mem_max=247.5, mem_commited=21.31, mem_used=13.59, mem_used_readable=13.59 MB, mem_unit=MB, disk_usage=0.0, disk_usage_readable=2.01 MB, disk_usage_details={127.0.0.1=2.04 MB}, disk_unit=GB, uptime=877096, uptime_readable=14m37s, time_unit=ms, estimated_partition_count=0, dropped_mutations=0, pending_flushes=0, exception=java.lang.IllegalStateException: Scylla API server HTTP GET to URL '/column_family/metrics/key_cache_hit_rate/hugegraph:g_v' failed: API call is not supported yet}
at com.baidu.hugegraph.api.MetricsApiTest.testMetricsBackend(MetricsApiTest.java:130)
java.lang.IllegalStateException: Scylla API server HTTP GET to URL '/column_family/metrics/key_cache_hit_rate/hugegraph:g_v' failed: API call is not supported yet
at com.scylladb.jmx.api.APIClient.getException(APIClient.java:140)
at com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:187)
at com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:209)
at com.scylladb.jmx.api.APIClient.getDoubleValue(APIClient.java:263)
at com.scylladb.jmx.api.APIClient.lambda$getReader$2(APIClient.java:251)
at org.apache.cassandra.metrics.TableMetrics$Registry.lambda$newGauge$0(TableMetrics.java:106)
at org.apache.cassandra.metrics.MetricsRegistry.lambda$gauge$1(MetricsRegistry.java:81)
at org.apache.cassandra.metrics.MetricsRegistry$JmxGauge.getValue(MetricsRegistry.java:93) |
7ff7270
to
6738f59
Compare
f8927c0
to
fb9e54c
Compare
metrics.put(name(key), (long) this.sum(this.session, key)); | ||
} | ||
|
||
private String name(String key) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
static
private Object compactHost(String host) { | ||
try (NodeProbe probe = this.newNodeProbe(host)) { | ||
Compact compact = new Compact(); | ||
compact.execute(probe); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
compact all keyspaces?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no, this api is for keyspace scope, need to fix
metrics.put(name, number); | ||
} | ||
|
||
protected static void appendTimerMetrics(Map<String, Object> metrics, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
return Collections.unmodifiableList(this.tables); | ||
} | ||
|
||
protected Map<String, Object> getMetricsByHost(String host) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make private
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would be called by scylladb subclass
|
||
@Test | ||
public void testProperty() { | ||
int count = 1234; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use random to test multiple times
} | ||
|
||
@Test | ||
public void testCompact() throws RocksDBException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems no obvious signal for compacted
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
compactRange() is a synchronous blocked call. there is no way to verify whether the compaction is done, just can be tested in terms of performance.
this.commit(); | ||
|
||
// NOTE: maybe return "James" here | ||
Assert.assertEquals(null, get("person:1gname")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Assert.assertTrue(null == get("person:1gname") || "James".equals(get("person:1gname")));
Assert.assertEquals(null, value); | ||
|
||
this.rocks.session().put(TABLE, b("person:1gname"), b("James")); | ||
this.rocks.session().put(TABLE, b("person:1gage"), b("19")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
prefer this.rocks.session().put(TABLE, b("person:1gage"), b(19));
@@ -34,9 +38,12 @@ | |||
public String MEM_UNIT = "mem_unit"; | |||
|
|||
// Data load related metrics | |||
public String DATA_SIZE = "data_size"; | |||
public String DISK_USAGE = "disk_usage"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
change MEM_COMMITED
to MEM_COMMITTED
in line 36
Change-Id: I26660b28b64089f3b636020e721659db3466267b
Change-Id: I626f0f1df44d31d5e7fe4ee212b20c7f3a48e2d4
Change-Id: I1a5a09bf1372287d534f890b94a89bb8f604ee58
Change-Id: I9c05b101d83e9afb57c7f15c12e930063d91a7e4
Change-Id: I01f92ffdebb52d226bb84144eecca32d57dcf7c6
Change-Id: If5f6184117fcdeedeb4879f6cb90fe4c7c447cd7
Change-Id: Ica4ea713a9b74255c1bd4e5c0adbe233faeaceb2
also delete UnitUtilTest Change-Id: I0cefb5899ca16bc23157761f627e54fc7c958aa1
Change-Id: Ife76952188669ec37c32558b8b564837eebb7049
Change-Id: I67294a65d660670dc001f7d7176724693645dff8
Change-Id: I6469dd5fee87316cb681fb7818af7248d7648e27
Change-Id: Ia162ab02b217ba53130916bdd28b9dfed8bf425f
Change-Id: Iec1304acaac89beaa893b1fb56aa671a6daa695e
Change-Id: I321121bd43ef4895a4a9fe815d818d62eb74c925
Change-Id: I3f0617773fd6d2d4a8ea949f5c3c1f12860edeb6
Change-Id: Idc50b3f76200affb21dd3e2cb6bf5b7c0a48335d
* add compact api for rocksdb/cassandra/hbase backend * increase and standardize some metrics * increase metrics for cassandra/hbase and unify display format * fix codecov missing some files coverage by jacococli * add ScyllaDBMetrics * make cassandra compact in the scope of keyspace Change-Id: I26660b28b64089f3b636020e721659db3466267b
improve #1373
Change-Id: I26660b28b64089f3b636020e721659db3466267b