-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Log size partition cost #569
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wycccccc 概念很不錯,不過程式碼風格要調整一下,請參考我給的建議去調整,謝謝
.collect( | ||
Collectors.toMap( | ||
Map.Entry::getKey, | ||
topicPartitionCollectionEntry -> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
這一段也是重複的程式碼:
Function<Map.Entry<TopicPartition, Collection<HasBeanObject>>, Double> toDouble =
e ->
LogMetrics.Log.meters(e.getValue(), LogMetrics.Log.SIZE).stream()
.mapToDouble(HasValue::value)
.findAny()
.orElse(0.0D);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
太不敏感了,已修正。
return new PartitionCost() { | ||
@Override | ||
public Map<TopicPartition, Double> value(String topic) { | ||
return clusterBean.mapByPartition().entrySet().stream() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
這邊用mapByReplica
比較適合,因為mapByReplica
才會針對所有的replica去撈資料,mapByPartition
一個partition只會撈一筆而已,可能不會考慮到replica還沒sync完成的狀態
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
因爲mapByReplica
的存在,我一開始認爲mapByPartition
會回傳的是leader partition。但看了一下實做發現並不是。已將partitionCost
修改爲該方法應該有的樣子。
另外我感覺mapByPartition
的設計似乎缺少明確的意義,它似乎只是一個劣化版的mapByReplica
,不如把它修改成回傳leader partition 或者直接移除。
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
另外我感覺mapByPartition的設計似乎缺少明確的意義,它似乎只是一個劣化版的mapByReplica,不如把它修改成回傳leader partition 或者直接移除。
同意,mapByPartition
似乎可以被mapByReplica
所取代
@qoo332001 你覺得呢?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wycccccc 感謝更新,一樣有幾個建議請看一下
@Override | ||
public Map<TopicPartition, Double> value(String topic) { | ||
var leaderPartitions = | ||
clusterInfo.availableReplicas(topic).stream() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
var replicas =
clusterInfo.availableReplicaLeaders(topic).stream()
.map(r -> TopicPartitionReplica.of(r.topic(), r.partition(), r.nodeInfo().id()))
.collect(Collectors.toUnmodifiableSet());
return clusterBean.mapByReplica().entrySet().stream()
.filter(
topicPartitionCollectionEntry ->
replicas.contains(topicPartitionCollectionEntry.getKey()))
.collect(
Collectors.toMap(
topicPartitionCollectionEntry ->
TopicPartition.of(
topicPartitionCollectionEntry.getKey().topic(),
topicPartitionCollectionEntry.getKey().partition()),
topicPartitionCollectionEntry ->
toDouble.apply(topicPartitionCollectionEntry)));
這樣如何?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
這樣確實更簡潔了,已修改成此邏輯。
}; | ||
} | ||
|
||
Function<Map.Entry<TopicPartitionReplica, Collection<HasBeanObject>>, Double> toDouble = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
麻煩加上private final
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
TopicPartition.of( | ||
topicPartitionCollectionEntry.getKey().topic(), | ||
topicPartitionCollectionEntry.getKey().partition()), | ||
topicPartitionCollectionEntry -> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
toDouble
加上 private final 後就可以簡化成toDouble
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
topicPartitionCollectionEntry.getKey().topic(), | ||
topicPartitionCollectionEntry.getKey().partition()), | ||
topicPartitionCollectionEntry -> | ||
toDouble.apply(topicPartitionCollectionEntry))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
toDouble
加上 private final 後就可以簡化成toDouble
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
commit 2bd46a0f86fe3ad9a0fabebb35e6d24e1c5e2993 Author: Zheng-Xian Li <[email protected]> Date: Fri Aug 19 18:35:39 2022 +0800 AAAAAAAAAAA commit 7a2c6f4c2e0ddb0d77909bcf4645eaf1792a9df2 Author: Zheng-Xian Li <[email protected]> Date: Fri Aug 19 17:35:32 2022 +0800 yes commit 5bea879257575c339b7f079373854e1902c44cb4 Author: Zheng-Xian Li <[email protected]> Date: Fri Aug 19 17:04:56 2022 +0800 qqq commit 6f396c3101112b0321a7709495e1a4bb274e581c Merge: e5a7f516 23d95f0 Author: Zheng-Xian Li <[email protected]> Date: Fri Aug 19 16:53:13 2022 +0800 Merge remote-tracking branch 'qoo332001/migrateCost' into Balancer # Conflicts: # app/src/main/java/org/astraea/app/cost/HasMoveCost.java # app/src/main/java/org/astraea/app/cost/MoveCost.java # app/src/test/java/org/astraea/app/cost/MoveCostTest.java commit e5a7f51678b50a4b77d9d6b8e1d281b2eadf47ae Author: Zheng-Xian Li <[email protected]> Date: Fri Aug 19 16:52:15 2022 +0800 fix merge commit 21fc423a05e151c825d761b5bc2c6554f1cbd3b1 Merge: 6c681fef 4e9544d Author: Zheng-Xian Li <[email protected]> Date: Fri Aug 19 16:27:24 2022 +0800 Merge branch 'main' into Balancer # Conflicts: # app/src/main/java/org/astraea/app/App.java # app/src/main/java/org/astraea/app/admin/ClusterBean.java # app/src/main/java/org/astraea/app/balancer/BalancerUtils.java # app/src/main/java/org/astraea/app/cost/ClusterCost.java # app/src/main/java/org/astraea/app/cost/CostUtils.java # app/src/main/java/org/astraea/app/cost/HasClusterCost.java # app/src/main/java/org/astraea/app/cost/LoadCost.java # app/src/main/java/org/astraea/app/cost/ReplicaDiskInCost.java # app/src/main/java/org/astraea/app/cost/ReplicaLeaderCost.java # app/src/main/java/org/astraea/app/metrics/jmx/MBeanClientImpl.java # app/src/test/java/org/astraea/app/balancer/BalancerUtilsTest.java # app/src/test/java/org/astraea/app/cost/ReplicaDiskInCostTest.java commit 6c681fefe2d8a89fc6f60e6502ba4c69a87d91d8 Author: Zheng-Xian Li <[email protected]> Date: Fri Aug 19 16:21:13 2022 +0800 initial commit 4e9544d Author: Chia-Ping Tsai <[email protected]> Date: Fri Aug 19 12:24:28 2022 +0800 Pre-create hash code of NodeIno (opensource4you#600) commit c4a4700 Author: Chia-Ping Tsai <[email protected]> Date: Thu Aug 18 23:08:28 2022 +0800 Optimize dispatcher for large partitions (opensource4you#599) commit 61d6fc5 Author: Ching-Hong Fang <[email protected]> Date: Thu Aug 18 15:47:03 2022 +0800 Reconstruct `ReportFormat` (opensource4you#593) commit 23d95f0 Author: qoo332001 <[email protected]> Date: Thu Aug 18 14:28:16 2022 +0800 update commit a544ee7 Author: Zheng-Xian Li <[email protected]> Date: Thu Aug 18 11:56:27 2022 +0800 Revise `RebalancePlanProposal` (opensource4you#539) * revise proposal return type remove `Optional` * add javadoc * fix * fix javadoc * ShufflePlanGenerator should return just one plan when unable to propose * fix merge commit eff7f21 Author: Zheng-Xian Li <[email protected]> Date: Thu Aug 18 02:43:40 2022 +0800 Address the issue of replication throttle in Kafka broker implementation (opensource4you#594) commit d40e970 Author: Xiang-Jun Sun <[email protected]> Date: Thu Aug 18 00:57:38 2022 +0800 add costUtils (opensource4you#575) commit 05e6623 Author: qoo332001 <[email protected]> Date: Wed Aug 17 04:35:11 2022 +0800 add comment commit 5e0e666 Merge: a3abf0c e8077e7 Author: qoo332001 <[email protected]> Date: Wed Aug 17 04:29:23 2022 +0800 Merge branch 'main' of https://github.com/skiptests/astraea into migrateCost commit a3abf0c Author: qoo332001 <[email protected]> Date: Wed Aug 17 04:29:19 2022 +0800 add moveCost commit e8077e7 Author: Zheng-Xian Li <[email protected]> Date: Wed Aug 17 00:51:17 2022 +0800 Implementation of `ReplicationThrottler` (opensource4you#574) commit 0688360 Author: Chia-Ping Tsai <[email protected]> Date: Mon Aug 15 13:25:11 2022 +0800 Upgrade zookeeper from 3.7.0 to 3.7.1 (opensource4you#587) commit 7232e14 Author: Xiang-Jun Sun <[email protected]> Date: Sun Aug 14 23:51:51 2022 +0800 removeMapByPartition (opensource4you#583) * removeMapByPartition * fix issues * remove redundant code * spotless * remove redundan comment commit 5703206 Author: Chia-Ping Tsai <[email protected]> Date: Fri Aug 12 19:22:58 2022 +0800 Enable to trace the elapsed time of assigning/revoking partitions (opensource4you#581) commit 270a058 Author: Yi-Chen Wang <[email protected]> Date: Fri Aug 12 17:44:28 2022 +0800 Enhance NodeTopicSizeCost to offer partition cost (opensource4you#569) commit 666c5b4 Author: Chia-Ping Tsai <[email protected]> Date: Fri Aug 12 11:26:42 2022 +0800 Simplify performance report (opensource4you#578) commit 81b360f Author: Geordie <[email protected]> Date: Fri Aug 12 02:27:34 2022 +0800 Missing stacktrace when using Utils.packException (opensource4you#567) commit 30b8996 Author: Chia-Ping Tsai <[email protected]> Date: Fri Aug 12 02:08:59 2022 +0800 Enable consumer to trace all subscriptions (opensource4you#576) commit 20de1dd Author: Xiang-Jun Sun <[email protected]> Date: Thu Aug 11 19:38:19 2022 +0800 add ClusterCost in ReplicaDiskInCost (opensource4you#563) commit 3c45fc5 Author: Chia-Ping Tsai <[email protected]> Date: Thu Aug 11 14:45:39 2022 +0800 Add simple chaos monkey to kill consumers (opensource4you#573) commit 69303f1 Author: Chia-Ping Tsai <[email protected]> Date: Thu Aug 11 01:20:09 2022 +0800 Performance tool gets hanging when number of consumers is smaller tha… (opensource4you#571) commit ae0147a Author: Ching-Hong Fang <[email protected]> Date: Thu Aug 11 00:08:49 2022 +0800 Wait for file writer complete (opensource4you#570) commit 0d62821 Author: Kuan-Po Tseng <[email protected]> Date: Wed Aug 10 23:34:25 2022 +0800 Fix Utils.swallowException (opensource4you#572) commit a16c6fd Author: Chia-Ping Tsai <[email protected]> Date: Wed Aug 10 11:03:35 2022 +0800 Enable to unsubscribe/resubscribe topics/partitions (opensource4you#564) commit 1648e7b Author: Zhi-Mao Teng <[email protected]> Date: Mon Aug 8 23:20:17 2022 +0800 Translate Run Grafana (opensource4you#373) * Translate Run Grafana * Add more description and rename file * change picture link * add a link of Prometheus doc to run_grafana * Add a link to README * fixed some descriptions commit 09b2d94 Author: Xiang-Jun Sun <[email protected]> Date: Mon Aug 8 21:29:27 2022 +0800 add ClusterCost interface (opensource4you#514) commit 89c18f7 Author: Zheng-Xian Li <[email protected]> Date: Mon Aug 8 19:00:03 2022 +0800 Implementation of `ReplicationThrottler` (opensource4you#478) commit c31a578 Author: Chia-Ping Tsai <[email protected]> Date: Mon Aug 8 01:29:05 2022 +0800 Upgrade kafka from 3.2.0 to 3.2.1 (opensource4you#561) commit 0293711 Author: Chia-Ping Tsai <[email protected]> Date: Mon Aug 8 01:26:31 2022 +0800 Upgrade gradle from 7.5.0 to 7.5.1 (opensource4you#560) commit 8d83297 Author: Zhi-Mao Teng <[email protected]> Date: Sun Aug 7 23:49:28 2022 +0800 Translate run_prometheus.sh (opensource4you#485) * Translate run_prometheus.sh * delete a redundant photo * add more descriptions to readme * fixed words commit 95a2fc7 Author: Ching-Hong Fang <[email protected]> Date: Sun Aug 7 21:26:27 2022 +0800 Add Consumer Node Metrics (opensource4you#552) commit c52ca10 Author: YiHuan <[email protected]> Date: Sun Aug 7 17:32:40 2022 +0800 Add YHL to Authors (opensource4you#558) commit 98daa0b Author: Geordie <[email protected]> Date: Sun Aug 7 00:27:23 2022 +0800 Add Kafka controller metrics (opensource4you#554) commit d70b167 Author: chaohengstudent <[email protected]> Date: Sat Aug 6 15:48:04 2022 +0800 Add Chao-Heng to author list (opensource4you#550) commit c5f9fba Author: Zhi-Mao Teng <[email protected]> Date: Sat Aug 6 14:48:38 2022 +0800 Translate Performance Benchmark (opensource4you#374) * Translate Performance Benchmark * Add example of using performance tool * rename and add more description * add more descriptions * delete some descriptions * add more descriptions and delete some redundant words * fix some descriptions Co-authored-by: chinghongfang <[email protected]> commit 4cb9edb Author: Chia-Ping Tsai <[email protected]> Date: Fri Aug 5 21:46:12 2022 +0800 Fix NodeLatencyCostTest#testCost (opensource4you#548) commit 50aa89e Author: Xiang-Jun Sun <[email protected]> Date: Fri Aug 5 15:56:05 2022 +0800 simplify sizeTimeSeries (opensource4you#547) commit b0203a7 Author: Geordie <[email protected]> Date: Fri Aug 5 15:13:27 2022 +0800 Fix TopicHandlerTest#testDeleteTopic (opensource4you#545) commit c204e85 Author: Xiang-Jun Sun <[email protected]> Date: Fri Aug 5 14:41:13 2022 +0800 Remove properties in cost (opensource4you#524) commit 0e3d480 Author: Chia-Ping Tsai <[email protected]> Date: Thu Aug 4 00:18:05 2022 +0800 Apply Utils.constructCostFunction (opensource4you#538) commit b143c0e Author: Kuan-Po Tseng <[email protected]> Date: Wed Aug 3 23:56:59 2022 +0800 Remove limit in RecordHandler#get (opensource4you#542) commit 29fbf4e Author: Zheng-Xian Li <[email protected]> Date: Wed Aug 3 23:54:47 2022 +0800 Balance Process Demo (opensource4you#534) commit aa8af06 Author: Geordie <[email protected]> Date: Wed Aug 3 00:45:07 2022 +0800 Add delete group api (opensource4you#509) commit 3133d89 Author: Haser0305 <[email protected]> Date: Tue Aug 2 23:32:42 2022 +0800 Add Jia-Sheng Chen (opensource4you#540) commit 75fce1a Author: Zheng-Xian Li <[email protected]> Date: Tue Aug 2 00:55:26 2022 +0800 Construct `CostFunction` via Reflection (opensource4you#535) * construct CostFunction by reflection * simple commit f52db19 Author: Chia-Ping Tsai <[email protected]> Date: Sun Jul 31 21:10:48 2022 +0800 Refactor ReplicaManager metrics (opensource4you#532) commit c9022f2 Author: Chia-Ping Tsai <[email protected]> Date: Sun Jul 31 19:11:06 2022 +0800 Refactor network metrics (opensource4you#531) commit 5ff8628 Author: Chia-Ping Tsai <[email protected]> Date: Sun Jul 31 16:44:50 2022 +0800 Refactor producer metrics (opensource4you#530) commit c68fbdd Author: Geordie <[email protected]> Date: Sun Jul 31 02:15:02 2022 +0800 Add delete topic api (opensource4you#507) commit 814c5d8 Author: Chia-Ping Tsai <[email protected]> Date: Sat Jul 30 23:55:22 2022 +0800 Fix testDeleteRecord (opensource4you#529) commit be13aab Author: Chia-Ping Tsai <[email protected]> Date: Sat Jul 30 04:49:28 2022 +0800 Only take recent producer metrics (opensource4you#528) commit a45388a Author: Chia-Ping Tsai <[email protected]> Date: Sat Jul 30 02:21:08 2022 +0800 Fix node metrics (opensource4you#527) commit 381d913 Author: Chia-Ping Tsai <[email protected]> Date: Fri Jul 29 22:40:51 2022 +0800 Refactor broker topic metrics (opensource4you#521) commit c7fb4a9 Author: Chia-Ping Tsai <[email protected]> Date: Fri Jul 29 22:40:22 2022 +0800 Remove CostUtils (opensource4you#522) commit 5d29e0a Author: Xiang-Jun Sun <[email protected]> Date: Fri Jul 29 15:08:50 2022 +0800 remove PartitionScore (opensource4you#525) commit bc502e4 Author: Chia-Ping Tsai <[email protected]> Date: Fri Jul 29 00:51:43 2022 +0800 Refactor host metrics (opensource4you#520) commit 554d14d Author: Chia-Ping Tsai <[email protected]> Date: Thu Jul 28 23:50:54 2022 +0800 Refactor MBeanClient (opensource4you#519) commit cb94360 Author: Ching-Hong Fang <[email protected]> Date: Thu Jul 28 22:38:35 2022 +0800 Use the same unit on bytes recording (opensource4you#511) commit f09b48a Author: Chia-Ping Tsai <[email protected]> Date: Thu Jul 28 22:02:55 2022 +0800 Refactor MemoryCost (opensource4you#518) commit 6332d0a Author: Chia-Ping Tsai <[email protected]> Date: Thu Jul 28 21:05:26 2022 +0800 Refactor CpuCost (opensource4you#517) commit 615d1e4 Author: Chia-Ping Tsai <[email protected]> Date: Thu Jul 28 20:43:37 2022 +0800 Fix testConsumerRebalanceListener (opensource4you#516) commit 4e787f9 Author: Chia-Ping Tsai <[email protected]> Date: Thu Jul 28 20:41:42 2022 +0800 Move purgatory metrics to separate file (opensource4you#515) commit 9a7731e Author: Chia-Ping Tsai <[email protected]> Date: Thu Jul 28 12:00:23 2022 +0800 Remove ThreadPool (opensource4you#513) commit 5dc5de3 Author: Chia-Ping Tsai <[email protected]> Date: Thu Jul 28 02:01:08 2022 +0800 Refactor Performance tool (opensource4you#512)
#553 從其中切割出,爲NodeTopicSize加入partition cost。