chore(pd-store-dev): integrate `pd-grpc`, `pd-common`, `pd-client` into hugegraph #2460

VGalaxies · 2024-02-25T07:19:03Z

subtask of #2265

During the code review, I found the following issues:

Similar functionality appears multiple times, such as stub connection-related code, with redundancy between PDClient.StubProxy and AbstractClientStubProxy.
Package partitioning:
1. PDPulse, PDPulseImpl should in pulse
2. PDWatch, PDWatchImpl should in watch
Unused code, see below

For the pd-client submodule, its structure is as follows:

.
├── client
│  ├── AbstractClient.java // Encapsulation of RPC calls and error handling
│  ├── AbstractClientStubProxy.java // Management of multiple host stubs
│  ├── Channels.java // Maintenance of host -> ManagedChannel
│  ├── ClientCache.java // Cache for store, partition, shard group, graph
│  ├── Discoverable.java // unused
│  ├── DiscoveryClient.java // unused
│  ├── DiscoveryClientImpl.java // unused
│  ├── KvClient.java // Encapsulation of kv.proto RPC calls, inherits from AbstractClient
│  ├── LicenseClient.java // unused
│  ├── PDClient.java // Encapsulation of pdpb.proto RPC calls, encapsulates PDWatch
│  ├── PDConfig.java // Configuration of PD RPC port
│  ├── PDPulse.java // pd_pulse.proto, Partition heartbeat, Bidirectional communication interface of pd-client and pd-server
│  ├── PDPulseImpl.java
│  ├── PDWatch.java // pd_watch.proto, Listen for changes in store, partition, shard group, graph
│  └── PDWatchImpl.java
├── pulse
│  ├── PartitionNotice.java
│  └── PulseServerNotice.java
└── watch
   ├── NodeEvent.java // Store node event
   ├── PartitionEvent.java // Partition event
   ├── PDWatcher.java // unused
   └── WatchType.java // unused

TODO:

unit tests
code style
CJK characters (ignored)
dependency-review

…-pd/hg-pd-client/ pom.xml hugegraph-pd/pom.xml

codecov · 2024-02-25T07:27:42Z

Codecov Report

Attention: Patch coverage is 0% with 339 lines in your changes are missing coverage. Please review.

Project coverage is 63.23%. Comparing base (47a68f0) to head (19a7184).

Files	Patch %	Lines
...org/apache/hugegraph/pd/common/PartitionCache.java	0.00%	218 Missing ⚠️
.../java/org/apache/hugegraph/pd/common/HgAssert.java	0.00%	44 Missing ⚠️
...in/java/org/apache/hugegraph/pd/common/KVPair.java	0.00%	20 Missing ⚠️
...ava/org/apache/hugegraph/pd/common/GraphCache.java	0.00%	16 Missing ⚠️
...apache/hugegraph/pd/common/PDRuntimeException.java	0.00%	14 Missing ⚠️
...org/apache/hugegraph/pd/common/PartitionUtils.java	0.00%	14 Missing ⚠️
...va/org/apache/hugegraph/pd/common/PDException.java	0.00%	13 Missing ⚠️

Additional details and impacted files

@@                Coverage Diff                 @@
##             pd-store-dev    #2460      +/-   ##
==================================================
- Coverage           63.80%   63.23%   -0.57%     
+ Complexity            829      827       -2     
==================================================
  Files                 511      518       +7     
  Lines               42622    42959     +337     
  Branches             5947     5981      +34     
==================================================
- Hits                27193    27164      -29     
- Misses              12679    13040     +361     
- Partials             2750     2755       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…1.39.0 to 1.53.0

…on from 1.39.0 to 1.53.0" This reverts commit e4cd901.

JackyYangPassion · 2024-03-05T02:19:40Z

...ph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/AbstractClientStubProxy.java

+
+    public String nextHost() {
+        String host = hostList.poll();
+        hostList.offer(host);   //移到尾部


Translate comments into English.

Translate comments into English.

We will do it together in another PR (later), reduce (style)time cost in each PR (So as others)

hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/ClientCache.java

hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/DiscoveryClient.java

hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/DiscoveryClientImpl.java

hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/PDClient.java

Pengzna · 2024-03-11T06:11:34Z

hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/ClientCache.java

+    private volatile Map<String, GraphCache> caches = new ConcurrentHashMap<>();
+
+    public ClientCache(org.apache.hugegraph.pd.client.PDClient pdClient) {
+        groups = new ConcurrentHashMap<>();
+        stores = new ConcurrentHashMap<>();
+        client = pdClient;
+    }


Maybe we can consider to put the initialization of member variables all in the constructor for a cleaner code style. This can be fixed in the next style related PR. @VGalaxies

Maybe we can consider to put the initialization of member variables all in the constructor for a cleaner code style. This can be fixed in the next style related PR. @VGalaxies

Yep, we could mark a TODO first & enhance it with the v4.0 together

imbajin · 2024-03-11T07:52:52Z

pom.xml

+        <!-- enable it by default in ARM Mac to handle the compilation problems:) -->
+        <profile>


enhance the building way here

TODO: also clean it in other repos

imbajin

PS: Historical issues will be optimized over time:)

VGalaxies added 2 commits February 25, 2024 15:12

git add hugegraph-pd/hg-pd-grpc/ hugegraph-pd/hg-pd-common/ hugegraph…

2bdf897

…-pd/hg-pd-client/ pom.xml hugegraph-pd/pom.xml

chore: compile

2c4ff51

VGalaxies requested a review from imbajin February 25, 2024 07:19

VGalaxies self-assigned this Feb 25, 2024

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. feature New feature labels Feb 25, 2024

VGalaxies changed the title ~~chore(pd-store-dev): integrate pd-grpc, pd-common, pd-client hugegraph~~ chore(pd-store-dev): integrate pd-grpc, pd-common, pd-client into hugegraph Feb 25, 2024

VGalaxies mentioned this pull request Feb 25, 2024

[Summary] integrate new modules PD and Store components into hugegraph (Breaking Change) #2265

Open

26 tasks

VGalaxies added the pd PD module label Feb 25, 2024

VGalaxies added 3 commits February 26, 2024 18:23

chore: remove JAVA 8 ci

c7f4a95

git add hugegraph-pd/hg-pd-test/

cd7d7fc

chore: adapt hg-pd-common tests

a5fd93f

VGalaxies force-pushed the intro-pd-client branch from 5ff9c96 to a5fd93f Compare February 26, 2024 12:11

VGalaxies added 10 commits February 26, 2024 20:36

fix: update DEFAULT_VERSION

4af39d0

chore: remove start-pd.sh

9ce9fc2

Merge branch 'pd-store-dev' into intro-pd-client

94b518c

chore: format hg-pd-test

f60af68

chore: format hg-pd-client

8a671de

chore: format hg-pd-common

c1cf4a2

chore: format hg-pd-grpc

f086c44

chore: bump jackson.version from 2.13.0 to 2.14.2, grpc.version from …

e4cd901

…1.39.0 to 1.53.0

Revert "chore: bump jackson.version from 2.13.0 to 2.14.2, grpc.versi…

37982e5

…on from 1.39.0 to 1.53.0" This reverts commit e4cd901.

chore: set dependency-review fail-on-severity from low to critical

8214ac0

JackyYangPassion reviewed Mar 5, 2024

View reviewed changes

JackyYangPassion approved these changes Mar 5, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Mar 5, 2024

JackyYangPassion approved these changes Mar 5, 2024

View reviewed changes

Pengzna reviewed Mar 11, 2024

View reviewed changes

chore: enhance the compilation in ARM-Mac

07ed6ff

imbajin reviewed Mar 11, 2024

View reviewed changes

imbajin and others added 2 commits March 11, 2024 16:20

chore: clean .flattened-pom.xml

16d9e16

TODO: also clean it in other repos

chore: remove maven prerequisites

19a7184

imbajin approved these changes Mar 11, 2024

View reviewed changes

imbajin merged commit 3dd06f2 into pd-store-dev Mar 11, 2024
13 of 15 checks passed

imbajin deleted the intro-pd-client branch March 11, 2024 09:03

VGalaxies mentioned this pull request Mar 16, 2024

[Summary] integrate new module PD into hugegraph #2481

Open

9 tasks

VGalaxies mentioned this pull request Apr 1, 2024

feat(pd): integrate pd-grpc&pd-common&pd-client #2498

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(pd-store-dev): integrate `pd-grpc`, `pd-common`, `pd-client` into hugegraph #2460

chore(pd-store-dev): integrate `pd-grpc`, `pd-common`, `pd-client` into hugegraph #2460

VGalaxies commented Feb 25, 2024 •

edited

Loading

codecov bot commented Feb 25, 2024 •

edited

Loading

JackyYangPassion Mar 5, 2024

imbajin Mar 5, 2024

Pengzna Mar 11, 2024 •

edited

Loading

imbajin Mar 11, 2024

imbajin Mar 11, 2024 •

edited

Loading

imbajin left a comment

		<!-- enable it by default in ARM Mac to handle the compilation problems:) -->
		<profile>

chore(pd-store-dev): integrate pd-grpc, pd-common, pd-client into hugegraph #2460

chore(pd-store-dev): integrate pd-grpc, pd-common, pd-client into hugegraph #2460

Conversation

VGalaxies commented Feb 25, 2024 • edited Loading

codecov bot commented Feb 25, 2024 • edited Loading

Codecov Report

JackyYangPassion Mar 5, 2024

Choose a reason for hiding this comment

imbajin Mar 5, 2024

Choose a reason for hiding this comment

Pengzna Mar 11, 2024 • edited Loading

Choose a reason for hiding this comment

imbajin Mar 11, 2024

Choose a reason for hiding this comment

imbajin Mar 11, 2024 • edited Loading

Choose a reason for hiding this comment

imbajin left a comment

Choose a reason for hiding this comment

chore(pd-store-dev): integrate `pd-grpc`, `pd-common`, `pd-client` into hugegraph #2460

chore(pd-store-dev): integrate `pd-grpc`, `pd-common`, `pd-client` into hugegraph #2460

VGalaxies commented Feb 25, 2024 •

edited

Loading

codecov bot commented Feb 25, 2024 •

edited

Loading

Pengzna Mar 11, 2024 •

edited

Loading

imbajin Mar 11, 2024 •

edited

Loading