Skip to content
This repository has been archived by the owner on Mar 31, 2023. It is now read-only.

[Data Plane Mgr] Support Pulsar Publisher #481

Merged
merged 10 commits into from
Nov 30, 2020
Merged

[Data Plane Mgr] Support Pulsar Publisher #481

merged 10 commits into from
Nov 30, 2020

Conversation

VanderChen
Copy link
Contributor

@VanderChen VanderChen commented Nov 18, 2020

This PR proposes the following change:

  • Add pulsar client as a publisher
  • Fix gRPC client to allow it coexist with the new pulsar client
  • Add logic to support GS programming based on port-level configuration

@xieus xieus requested review from chenpiaoping and Gzure November 19, 2020 07:02
@xieus xieus added the feature feature development label Nov 19, 2020
@xieus xieus added this to the Version 1.0.2020.11.30 milestone Nov 19, 2020
Copy link
Contributor

@xieus xieus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. This is a good start. Please resolve the comments one by one, and work with @chenpiaoping if you have any question.

#####Pulsar configuration#####
pulsar.url=pulsar://127.0.0.1:6650
pulsar.unicast.topic=unicast-topic1
host.ip.to.group.topic.map=group-topic1:192.168.131.131,10.10.10.11 group-topic2:192.168.131.131,11.11.11.12
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an OK short-term solution. Let us plan a long-term solution. One possibility to store the mapping in node metadata manager, as this is a host-level resource.

@chenpiaoping @VanderChen what do you think of?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create a tracking issue #490

@VanderChen If you can take this item, that would be great. Let us sync on Slack for more details.

@Autowired
private PulsarConfiguration configuration;

private Map<String, String> hostIpToGroupTopic;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let us plan to build these mapping from node metadata manager and store it in the local cache under development.

this.hostIpToGroupTopic = parseTopicConfig(configuration.getHostIpToGroupTopicMap());
this.groupTopicToMulticastTopic = parseTopicConfig(configuration.getGroupTopicToMulticastTopicMap());
}catch (Exception e) {
throw new Exception("Parse topic config error: " + e);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can define an explicit exception for this parsing error. Please find an example in https://github.com/futurewei-cloud/alcor/tree/master/services/data_plane_manager/src/main/java/com/futurewei/alcor/dataplane/exception

The idea is that we know every possible exception our own program may throw, and define it before hand. In each of the predefined exception, we can define its error code and message, and come to the debugging time, we can quickly locate where the exception is thrown and why based on the given error code and message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add a predefined exception to solve this problem.

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class UnicastFunction implements Function<UnicastGoalStateByte, UnicastGoalStateByte> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like this function implementation, which is neat.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, where is this UnicastFunction used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both UnicastFunction and MulticastFuncation will be bundled as jar packages. These will be deployed to the Pulsar cluster as function workers by the Pulsar CLI tool.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to use Java interface to deploy function workers. This will enable automatic deployment when the dpm service starts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool looking forward to it 👍

public class MulticastFunction implements Function<MulticastGoalStateByte, MulticastGoalStateByte> {
private static final Logger LOG = LoggerFactory.getLogger(MulticastFunction.class);
public static final ThreadPoolExecutor executor = new ThreadPoolExecutor(
20,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should use configuration here.

Check the thread pool configurations in application.properties

grpc.min-threads = 100
grpc.max-threads = 200

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll fix this in the next commit.

}

Producer<UnicastGoalStateByte> producer = pulsarClient
.newProducer(JSONSchema.of(UnicastGoalStateByte.class))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to JSONSchema, what does other schema Pulsar support?

"unicastGoalStates: {}", nextTopic, unicastGoalState);
}

List<Goalstateprovisioner.GoalStateOperationReply.GoalStateOperationStatus> tempList = new ArrayList<>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we don't have a response from MQ for now. @chenpiaoping Let us plan to design a feedback channel here: it would be ideal if we can reuse the same topic with different key.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create a tracking issue #489 @chenpiaoping

@xieus xieus changed the title Merge pulsar to dpm [Data Plane Mgr] Support Pulsar Publisher Nov 19, 2020
@xieus xieus linked an issue Nov 19, 2020 that may be closed by this pull request
// TODO: Find a field to decide client
if (true){
return pulsarDataPlaneClient.createGoalStates(unicastGoalStates, multicastGoalState);
if (networkConfig.getPortEntities().get(0).isFastPath()){
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a question about this. Will there be different isFastPath values in multiple portEntity? How to deal with this situation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IsFastPath is a port-level config. Although we allow different values for multiple port entities for the sake of flexibility, we don't have such a scenario for now. In terms of implementation, we could make a port-level decision.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, for the same NetworkConfiguration, we need to divide it into two groups to generate Goalstate based on whether it is fastpath. Am I following this logic to modify

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a NetworkConfiguration includes multiple port entities, DPM will process each entity independently and likely send the corresponding GS to its target host.

At the moment of sending the GS, we need to use the isFastPath field to determine go with the fast path or MQ scaling path.

@VanderChen VanderChen requested a review from xieus November 30, 2020 04:08
@codecov-io
Copy link

Codecov Report

Merging #481 (3b1a466) into master (2d1490d) will decrease coverage by 0.00%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #481      +/-   ##
============================================
- Coverage     36.59%   36.59%   -0.01%     
  Complexity     1162     1162              
============================================
  Files           453      453              
  Lines         10869    10869              
  Branches       1393     1393              
============================================
- Hits           3978     3977       -1     
- Misses         6349     6350       +1     
  Partials        542      542              
Impacted Files Coverage Δ Complexity Δ
...alcor/elasticipmanager/dao/ElasticIpAllocator.java 63.53% <0.00%> (-0.28%) 48.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d1490d...3b1a466. Read the comment docs.

Copy link
Contributor

@xieus xieus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @VanderChen.

I left a few comments and create two tracking issue #489 and #490 for two pending requests. Please take a look.

@@ -517,24 +522,13 @@ private UnicastGoalState buildUnicastGoalState(NetworkConfiguration networkConfi
return unicastGoalState;
}

private List<Map<String, List<GoalStateOperationStatus>>> createPortConfiguration(NetworkConfiguration networkConfig) throws Exception {
private List<Map<String, List<GoalStateOperationStatus>>> doCreatePortConfiguration(NetworkConfiguration networkConfig,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like this method abstraction and its renaming!

Comment on lines +1 to +15
pulsar-admin functions create \
--jar target/dataplanemanager-0.1.0-SNAPSHOT.jar \
--classname com.futurewei.alcor.dataplane.client.pulsar.function.UnicastFunction \
--tenant public --namespace default \
--name unicast-function \
--inputs persistent://public/default/unicast-topic1 \
--output persistent://public/default/group-topic1

pulsar-admin functions create \
--jar target/dataplanemanager-0.1.0-SNAPSHOT.jar \
--classname com.futurewei.alcor.dataplane.client.pulsar.MulticastFunction \
--tenant public --namespace default \
--name multicast-function \
--inputs persistent://public/default/multicast-topic1 \
--output persistent://public/default/group-topic1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@VanderChen I was wondering how the function-deploy.sh gets triggered.

"unicastGoalStates: {}", nextTopic, unicastGoalState);
}

List<Goalstateprovisioner.GoalStateOperationReply.GoalStateOperationStatus> tempList = new ArrayList<>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create a tracking issue #489 @chenpiaoping

#####Pulsar configuration#####
pulsar.url=pulsar://127.0.0.1:6650
pulsar.unicast.topic=unicast-topic1
host.ip.to.group.topic.map=group-topic1:192.168.131.131,10.10.10.11 group-topic2:192.168.131.131,11.11.11.12
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create a tracking issue #490

@VanderChen If you can take this item, that would be great. Let us sync on Slack for more details.

@xieus xieus merged commit a893709 into futurewei-cloud:master Nov 30, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature feature development
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Microservice] DPM refactor to support MQ pub/sub client
4 participants