Cpp client: add multiTopicsConsumer #1996

zhaijack · 2018-06-20T02:27:43Z

In PR #1103, we add Multi-Topics-Consumer in java client. This is a catch up work to add it in cpp client.

sijie · 2018-06-20T16:02:48Z

@merlimat @ivankelly can you review this? we need this in 2.1

merlimat · 2018-06-20T16:06:16Z

can you review this? we need this in 2.1

Then we should also have the Python, C and Go versions

sijie · 2018-06-20T16:13:59Z

Then we should also have the Python, C and Go versions

I guess this will take extra weeks to get those completed. Are you suggesting dropping this for 2.1 release?

merlimat · 2018-06-20T16:06:56Z

pulsar-client-cpp/include/pulsar/Client.h

@@ -99,6 +99,15 @@ class Client {
    void subscribeAsync(const std::string& topic, const std::string& consumerName,
                        const ConsumerConfiguration& conf, SubscribeCallback callback);

+    Result subscribe(const std::vector<std::string>& topics, const std::string& consumerName,


consumerName -> subscriptionName

merlimat · 2018-06-20T16:07:24Z

pulsar-client-cpp/include/pulsar/Message.h

+    /**
+     * Only for MultiTopicsConsumer to set a valid topicName
+     */
+    void setTopicName(std::string topicName) const;


This should not be exposed in API.

merlimat · 2018-06-20T16:07:41Z

pulsar-client-cpp/include/pulsar/Message.h

+    /**
+     * Only for MultiTopicsConsumer to get a valid topicName
+     */
+    std::string& getTopicName() const;


return type should be const std::string&

In a single topic, we should have the information as well, so we should restrict this method only for the MultiTopicsConsumer case

Thanks @merlimat. Currently, as you suggested, move set/getTopicName into private. Is it enough? e.g. single ConsumerImpl could call these 2 methods, but It is not useful at all, maybe no developer will use it inside ComsumerImpl.

Or we could use this way to make set/getTopicName only available to MultiTopicsConsumer?

merlimat · 2018-06-20T16:08:49Z

pulsar-client-cpp/include/pulsar/MessageId.h

+    /**
+     * Only for MultiTopicsConsumer to set a valid topicName
+     */
+    void setTopicName(std::string topicName);


This should not be in public API

merlimat · 2018-06-20T16:10:38Z

pulsar-client-cpp/include/pulsar/MessageId.h

+    /**
+     * Only for MultiTopicsConsumer to get a valid topicName
+     */
+    std::string& getTopicName() const;


I wouldn't expose this in public API for MessageId. It's useful for Message but, unless strictly needed, I'd leave it out from MessasgeId interface

Thanks, Seems need it. in acknowledgeAsync, we use getTopicName(), then use the topicName to find which sub-consumer to do the ack. Also, in UnAckedMessageTracker to remove MessageIds that belongs to a sub-consumer.
will move this into private.

Sure, I understand that we need it internally, but we should be careful to expose in API. This method could only be exposed in MessageIdImpl

merlimat · 2018-06-20T16:11:55Z

pulsar-client-cpp/lib/MessageIdImpl.h


 class MessageIdImpl {
   public:
-    MessageIdImpl() : ledgerId_(-1), entryId_(-1), partition_(-1), batchIndex_(-1) {}
+    MessageIdImpl() : ledgerId_(-1), entryId_(-1), partition_(-1), batchIndex_(-1), topicName_(emptyString) {}


no need to use emptyString. initializing with topicName_() will have same effect

merlimat · 2018-06-20T16:12:16Z

pulsar-client-cpp/lib/MessageImpl.cc


-MessageImpl::MessageImpl() : metadata(), payload(), messageId(), cnx_(0) {}
+MessageImpl::MessageImpl() : metadata(), payload(), messageId(), cnx_(0), topicName_(emptyString) {}


same here, no need for emptyString

merlimat · 2018-06-20T16:20:51Z

I guess this will take extra weeks to get those completed. Are you suggesting dropping this for 2.1 release?

Most users will access this feature through either Python (or Go) bindings, rather than using straight C++ API. I think that it would be fine to plan for 2.2 and include bindings as well.

sijie · 2018-06-20T16:21:28Z

okay. moved to 2.2

ivankelly

I haven't gone through thoroughly. I'll give it another pass tomorrow.

ivankelly · 2018-06-21T14:22:20Z

pulsar-client-cpp/lib/ClientImpl.cc

+            lock.unlock();
+            callback(ResultAlreadyClosed, Consumer());
+            return;
+        } else if (!topics.empty() && !(topicNamePtr = MultiTopicsConsumerImpl::topicNamesValid(topics))) {


This assignment is very hidden. I would move the call to topicNamesValid to just after the lock.

ivankelly · 2018-06-21T15:42:49Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+
+DECLARE_LOG_OBJECT()
+
+namespace pulsar {


using namespace pulsar;

Will avoid the braces. applies for whole file. Don't do it in a header though.

ivankelly · 2018-06-21T15:43:50Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+                                                 const LookupServicePtr lookupServicePtr)
+    : client_(client),
+      subscriptionName_(subscriptionName),
+      topic_(topicName ? topicName->toString() : "EmptyTopics"),


You've already got special handling to avoid empty topicName in another place.

Thanks, no, not handling it in clientimpl. It is possible we subscribe on empty topics when pattern subscribe added.

ivankelly · 2018-06-21T15:45:37Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+void MultiTopicsConsumerImpl::start() {
+    if (topics_.empty()) {
+        setState(Ready);
+        LOG_INFO("No topics passed in when create MultiTopicsConsumer.");


INFO may be a bit loud for this.

ivankelly · 2018-06-21T15:47:26Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+    }
+}
+
+void MultiTopicsConsumerImpl::handleOneTopicSubscribe(Result result, Consumer consumer,


This is a callback, but the name doesn't indicate it.
handleOneTopicSubscribe -> handleOneTopicSubscribed

ivankelly · 2018-06-21T15:53:09Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+    lock.unlock();
+    allTopicPartitionsNumber_->fetch_add(partitionsNumber);
+
+    boost::shared_ptr<std::atomic<int>> atomicIntPtr = boost::make_shared<std::atomic<int>>(0);


atomicIntPtr says nothing about what this is for. Also, you can just initialize with the number of partition, and decrement each time. This would avoid you having to pass partitionsNumber to the callback

Same for the similar patternin topics.

ivankelly · 2018-06-21T15:55:30Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+    }
+}
+
+void MultiTopicsConsumerImpl::handleUnsubscribeAsync(Result result,


-> handleUnsubscribed

ivankelly · 2018-06-21T15:56:14Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+    }
+}
+
+void MultiTopicsConsumerImpl::handleUnsubscribeOneTopicAsync(


-> handleOneTopicUnsubscribed

zhaijack · 2018-06-24T23:44:27Z

@ivankelly, Thanks for the comments, change it accordingly.

ivankelly · 2018-06-25T08:33:16Z

pulsar-client-cpp/lib/MultiTopicsBrokerConsumerStatsImpl.cc

+#include <algorithm>
+#include <numeric>
+
+namespace pulsar {


this can be changed to using namespace pulsar also

ivankelly · 2018-06-25T08:35:31Z

pulsar-client-cpp/lib/MessageImpl.h


    const std::string& getPartitionKey() const;
    bool hasPartitionKey() const;

    uint64_t getPublishTimestamp() const;
    uint64_t getEventTimestamp() const;

+    /**
+     * Only for MultiTopicsConsumer to get a valid topicName


There's no reason this can't work for non-multitopic consumers also. In fact, there was a request for this from a user.

Thanks, right, I found that request. cpp work tracked in issue #2190

ivankelly · 2018-06-25T08:48:05Z

pulsar-client-cpp/lib/MultiTopicsBrokerConsumerStatsImpl.cc

+bool MultiTopicsBrokerConsumerStatsImpl::isValid() const {
+    bool isValid = true;
+    for (int i = 0; i < statsList_.size(); i++) {
+        isValid &= statsList_[i].isValid();


while technically this will work, it's strange to use a bitwise operator here.
boolean operators should be used with bools.

isValid = isValid && statsLists_[i].isValid();

ivankelly · 2018-06-25T08:50:42Z

pulsar-client-cpp/lib/MultiTopicsBrokerConsumerStatsImpl.cc

+        return false;
+    }
+
+    bool isValid = true;


Duplicate code with isValid function above.

ivankelly · 2018-06-25T08:51:39Z

pulsar-client-cpp/lib/MultiTopicsBrokerConsumerStatsImpl.h

+    virtual uint64_t getMsgBacklog() const;
+
+    /** Returns the BrokerConsumerStatsImpl at of ith partition */
+    BrokerConsumerStats getBrokerConsumerStats(int index);


Shouldn't the functions belong and including this one be private?

ivankelly · 2018-06-25T09:16:32Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+    assert(previous > 0);
+
+    if (result != ResultOk) {
+        state_ = Failed;


state shouldn't be set here. It will be set when the failure on the promise propagates up.

ivankelly · 2018-06-25T09:16:51Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+
+    if (partitionsNeedCreate->load() == 0) {
+        topicSubResultPromise->setValue(Consumer(shared_from_this()));
+        return;


ivankelly · 2018-06-25T09:19:29Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+                  << result << " subscription - " << subscriptionName_);
+    }
+
+    if (consumerUnsubed->load() == allTopicPartitionsNumber_->load()) {


rename allTopicPartitionsNumber_ to numTopicPartitions_

ivankelly · 2018-06-25T09:26:09Z

pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc

+// subscribe for passed in topic
+Future<Result, Consumer> MultiTopicsConsumerImpl::subscribeOneTopicAsync(const std::string& topic) {
+    TopicNamePtr topicName;
+    ConsumerSubResultPromisePtr topicPromise = boost::make_shared<Promise<Result, Consumer>>();


shouldn't there be a check that the topic is in the same namespace as preexisting topics?

ivankelly · 2018-06-25T09:29:13Z

pulsar-client-cpp/tests/BasicEndToEndTest.cc

+    std::string url3 =
+        adminUrl + "admin/persistent/prop/unit/ns3/testMultiTopicsConsumerDifferentNamespace3/partitions";
+
+    int res = makePutRequest(url1, "2");


In form these tests are verify similar to the integration tests (i.e. Pulsar is running somewhere else). They should be able to use the commandline for doing stuff like this. Not something for this patch, but something to think about.

merlimat · 2018-06-25T17:02:03Z

pulsar-client-cpp/include/pulsar/MessageId.h

@@ -22,13 +22,12 @@
 #include <iosfwd>
 #include <stdint.h>
 #include <boost/shared_ptr.hpp>
+#include <lib/MessageIdImpl.h>


We cannot include lib/*.h headers from public API. We need to just do the forward declaration

merlimat · 2018-06-25T17:03:45Z

pulsar-client-cpp/lib/ClientImpl.cc

+
+    consumer->getConsumerCreatedFuture().addListener(
+        boost::bind(&ClientImpl::handleConsumerCreated, shared_from_this(), _1, _2, callback, consumer));
+    Lock lock(mutex_);


Since we're acquiring the mutex above as well, I'd rather have a single continuous section with the mutex

merlimat · 2018-06-25T17:04:35Z

pulsar-client-cpp/lib/MessageImpl.cc

@@ -78,4 +78,12 @@ void MessageImpl::setPartitionKey(const std::string& partitionKey) {
 }

 void MessageImpl::setEventTimestamp(uint64_t eventTimestamp) { metadata.set_event_time(eventTimestamp); }
+
+void MessageImpl::setTopicName(std::string topicName) {


const std::string&

merlimat · 2018-06-25T17:07:05Z

pulsar-client-cpp/lib/MessageImpl.h

@@ -43,13 +43,24 @@ class MessageImpl {
    SharedBuffer payload;
    MessageId messageId;
    ClientConnection* cnx_;
+    std::string topicName_;


As an optimization, to avoid to create (and copy) the topic string object for each message id, we could use a std::string* to store the name. This would work if we pass a reference of the topic name held by the consumer instance, which will be valid.

The getTopicName() would still have the same signature.

merlimat · 2018-06-25T17:07:54Z

pulsar-client-cpp/lib/MultiTopicsBrokerConsumerStatsImpl.cc

+    return isValid;
+}
+
+const std::string MultiTopicsBrokerConsumerStatsImpl::getAddress() const {


Remove const

Thanks. @merlimat, the const in const std::string, or const {)? not quite understand the reason of remove const?

merlimat · 2018-06-25T17:08:01Z

pulsar-client-cpp/lib/MultiTopicsBrokerConsumerStatsImpl.cc

+    return str;
+}
+
+const ConsumerType MultiTopicsBrokerConsumerStatsImpl::getType() const {


Remove const

merlimat · 2018-06-25T17:09:11Z

pulsar-client-cpp/lib/MultiTopicsBrokerConsumerStatsImpl.cc

+}
+
+const std::string MultiTopicsBrokerConsumerStatsImpl::getAddress() const {
+    std::string str;


Instead of concatenating a string with +=, we should use std::stringstream

merlimat · 2018-06-25T17:10:32Z

pulsar-client-cpp/lib/TopicName.cc

@@ -224,4 +224,7 @@ const std::string TopicName::getTopicPartitionName(unsigned int partition) {
    topicPartitionName << toString() << PartitionedProducerImpl::PARTITION_NAME_SUFFIX << partition;
    return topicPartitionName.str();
 }
+
+boost::shared_ptr<NamespaceName> TopicName::getNamespaceName() { return namespaceName_; }


NamespaceNamePtr

zhaijack · 2018-07-18T23:43:06Z

@merlimat @ivankelly Thanks for the comments, It is updated, please help review it.

zhaijack · 2018-07-19T12:05:23Z

retest this please

for Jenkins: C++ / Python Tests timeout

ivankelly

lgtm #shipit

zhaijack · 2018-07-21T05:37:02Z

retest this please

In PR #1103, we add Multi-Topics-Consumer in java client. This is a catch up work to add it in cpp client.

sijie · 2018-08-27T08:52:17Z

merged as 500e393 in branch-2.1

sijie requested review from ivankelly and merlimat June 20, 2018 15:13

sijie added this to the 2.1.0-incubating milestone Jun 20, 2018

sijie added component/c++ area/client labels Jun 20, 2018

merlimat reviewed Jun 20, 2018

View reviewed changes

sijie modified the milestones: 2.1.0-incubating, 2.2.0-incubating Jun 20, 2018

zhaijack force-pushed the cpp-multi-topics branch from 64b5d06 to 7e43452 Compare June 21, 2018 15:19

ivankelly requested changes Jun 21, 2018

View reviewed changes

zhaijack force-pushed the cpp-multi-topics branch from a55dd4e to 7993d52 Compare June 22, 2018 12:39

ivankelly requested changes Jun 25, 2018

View reviewed changes

merlimat reviewed Jun 25, 2018

View reviewed changes

merlimat mentioned this pull request Jun 26, 2018

Python pulsar-client lacks multi-topic support #2033

Closed

zhaijack force-pushed the cpp-multi-topics branch from 7993d52 to f53c6cf Compare July 18, 2018 11:56

ivankelly approved these changes Jul 19, 2018

View reviewed changes

jiazhai added 5 commits July 20, 2018 15:19

add multiTopicsConsumer

ea612d0

fix clang format issue

7fa63dd

Change following matteo's comments

05d9932

un-export get/setTopicName in Message and MessageId

f240df3

change following ivan's comments

617f1d5

jiazhai added 2 commits July 20, 2018 15:19

change following ivan and matteo's comments

293fd92

fix clang issue

c46e395

zhaijack force-pushed the cpp-multi-topics branch from e1cf0cf to c46e395 Compare July 20, 2018 07:20

sijie approved these changes Jul 23, 2018

View reviewed changes

sijie merged commit 2543ccf into apache:master Jul 23, 2018

sijie pushed a commit that referenced this pull request Aug 27, 2018

Cpp client: add multiTopicsConsumer (#1996)

500e393

In PR #1103, we add Multi-Topics-Consumer in java client. This is a catch up work to add it in cpp client.

sijie modified the milestones: 2.2.0-incubating, 2.1.1-incubating Aug 27, 2018

merlimat mentioned this pull request Aug 28, 2018

Multitopic subscription missing in most clients especially C++? #2463

Closed

sijie mentioned this pull request Mar 27, 2019

cpp client work to get the topic information from consumed message #2190

Closed


		MessageImpl::MessageImpl() : metadata(), payload(), messageId(), cnx_(0) {}
		MessageImpl::MessageImpl() : metadata(), payload(), messageId(), cnx_(0), topicName_(emptyString) {}

Cpp client: add multiTopicsConsumer #1996

Cpp client: add multiTopicsConsumer #1996

Conversation

zhaijack commented Jun 20, 2018

sijie commented Jun 20, 2018

merlimat commented Jun 20, 2018

sijie commented Jun 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merlimat commented Jun 20, 2018

sijie commented Jun 20, 2018

ivankelly left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaijack commented Jun 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaijack Jul 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaijack commented Jul 18, 2018

zhaijack commented Jul 19, 2018 • edited Loading

ivankelly left a comment

Choose a reason for hiding this comment

zhaijack commented Jul 21, 2018

sijie commented Aug 27, 2018

zhaijack Jul 18, 2018 •

edited

Loading

zhaijack commented Jul 19, 2018 •

edited

Loading