Issue 937: add CommandGetLastMessageId to make reader know the end of topic #1066

zhaijack · 2018-01-15T12:45:04Z

This is the first version to collect suggestion to achieve it.

Motivation

We have recently introduced the concept of topic "Reader" as an alternative to the traditional pub-sub consumer abstraction.

A common followup request has been to have a way to identify when the reader has reached the last published entry on the topic.

There is no currently direct way to achieve that and using readNext(timeout) doesn't help because in case the client is not connected to broker, it doesn't mean that there are no more messages to read.

There are a few workaround that are not easy or desirable (eg: terminating the topic, or using HTTP admin API to check the backlog on the reader).
Since this is a common theme, we should have a good way to handle this.

Modifications

add CommandGetLastMessageId in PulsarApi.proto;
implementation CommandGetLastMessageId;
add a testcase in TopicReaderTest

Result

A new command CommandGetLastMessageId added in binary protocol

zhaijack · 2018-01-17T07:03:52Z

retest this please

ivankelly

Looks good. A few minor comments.

What happens when this is called but nothing has yet been written to the topic?

Also, there's a lot of changes unrelated to the logic changes (such as moving headers, whitespace). It'd be nice for these to be in a separate commit, even if in same pull request.

ivankelly · 2018-01-17T14:34:46Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedger.java

@@ -323,14 +321,16 @@
    public boolean isTerminated();

    /**
-     * Returns managed-ledger config 


unrelated change, adds noise to an already large review.

Thanks, this is the auto changing of intellij-idea for checkstyle.

ivankelly · 2018-01-17T14:35:12Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedger.java

     * @param config
     */
    void setConfig(ManagedLedgerConfig config);
+
+    Position getLastConfirmedEntry();


Needs javadoc

Thanks, will add if needed

ivankelly · 2018-01-17T14:38:22Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java

+
+            Position position = consumer.getSubscription().getTopic().getLastMessageId();
+
+            log.info("[{}] [{}][{}] Get LastMessageId {}", remoteAddress,


info is a bit loud for this. debug would be better.

Thanks, will change into debug

ivankelly · 2018-01-17T14:40:42Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java

+
+            log.info("[{}] [{}][{}] Get LastMessageId {}", remoteAddress,
+                consumer.getSubscription().getTopic().getName(), consumer.getSubscription().getName(), position);
+            MessageIdData messageId = MessageIdData.newBuilder()


Missing partition, which i guess you get from the subscription. How does this deal with batch messages? What if the returned position is a batch message? The message id returned to the client wouldn't actually exist in the stream. Not sure how to deal with this.

Thanks. Seems we got here only ledgerid and messageid. will noted in the client api that it only support non-partitioned topic. partition and batch index here should be default value.

It would be good to at least support partition id so that partitions can be compacted (this feature is needed for compaction). You can likely get it from the subscription or topic name. @merlimat could you weight in on this.

You probably can't get batch id because it's embedded in the message, so only the client has access. The only way to get this would be to implement the whole feature as sending the entire last message to the client, which likely isn't sensible. However, this does mean equality and comparisons may be messed up, as the returned last message id may not actually match the message id of a message in the topic.

noticed this just now, will add partition here, if it is needed.

Actually, maybe it would make more sense to add partition at the client side? From the broker point of view, a partition really just looks like another topic, it has it's own managed ledger. It's only from the client side that partition and batch id are important. (@merlimat @sijie what are your opinions on this)?

@merlimat @sijie could you chime in on whether the partition id should be added on the server side or on the client side?

I think it should be at client side, because currently "partition" related logic is at client side.

ivankelly · 2018-01-17T14:41:34Z

...-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java

@@ -903,6 +904,11 @@ public boolean isEncryptionRequired() {
        return CompletableFuture.completedFuture(null);
    }

+    @Override
+    public Position getLastMessageId() {
+        throw new UnsupportedOperationException("getLastMessageId is not supported on non-persistent topic");


This should be noted in the javadoc for the client api.

Thanks. will change it.

ivankelly · 2018-01-17T14:43:27Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/Consumer.java

@@ -278,4 +277,19 @@
     * @return a future to track the completion of the seek operation
     */
    CompletableFuture<Void> seekAsync(MessageId messageId);
+
+    /**
+     * Gets last message id of Topic.


Be more explicit about what the last message id of the topic actually is. It's the message id of the last message for the topic which has been published successfully to the broker.

Thanks. will add it.

ivankelly · 2018-01-17T14:44:11Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/Reader.java

@@ -62,4 +62,6 @@
     * Return true if the topic was terminated and this reader has reached the end of the topic
     */
    boolean hasReachedEndOfTopic();
+
+    MessageId getLastMessageId() throws PulsarClientException;


async version and javadoc.

thanks, will add it.

ivankelly · 2018-01-17T14:52:15Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java

+    public CompletableFuture<MessageIdData> sendGetLastMessageId(ByteBuf request, long requestId) {
+        CompletableFuture<MessageIdData> future = new CompletableFuture<>();
+
+        pendingGetLastMessageIdRequests.put(requestId, future);


@merlimat how are timeouts handled with Pulsar's RPC?

Unfortunately we don't have a general way of handling it.

merlimat · 2018-01-19T22:52:22Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/Consumer.java

+     *
+     * @return the last message id
+     */
+    MessageId getLastMessageId() throws PulsarClientException;


Rather than exposing this method, I would prefer to have a simpler way to check it, for a couple of reasons:

Using and comparing message ids can be tricky (was that < or > ? :) )

MessageId lastMessageId = consumer.getLastMessageId(); MessageId currentMessageId = null; while (currentMessageId == null || currentMessageId.compareTo(lastMessageId) <= 0) { Message msg = consumer.receive(); ///// consumer.acknowledge(msg); }

Most typical code would be like the above snippet and do a tight loop asking the broker for the last message id, without any throttling or flow-control

Rather, I would prefer something like:

while (consumer.hasMessageAvaiable()) { Message msg = consumer.receive(); // Do something consumer.acknowledge(msg); } // Exits loops when the consumer has caught up with the writer

Internally, the client library can ask the broker for the last message id, and cache it. For example if the lastMessageId=123:12 but the consumer is currently receiving 110:5, there's no need to keep asking the broker.

hmm consumer.hasMessageAvaiable() - I am not sure it is a good idea. However it introduces non-deterministic behaviour to users, where user will have no idea where to stop. but I agree rate-limiting is also important, however should we consider rate-limiting at broker side for this message, rather than do the rate-limiting at client side. In that way, it is a probably a better solution for long term. thoughts?

In the above example, rate limiting in broker will slow down the consumption

Thanks for all the comments.
If needed, will add a position var to track the last position inincomingMessages to achieve consumer.hasMessageAvaiable().

zhaijack · 2018-01-22T14:40:56Z

@ivankelly Thanks.
If no data write to topic, the lastMessageId contains an entryID of value -1, and we should return false for consumer.hasMessageAvaiable().
The unrelated changes are the auto changing of intellij-idea for checkstyle.

zhaijack · 2018-01-25T01:28:11Z

@ivankelly @sijie @merlimat Would you please help take a look of the new change? Thanks.

zhaijack · 2018-01-31T11:50:59Z

rebased master to fix conflicts

sijie · 2018-02-01T06:10:02Z

/cc @merlimat for review

sijie

one comment about bumping the protocol version. otherwise, the change looks good to me +1

sijie · 2018-02-01T07:00:25Z

pulsar-common/src/main/proto/PulsarApi.proto

@@ -415,6 +415,16 @@ message CommandConsumerStatsResponse {
        optional uint64 msgBacklog                  = 15;
 }

+message CommandGetLastMessageId {


since we are introducing a new command, can you bump the "ProtocolVersion"?

Thanks, will add it.

bumped proto version to v11

@zhaijack The client should check that the broker supported version is >= v11 before sending the command.

If the broker doesn't support the feature, in this case I don't think there's a possible fallback, so we should just throw exception to user.

Thanks. will add the checking

zhaijack · 2018-02-01T11:26:43Z

retest this please

merlimat

The change look overall good to me. Just left few suggestions.

It would also be good to have C++/Python implementation after this PR is merged so that we have the same feature for Reader in all langs

merlimat · 2018-02-06T23:43:00Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/Consumer.java

+    /**
+     * Asynchronously Check if there is message that has been published successfully to the broker in the topic.
+     */
+    CompletableFuture<Boolean> hasMessageAvailableAsync();


I think this feature is mostly useful on the Reader interface. I think we should hide it from here, at least until we can find a concrete use case for it.

Get it. will hide it here.

merlimat · 2018-02-06T23:43:20Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/Reader.java

+    /**
+     * Check if there is message that has been published successfully to the broker in the topic.
+     */
+    Boolean hasMessageAvailable() throws PulsarClientException;


Boolean -> boolean

To minimize the time "blocked", we could include the last-published message id when the broker sends the CommandMessage so that the client can avoid asking, most of the time, since it will already see that the last was ahead of current positition.

Yes. we have keep the value as lastMessageIdInBroker in ConsumerImpl, it will return from client most of the time, will not call into broker.

merlimat · 2018-02-06T23:44:30Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/Reader.java

@@ -62,4 +62,14 @@
     * Return true if the topic was terminated and this reader has reached the end of the topic
     */
    boolean hasReachedEndOfTopic();
+
+    /**
+     * Check if there is message that has been published successfully to the broker in the topic.


"Check if there is any message available to read from the current position"

merlimat · 2018-02-06T23:55:57Z

pulsar-common/src/main/proto/PulsarApi.proto

@@ -415,6 +415,16 @@ message CommandConsumerStatsResponse {
        optional uint64 msgBacklog                  = 15;
 }

+message CommandGetLastMessageId {


@zhaijack The client should check that the broker supported version is >= v11 before sending the command.

If the broker doesn't support the feature, in this case I don't think there's a possible fallback, so we should just throw exception to user.

merlimat · 2018-02-06T23:58:14Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java

-    private MessageIdImpl() {
-        this(-1, -1, -1);
+    // create MessageIdImpl use value of MessageId.earliest
+    public MessageIdImpl() {


Since the MessageidImpl is immutable, couldn't we just reference MessageId.earliest instead of constructing a new one equal to that?

Thanks, will change it back, and reference earliest when use this. I thought it may make this clearer than hard coded -1,

merlimat · 2018-02-07T00:05:49Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java

+    public CompletableFuture<MessageIdData> sendGetLastMessageId(ByteBuf request, long requestId) {
+        CompletableFuture<MessageIdData> future = new CompletableFuture<>();
+
+        pendingGetLastMessageIdRequests.put(requestId, future);


Unfortunately we don't have a general way of handling it.

merlimat · 2018-02-07T00:10:41Z

pulsar-broker/src/test/java/org/apache/pulsar/client/api/TopicReaderTest.java

+
+
+    @Test
+    public void testSimpleReaderReachEndofTopic() throws Exception {


One scenario in which we need to pay attention is when batches are used.

With batches, multiple messages are stored in a single BK entry, and the broker treat a batch as a unit, which the consumer will finally break up, presenting the individual messages to application.

thanks, will add the test.

merlimat · 2018-02-07T00:12:29Z

pulsar-broker/src/test/java/org/apache/pulsar/client/api/TopicReaderTest.java

@@ -28,8 +31,10 @@
 import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
-
+import javax.validation.constraints.AssertFalse;


Use testng assertFalse

Thanks. will change it.

zhaijack · 2018-02-09T15:07:57Z

retest this please

merlimat · 2018-02-09T20:20:06Z

pulsar-common/src/main/proto/PulsarApi.proto

@@ -135,6 +135,7 @@ enum ProtocolVersion {
 	v9 = 9;  // Added end of topic notification
 	v10 = 10;// Added proxy to broker
 	v11 = 11;// C++ consumers before this version are not correctly handling the checksum field
+			 //Added get topic's last messageId from broker


@zhaijack There was a merge issue here, v11 was already taken in master and this PR should be adding v12 now.

merlimat · 2018-02-10T01:20:17Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+    public CompletableFuture<Boolean> hasMessageAvailableAsync() {
+        final CompletableFuture<Boolean> booleanFuture = new CompletableFuture<>();
+
+        if (lastMessageIdInBroker.compareTo(lastDequeuedMessage) > 0 &&


We should do this check also in the sync hasMessageAvailable() case, so that we can avoid creating a CompletableFuture each time we check, if the lastMessageId is already cached.

Thanks. will add it.

merlimat · 2018-02-10T01:20:52Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+                .failedFuture(new PulsarClientException.AlreadyClosedException("Consumer was already closed"));
+        }
+
+        if (cnx().getRemoteEndpointProtocolVersion() < ProtocolVersion.v11.getNumber()) {


If it's not connected cnx() will return null here.

Thanks, will move the statement of isConnected up.

merlimat · 2018-02-10T01:23:05Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+                        cnx().getRemoteEndpointProtocolVersion()));
+        }
+
+        if (!isConnected()) {


If it's currently not connected, we should try to mask the exception from the user if the failure is transient.

There is already an operationTimeout defined in client, it would be good to have a way to retry internally with backoff up to that amount of time.

sijie · 2018-02-13T14:41:06Z

@zhaijack : can you address @merlimat and @ivankelly 's comments? we need this change for function worker.

zhaijack · 2018-02-13T14:45:48Z

Thanks, have resolved all the comments, and fixed rebase conflict.

merlimat

Looks good, just one final comment on the retries when disconnected.

merlimat · 2018-02-13T21:24:22Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+        }
+
+        if (!isConnected()) {
+            long opTimeoutMs = client.getConfiguration().getOperationTimeoutMs();


This implementation will block the caller of an asynchrounous method, which might be unexpected. One way to do the retries would be to have an internal method that gets called asynchrounously in recursion. For example, something like :

private CompletableFuture<MessageId> internalGetLastMessageIdAsync(Backoff backoff, long remainingTime) { if (connected) { // write on socket and return future } else { // if time is not elapsed yet... long nextDelay = backoff.next(); executor.schedule(() -> { remainingTime -= (timeSpentSinceLastCall); internalGetLastMessageIdAsync(backoff, remainingTime); }, nextDelay, TimeUnit.MILLISECONDS); } }

(Don't read too much in the previous example, I'm just trying to illustrate the basic idea)

Thanks, will change this.

merlimat · 2018-02-13T21:26:03Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+            }
+        }
+
+        if (cnx().getRemoteEndpointProtocolVersion() < ProtocolVersion.v12.getNumber()) {


Nit: please wrap this in a method like :

void Commands.peerSupportsGetLastMessageId(ClientCnx cnx);

merlimat · 2018-02-14T01:20:41Z

pulsar-common/src/main/java/org/apache/pulsar/common/api/Commands.java

@@ -952,4 +952,8 @@ public static ByteBuf newLookup(String topic, boolean authoritative, String orig
        lookupBroker.recycle();
        return res;
    }
+
+    public static boolean peerSupportsGetLastMessageId() {
+        return getCurrentProtocolVersion() >= ProtocolVersion.v12.getNumber();


This is just checking our own protocol version (which is always the "latest" from when we compiled the protobuf), though we need to check the other side version in this case the broker.

this should be like :

public static boolean peerSupportsGetLastMessageId(ClientCnx cnx) { return ctx.getRemoteEndpointProtocolVersion() >= ProtocolVersion.v12.getNumber(); }

merlimat · 2018-02-14T01:21:20Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

@@ -145,6 +151,9 @@
        this.batchMessageAckTracker = new ConcurrentSkipListMap<>();
        this.readCompacted = conf.getReadCompacted();

+        this.getLastIdExecutor = Executors


This would create 1 thread per each consumer, we should reuse the executor that is already available from PulsarClientImpl

merlimat · 2018-02-14T01:57:40Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+    private void internalGetLastMessageIdAsync(final Backoff backoff,
+                                               final AtomicLong remainingTime,
+                                               CompletableFuture<MessageId> future) {
+        if (isConnected()) {


I missed this before, but we should make sure the connection doesn't change while we're executing this method.

isConnected() is checking the current connection, but that might change when we ask for cnx() few lines below.

We need to first get a reference on ClientCnx and use that throughout the method.

ClientCnx cnx = cnx(); if (cnx != null) { // check cnx.getRemoteEndpointProtocolVersion(); cnx.sendGetLastMessageId()... }

Thanks, will change it.

merlimat

👍

merlimat added the type/feature The PR added a new feature or issue requested a new feature label Jan 15, 2018

merlimat added this to the 1.22.0-incubating milestone Jan 15, 2018

merlimat assigned zhaijack Jan 15, 2018

zhaijack force-pushed the issue_937 branch from 30b97fa to 8806f5f Compare January 17, 2018 01:01

ivankelly mentioned this pull request Jan 17, 2018

Compact algo #1044

Merged

ivankelly reviewed Jan 17, 2018

View reviewed changes

merlimat reviewed Jan 19, 2018

View reviewed changes

zhaijack force-pushed the issue_937 branch 2 times, most recently from 8553269 to 9c41a3d Compare January 22, 2018 14:35

zhaijack force-pushed the issue_937 branch from d67133f to fa0863a Compare January 31, 2018 11:49

sijie approved these changes Feb 1, 2018

View reviewed changes

zhaijack mentioned this pull request Feb 6, 2018

PIP-13-2/3: support regex based subscription #1165

Closed

merlimat reviewed Feb 7, 2018

View reviewed changes

zhaijack force-pushed the issue_937 branch from 0c77b60 to d749c40 Compare February 9, 2018 13:46

merlimat reviewed Feb 9, 2018

View reviewed changes

merlimat reviewed Feb 10, 2018

View reviewed changes

jiazhai added 7 commits February 13, 2018 17:58

add CommandGetLastMessageId to getlastMessageId of topic

f28020f

rebase master, change following comments

928e2e7

add partition index in GetLastMessageIdResponse

ec82b4d

fix rebase error

52c4d5b

bump proot version to v11

44b1280

change following comments

57ddf8a

change following comments2

0ef21df

zhaijack force-pushed the issue_937 branch from 97a91b1 to 0ef21df Compare February 13, 2018 09:58

merlimat reviewed Feb 13, 2018

View reviewed changes

change following comments3

2f90b85

merlimat reviewed Feb 14, 2018

View reviewed changes

change following comments

8da7703

merlimat reviewed Feb 14, 2018

View reviewed changes

get cnx() first

a5bd454

merlimat approved these changes Feb 14, 2018

View reviewed changes

merlimat modified the milestones: 1.22.0-incubating, 2.0.0-incubating Feb 14, 2018

merlimat merged commit 7404952 into apache:master Feb 14, 2018

sijie mentioned this pull request Apr 25, 2018

Allow reader to detect when it's caught up with writers #937

Closed

zhaijack mentioned this pull request May 30, 2018

Cpp client: add getLastMessageId and hasMessageAvailable in consmer and reader #1861

Merged

Technoboy- mentioned this pull request Jul 5, 2022

Discuss: Make hasMessageAvailable to interface Consumer. #16358

Open


		Position position = consumer.getSubscription().getTopic().getLastMessageId();

		log.info("[{}] [{}][{}] Get LastMessageId {}", remoteAddress,



		@Test
		public void testSimpleReaderReachEndofTopic() throws Exception {

Issue 937: add CommandGetLastMessageId to make reader know the end of topic #1066

Issue 937: add CommandGetLastMessageId to make reader know the end of topic #1066

Conversation

zhaijack commented Jan 15, 2018

Motivation

Modifications

Result

zhaijack commented Jan 17, 2018

ivankelly left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaijack Jan 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaijack commented Jan 22, 2018

zhaijack commented Jan 25, 2018

zhaijack commented Jan 31, 2018

sijie commented Feb 1, 2018

sijie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merlimat Feb 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaijack commented Feb 1, 2018

merlimat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merlimat Feb 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaijack commented Feb 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sijie commented Feb 13, 2018

zhaijack commented Feb 13, 2018

merlimat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivankelly left a comment •

edited

Loading

zhaijack Jan 21, 2018 •

edited

Loading

merlimat Feb 6, 2018 •

edited

Loading

merlimat Feb 6, 2018 •

edited

Loading