[@azure/event-hubs] Trying to close a subscription for which there is no Container for checkpoints, gives error

[marcodalessandro]

• Package Name: @azure/event-hubs
• Package Version: 5.2.2
• Operating system: Docker container running node:12.18.2-slim [linux; amd64]
• nodejs
  • version: 12.8.2
• browser
  • name/version:
• typescript
  • version: 3.9.7
• Is the bug related to documentation in
  • README.md
  • source code documentation
  • SDK API docs on https://docs.microsoft.com

Describe the bug
If you try to invoke subscription.close() on an eventhub subscription for which the container (keeping track of the checkpoints) has already been deleted, it gives this error:

{
  "stack": "Error: Error occurred while fetching the list of blobs. \nError: <?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.\nRequestId:<REDACTED>\nTime:2020-09-16T18:07:28.9964654Z</Message></Error>\n    at BlobCheckpointStore.<anonymous> (/app/node_modules/@azure/eventhubs-checkpointstore-blob/dist/index.js:111:23)\n    at Generator.throw (<anonymous>)\n    at rejected (/app/node_modules/tslib/tslib.js:108:69)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)",
  "message": "Error occurred while fetching the list of blobs. \nError: <?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.\nRequestId:<REDACTED>\nTime:2020-09-16T18:07:28.9964654Z</Message></Error>"
}

To Reproduce
Steps to reproduce the behavior:

The easiest way to repro is to delete the container that stores the checkpoints and then invoke the subscription.close().
PseudoCode:

     await deleteContainerForTheSubscription(...);
     // wait some time (container deletion goes through deleting and then delete phase)
     await subscription.close();

The subscription, even though it throws this error, seems not to be dangling and not to be polling the eventhub anymore.

Notes:

• The error thrown by subscription.close() is not propagated to the processError handler.
• If you try to delete the container when the subscription is still open (and you don't call the subscription.close()), at any polling (that I believe happens every 10 seconds by default), the error is instead propagated to the processError.

Expected behavior
It seems the subscription.close() is listing all the checkpoint containers before closing the subscription.
I would expect ignoring the list of current checkpoint containers.
If instead for any reasons this behavior has some design motivation, it could probably be explicitly noted in the documentation that before deleting the checkpoint all the subscription needs to be closed (but this is something that probably cannot be achieved in all scenarios).

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Our application is running on several K8 clusters and uses multiple instances of different microservices (stateless). One of this microservice (running with multiple instances) opens subscription on eventhub and uses @azure/[email protected] to keep track of checkpoints. Under some conditions we need to close the subscription and in our scenario each running instances tries to:

• close each subscription (open towards the same eventhub);
• delete the container with checkpoint (not needed anymore), handling any 409 or 404 since the operation is executed from all the instances of the same microservice.

[richardpark-msft]

Hi @marcodalessandro, let me explain what's happening first and then we can talk about a path forward.

The reason that the subscription causes a checkpoint store to occur is that the subscription, when stopped, is updating the remote blob store so any other consumers will be aware that the partitions are now abandoned (ie, no longer be read from). Without this the consumers would all have to continually check until they consider the partition ownership to be expired, which is time based.

This seems like the right behavior - when we are closed we attempt to do so gracefully.

Reading your issue I'm not sure if I understand what you are trying to accomplish or that the behavior as it is is incorrect. Can you help me understand why you manually delete checkpoints? The primary design feature behind checkpoints is that they are an implementation detail managed by the checkpoint store, which means making manual modifications are not accounted for.

Are checkpoints becoming corrupted?

[marcodalessandro]

Hey @richardpark-msft thanks for the answer :)

We delete the checkpoint mainly to clean up them when we don't need them anymore (i.e.: the eventhub to which they refer to, has been deleted).
In this scenario which actor should be in charge of deleting the checkpoint?
Being several instances of the same microservice running on the same eventhub in a multi processor scenario, and being them stateless, when an eventhub is deleted we would like (if possible) to avoid to orchestratate the deletion of the checkpoint container, that would mean to wait for all the subs to be closed and then delete the container (or using a GC that checks the status every now and then). I hope this helps explain our scenario

[richardpark-msft]

Ah, I see :)

In this case the eventprocessor.stop() is doing the abandon as the last step - it's the last critical piece so it's done after all of message retrieval code is deactivated.

You can see that here

Given that you specifically do not care about updates to the blob store at that point this error is ignorable for you when tearing down.

[richardpark-msft]

I'm going to resolve this issue with a doc update - it seems worth mentioning that .close() of the subscription does update remote state to indicate that it's no longer consuming from a partition, which can be useful to know.