Eventhub Blob CheckpointStore - Server failed to authenticate the request "Request date header too old"

[noirbee]

• Package Name: azure-eventhub-checkpointstoreblob
• Package Version: 1.1.1
• Operating System: Debian 10 "buster"
• Python Version: 3.7.3

Describe the bug
This is essentially (and intentionally) a duplicate of issue #14067, applied to the Blob Storage layer used for checkpointing in the eventhub client. Here's an example of the error we see in our logs:

An exception occurred during list_ownership for namespace '****' eventhub '****' consumer group '****'. Exception is ClientAuthenticationError("Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:fb6cef5b-001e-0076-6692-e88e95000000
Time:2021-01-12T03:23:55.3383493Z
ErrorCode:AuthenticationFailed
Error:None
AuthenticationErrorDetail:Request date header too old: 'Tue, 12 Jan 2021 03:08:02 GMT'")

I looked at updating the policies list in sdk/eventhub/azure-eventhub-checkpointstoreblob/azure/eventhub/extensions/checkpointstoreblob/_vendor/storage/blob/_shared/base_client.py (and base_client_async.py) in the same way as in #14629 and opening a PR, but unfortunately I can't even get the tests to run using tox, much less write one :(

To Reproduce
We have trouble to reproduce it ourselves as the error only shows on one of our environments (with almost non-existent load).
I expect the steps in #14067 would yield the same results.

Expected behavior
The Request date header too old exceptions should not be thrown.

[yunhaoling]

hey @noirbee , thanks for bring this to my attention.

Based on what you said, this is a known issue in the storage sdk and they have already get it fixed.
Currently the storage sdk code is vendored in the eventhub sdk, so I think we need to update the vendored storage code to accommodate the change.

You mentioned that "opening a PR", do you have link to your opened PR? I would be great if we could collaborate on fixing this :)

[yunhaoling]

hey @noirbee , I have updated the vendor storage blob in this PR: #16559

We'll do a bug fix release next month, thanks for your patience!

[yunhaoling]

hey @noirbee , we have released a patch version of azure-eventhub-checkpointstore(-aio) with the updated vendored storage sdk. please upgrade to the latest version to try out the fix.

I'm closing the issue now assuming the issue is fixed within the latest release, but don't hesitate to reopen if you're still meeting with troubles.
thanks for you patience!

pypi:
https://pypi.org/project/azure-eventhub-checkpointstoreblob/1.1.3/
https://pypi.org/project/azure-eventhub-checkpointstoreblob-aio/1.1.3/