Move credential and header policies after retry policy

[tasherif-msft]

This PR resolves #14701 and resolves #14067.

Case 1:
This case is related to the StorageHeadersPolicy. In the header policy the current_time is set. This policy is currently ran before the retry policy. This means that if the retry policy was retrying and the time period has exceeded 15 min difference between the pre-established current_time it would fail as it would be outdated.

Solution Move the headers policy after RetryPolicy, i.e, current time would get reset with every retry.

Case 2:
This case is related to the StorageCredentialPolicy, moving it after the retry policy means that the TokenCredential can be refreshed if it has expired while retrying.

[tasherif-msft]

/azp run python - storage - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tasherif-msft]

/azp run python - storage - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rakshith91]

lgtm - can you please associate this with an issue

[kasobol-msft]

Could you please add a test for that change?

[tasherif-msft]

Could you please add a test for that change?

It's going to be a little difficult to test these changes, I think the best way to test it is running the live tests pipeline

[tasherif-msft]

/azp run python - storage - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[kasobol-msft]

Could you please add a test for that change?

It's going to be a little difficult to test these changes, I think the best way to test it is running the live tests pipeline

One idea is to inject a pipeline policy that simulates 5xx error. I know we do it in .NET and Java, in Python the closest thing I recall is this

azure-sdk-for-python/sdk/storage/azure-storage-blob/tests/test_largest_block_blob.py

Line 302 in 48ff819

class PayloadDroppingPolicy(HTTPPolicy):

(for different purpose but I hope it gives some idea).
Then if any of our auth policies supports credential rotation then we could rotate credential and assert that retry went with new cred.

Example in .NET https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/storage/Azure.Storage.Common/tests/Shared/FaultyDownloadPipelinePolicy.cs