Fix a timing issue in the waiters retry delay calculation

Fix a timing issue in the waiters retry delay calculation #4391