JWT Auth Expiring Large Derivative Queue

[jonathangreen]

This issue is related to:
#1030
#116

This ticket comes out of the discussion from the February 27 CLAW call.

#1030 deals with tokens that are expired being sent from flysystem, but it is also possible that tokens will expire while the derivatives are sitting in the camel queue. With large batch ingests items could sit in the queue for a long time waiting to be processed. In 7.x installs I've seen items in the queue for days to weeks waiting during large batch ingests.

This necessitates some way to either have credentials that we can use that don't expire, or can be renewed. We could also tell people to guess and set their expiry longer then they need, but this seems like a sort of hackish solution to me.

Some possible solutions:

• Don't set the expiry at all, so the JWT never expires
• Implement some sort of JWT token refresh mechanism
  • https://auth0.com/docs/tokens/refresh-token/current
  • https://stackoverflow.com/questions/26739167/jwt-json-web-token-automatic-prolongation-of-expiration
• Stop using JWT and use some sort of API key / password instead

[whikloj]

This was partially resolved with the linked issues and PRs, but the current solution only works so long as you know the longest run time before you start. We will need a better asynchronous solution.