HTTPoison.get! follow_redirect does not work with an async request

[mplubratt]

I'm writing a streaming processor for some large files in a Hadoop cluster. The files are too big to keep in memory, so I want to use async get! requests into a Stream.resource. The Hadoop cluster will provide a redirect from the name node to the cluster node with the file actually on it. However, follow_redirect doesn't seem to work with an async request.

If I do an async request without follow_redirect, I get back a %HTTPoison.AsyncStatus, %HTTPoison.AsyncHeaders and %HTTPoison.AsyncEnd messages showing the 307 status along with the redirect "Location" in the headers.

If I do an async request with follow_redirect, I only get a %HTTPoison.AsyncRedirect. There is no %HTTPoison.AsyncStatus message with a 307, or a %HTTPoison.AsyncEnd, only the redirect. There are no other messages in the mailbox, and the request doesn't follow the redirect even though it's a GET, and the status is 307.

[mplubratt]

Buried in the hackney documentation, I found a single comment that follow_redirects are only followed when the response is received synchronously.

line 103-104
https://github.com/benoitc/hackney/blob/3e2b578d5d42534cc57f43749af6e8016f8e80fb/src/hackney.erl

Since this seems to be a hackney bug/"feature", my work around is to synchronously check the header for a 307 status first. If it is, follow the Location field in the header.

Documentation should probably be updated to reflect the hackney comment.