Collect history of responses if redirects occur.

[nitori]

Hi,

Accessing the history, or the information that a redirect occured at all (except for comparing original and final URL), is something I couldn't find and was missing from the requests library.

I'm completely new to this project, so I have no idea, if what I did, is the correct way to do it or if I missed something.

Feedback appreciated.

[asvetlov]

I support the PR but tests and documentation update should be added.
Would you manage the request up to be ready for merging?

[nitori]

I'll ... try? (I have never really done these things)

[asvetlov]

Please do. I'll help you.
Tests first.
Add new test to https://github.com/KeepSafe/aiohttp/blob/master/tests/test_client_functional.py
Create a test which has two server handlers: one for redirection and second for returning 200 OK response.
Check for history attribute of given client response.

[nitori]

Oh, I didn't see your comment until after I pushed the update. :-)

[asvetlov]

Github doesn't send email notification on pushing.
So please add a comment if you need feedback.

[asvetlov]

:class:list`` is even better

[nitori]

Instead of "List" in "List of :class:ClientResponse objects …"?

Or where exactly do you mean?

[asvetlov]

I mean

:class:`list` of :class:`ClientResponse` objects of preceding requests ...

[nitori]

Okay, I added a test.

Edit: maybe add try/finally block around the asserts to release resp/resp_redirect?

[asvetlov]

Perfect!

maybe add try/finally block around the asserts to release resp/resp_redirect?

Please do.

[nitori]

Done :-)

[asvetlov]

Let me sleep on it and merge tomorrow.

[popravich]

Hi,
The idea is good but there is an issue -- you simply collect responses into history list and then provide it with last non-redirecting response, howerer those responses (in the history) can not be accessed in full manner -- if someone try something like he will end with exception:

yield from resp.history[0].text()

[asvetlov]

Yes, all responses except the latest are released.
What do you suggest?
Requests has the same attribute IIRC.

[popravich]

I'm not quite sure what to do...
resp.read() / json() / text() coroutines use content attribute which is instance of some stream parser bound to connection.reader directly. And it all goes deeper into Connector/Connection stuff...
Need to look into this...

[asvetlov]

Good catch.
We may do better cleanup in .close()/.release()

Converted .history into readonly property by eac7028 BTW

[nitori]

If anything, I believe discarding the body and getting an exception when trying to access it should be the default behavior.

I assume that the common case is, that you're not even interested in the body of preceding requests (if anything, you're probably more interested in the headers or the URLs - as in my case), and in most cases, it doesn't contain any valuable information anyway.

So if you happen to be interested in the body of those requests, there should either be an option to completely load them or to somehow keep the connection opened, or to use allow_redirects=False and go through the steps yourself.

[asvetlov]

If anything, I believe discarding the body and getting an exception when trying to access it should be the default behavior.

Would you propose a patch?

[nitori]

Hm, I'd have to take a closer look at how things work, but I would like to give it a try.

As far as I can tell, awaiting .text() doesn't actually raise any exception but just returns an empty string on released or closed responses. Changing that would probably cause some applications, that rely on that, to break, and I don't know if it's worth to add an exemption just for redirects.

I assumed it already raises some kind of exception - so I'm not sure after all, if it's a good idea.

[asvetlov]

I don't care about .text() right now but .content should be protected from further usage.

[nitori]

We might as well just close the preceding responses instead of releasing them. If we do that, trying to read from .content will cause an aiohttp.errors.ServerDisconnectedError.

[asvetlov]

No, response releasing is crucial for HTTP keep-alive support.
.close() call breaks connection and closes socket but .release() allows to reuse the socket for next request.

Replacing .content with something like EmptyStream (https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/streams.py#L332) but raises RuntimeError on every read call may prevent reading from history responses.

Concurrent reading from different contents (bound with the same underlying transport) make a mess as @popravich mentioned.

Honestly using just EmptyStream is an option also: redirection responses stored in history should have no HTTP BODY anyway.

Did you get my point?

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.