Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to handle colossal sized response headers #388

Open
craigreyenga opened this issue Jun 15, 2024 · 4 comments
Open

Unable to handle colossal sized response headers #388

craigreyenga opened this issue Jun 15, 2024 · 4 comments
Labels
question Further information is requested

Comments

@craigreyenga
Copy link

My website has a link to TikTok on it. The HTTP response headers from TikTok are very big, which causes Muffet to fail.

	error when reading response headers: small read buffer. Increase ReadBufferSize. Buffer size=4096, contents: "HTTP/1.1 302 Moved Temporarily\r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: 47\r\nContent-Security-Policy: script-src 'unsafe-eval' sf16-website-login.neutral.ttwstatic.com s20.tiktokcdn.com"..."tiktok-row.net *.tiktok.com *.tiktok.ru *.tiktok.vn *.tiktokapis.com *.tiktokcdn-eu.com *.tiktokcdn-in.com *.tiktokcdn-us.com *.tiktokcdn.com *.tiktokcreativeone.com *.tiktokforbusinessoutbound.com *."	https://www.tiktok.com/

Pointing to the following HTML is enough to trigger the issue:

<!DOCTYPE html>
<html lang="en">
<head></head>
<body>
    <a href="https://www.tiktok.com/">TikTok</a>
</body>
</html>

Using curl -vvvv https://www.tiktok.com/ shows that the response headers are very large on TikTok.
@raviqqe
Copy link
Owner

raviqqe commented Jun 15, 2024

Did you try the buffer size option?

@craigreyenga
Copy link
Author

-b 8192: bad
-b 9216: good

I did not notice this option when I first started using muffet. Does this option refer to the total size of all HTTP headers in the response? I assume the body of the response is not a part of this buffer size.

Would it be possible to increase the default to 10 kB?

@raviqqe
Copy link
Owner

raviqqe commented Jun 26, 2024

The buffer size is used for every connection. So it's not something easy to modify as it's critical to the entire process's memory usage... 🤔

@raviqqe raviqqe added the question Further information is requested label Jun 26, 2024
@craigreyenga
Copy link
Author

The buffer size is used for every connection. So it's not something easy to modify as it's critical to the entire process's memory usage... 🤔

Does that memory usage accumulate as more and more requests are made?

Anyway, I will defer to your judgment. It's not a huge problem for me to add that parameter to each invocation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@craigreyenga @raviqqe