Is there a limit for size of HTTP request when using output-http?

[Kooper]

Hello,

Is there a limit on size of HTTP POST requests being proxied from tcp input to http output?

Using gor 0.9.4 I observed HTTP requests sent to destination are cut at 7884 byte.
After switching to gor 0.9.6 I see that HTTP requests sent to destination cut at 21623 byte.

Is it possible to increase the allowed size of POST? I actually have to proxy POST requests up to 1M.

[buger]

Well, actually it should be 5mb in both versions, but i feel like it do not intercept all the packets. This limit numbers you shown apply to all POST requests, or it is more sort of random limit?

[buger]

I added more debugging to 0.9.7 version, see release notes. It will be useful if you can check request headers, if it is Expect:100 continue or Transfer-Encoding:chunked request (better to send here all headers).

[buger]

Performance of raw input should improve after #170

After my initial testing, i confirm that for payloads large then 200kb change that both will be incomplete is very high. I'm not sure what is the real solution there, just not pass such requests at all, or continue send uncomplete post request, so you server at least can test this testcase :)

[buger]

You can try to play with params in "Tuning" section of README.

[Kooper]

With recent releases of Gor (e.g. 0.10.1) I see a drop in number of incomplete messages received by destination server. So it seems like this is more a performance issue.

I'm closing this bug as there is indeed no any size limit on requests, and incomplete requests were most probably caused by high server load. Recent Gor releases features much less CPU burden so I assume the issue fixed.

[buger]

I think improvement will be much better with Go 1.5 , since Gor is very GC
sensitive app.

On Wednesday, September 9, 2015, Victor Kupriyanov [email protected]
wrote:

Closed #167 #167.

—
Reply to this email directly or view it on GitHub
#167 (comment).

Sincerely yours Leonid Bugaev
[email protected]
http:// http://buger.github.comleonsbox.com

[Kooper]

Hmmm.. does it mean now it is built with Go 1.4 and it makes sense for me to rebuild it with 1.5?

I'm asking because I still have a slow memory leak (100 MB/day) - probably GC improvements in 1.5 can fix it?

UPD: actually gor process have grown to 600 MB in 6 days

[buger]

You can try for sure. Also remind me what input/output do you use, I will
check leak one more time.

On Wednesday, September 9, 2015, Victor Kupriyanov [email protected]
wrote:

Hmmm.. does it mean now it is built with Go 1.4 and it makes sense for me
to rebuild it with 1.5?

I'm asking because I still have a slow memory leak (10 MB/day) - probably
GC improvements in 1.5 can fix it?

—
Reply to this email directly or view it on GitHub
#167 (comment).

Sincerely yours Leonid Bugaev
[email protected]
http:// http://buger.github.comleonsbox.com

[Kooper]

gor --input-raw :8080 --output-http x.x.x.x:80

Will it be helpful to run with '-memprofile' option and provide you with output?

[buger]

Yes, It will help a lot. It will generate file which you need to send me.

On Wednesday, September 9, 2015, Victor Kupriyanov [email protected]
wrote:

gor --input-raw :8080 --output-http x.x.x.x:80

Will it be helpful to run with '-memprofile' option and provide you with
output?

—
Reply to this email directly or view it on GitHub
#167 (comment).

Sincerely yours Leonid Bugaev
[email protected]
http:// http://buger.github.comleonsbox.com

[Kooper]

Ok, I will run the 0.10.1 release executable (64 bit) with -memprofile and attach results here.

[Kooper]

So, I started the utility as "gor --input-raw :8080 --output-http x.x.x.x:80 -memprofile /tmp/gor-memprofile" and it created /tmp/gor-memprofile after something near 10 minutes of running. The generated file content is available here: http://pastebin.com/D2GvW9ZX

At this moment gor process occupied near 70MB of memory.

The file /tmp/gor-memprofile wasn't updated since this time, although gor is running for 3 hours now and occupies 103MB.

There are two error messages from the command:

2015/09/09 12:04:56 PANIC: pkg: runtime error: slice bounds out of range /gopath/src/github.com/buger/gor/raw_socket_listener/listener.go:199 (0x5969ee)
/goroot/src/runtime/asm_amd64.s:401 (0x454395)
/goroot/src/runtime/panic.go:387 (0x42ba78)
/goroot/src/runtime/panic.go:18 (0x42ac5e)
/gopath/src/github.com/buger/gor/raw_socket_listener/listener.go:237 (0x59425e)
/gopath/src/github.com/buger/gor/raw_socket_listener/listener.go:100 (0x592b41)
/goroot/src/runtime/asm_amd64.s:2232 (0x4564b1)

2015/09/09 13:55:52 PANIC: pkg: runtime error: slice bounds out of range /gopath/src/github.com/buger/gor/raw_socket_listener/listener.go:199 (0x5969ee)
/goroot/src/runtime/asm_amd64.s:401 (0x454395)
/goroot/src/runtime/panic.go:387 (0x42ba78)
/goroot/src/runtime/panic.go:18 (0x42ac5e)
/gopath/src/github.com/buger/gor/raw_socket_listener/listener.go:237 (0x59425e)
/gopath/src/github.com/buger/gor/raw_socket_listener/listener.go:100 (0x592b41)
/goroot/src/runtime/asm_amd64.s:2232 (0x4564b1)

I'm not sure, but I noticed memory usage increase on 10MB simultaneously with the first error message. The second message I missed.