Further optimize memory for Travis

[jihoonson]

Partially fixes #6013.

Our Travis CI is frequently failing and one of the common reasons is the oom reaper killed the unit test process. I found each Travis job needs different amount of heap and off-heap memory, so they should be optimized differently. Also, I'm thinking to add closeable blocking/non-blocking pools for only unit tests to release buffers as early as possible. I'll raise a PR soon.