-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TestJemalloc in pkg/server/status failed on Power Linux (ppc64le) platform #43522
Comments
We don't officially support PPC architecture. Not sure which option is correct. Maybe @petermattis has an opinion or can further route. |
Thanks for the report, @amitsadaphule. Did you debugging and research indicate why this problem exists on PPC, but not on amd64? The allocation and call to Disabling the thread cache sounds like a non-starter for performance. |
Thanks @jordanlewis for routing the question towards @petermattis. Thanks for your input @petermattis.
Thanks for confirming that --disable-tcache is not recommended since it'll cause performance issues. |
@amitsadaphule The test is simply verifying that the jemalloc stats are changing based on allocations. I'd be fine with sprinkling a |
@petermattis Thanks for clarifying. I'll try bumping up the constant in |
@petermattis The test passes if I allocate 256KB. I'll use that value for this test on PPC. Thanks for your help on this! |
46255: server: fix misleading TestJemalloc result for ppc64le r=petermattis a=amitsadaphule Fixes #43522 The test failed on ppc64le since the 16KiB constant in allocateMemory didn't exceed the thread cache on ppc64le. Brief failure log shown below: ``` === RUN TestJemalloc --- FAIL: TestJemalloc (0.00s) jemalloc_test.go:37: allocated stat not incremented on allocation: 2322976 ``` Configuring jemalloc with `--disable-tcache to disable the thread cache` or flushing thread cache as `mallctl("thread.tcache.flush", NULL, NULL, NULL, 0);` showed expected behavior. But had performance and code clutter trade-offs resp. So, updated the constant to suit ppc64le which also works for x86 and others. Release justification: Fixes misleading test result for jemalloc stat on PPC Release note: None Co-authored-by: Amit Sadaphule <[email protected]>
Description
I built the cockroachdb code (commit 319080c) on ppc64le (with a minor change necessary in the rocksdb makefile to suppress -march option usage in compilation) and when I executed the test cases in pkg/server/status, I found a test failure there.
To Reproduce
Here is the command I used to execute this particular test:
make test PKG=./pkg/server/status TESTFLAGS='-v -count=1'
And here is the log:
However, when I executed the same test on x86_64, the test did PASS.
Additional data
I tried debugging and researching the issue to find the root cause. The reason was the stats were not being updated after allocation by jemalloc package. When I dug deeper and did some research, I found that there were 2 possible solutions to the issue:
mallctl("thread.tcache.flush", NULL, NULL, NULL, 0);
Option #1 fixes the test failure, but the other part of the code (if it is not flushing thread cache) will still suffer from the issue.
Option #2 fixes the root cause, but may cause performance overhead as it does introduce syncronization in allocation requests.
Please suggest which of these options to use. OR a different solution if neither of these is the right way to fix the issue.
Environment:
The text was updated successfully, but these errors were encountered: