[Bug] Memory leak caused by UserGroupInformation.createProxyUser #772
Comments
|
cc @advancedxy @jerqi @smallzhongfeng This is a critical bug, I will fix this ASSP. |
zuston
added a commit
to zuston/incubator-uniffle
that referenced
this issue
Mar 28, 2023
|
Do you any any heap dump to indicate this memory leak. I just checked the related code, the proxy user created from the same ugi should equal to the same proxy user? |
After dumping all heap, this bug was found.
I have not get your point. |
Is it possible for you to upload the screenshot here?
I mean that two proxy ugi for the same proxy user should be equal to each other, that is what I have checked for. |
|
|
zuston
added a commit
that referenced
this issue
Mar 29, 2023
### What changes were proposed in this pull request? 1. To avoid memory leak by caching of proxy user UGI. ### Why are the changes needed? Fix: #772 The Hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak. As we know, the filesystem cache's key is built by the scheme、authority and UGI. The scheme and authority are not changed every time. But for UGI, if we invoke the createProxyUser, it will always create a new one, that means the every invoking `Filesystem.get()`, it will be cached due to different key. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? 1. Existing UTs 2. Added tests
xianjingfeng
pushed a commit
to xianjingfeng/incubator-uniffle
that referenced
this issue
Apr 5, 2023
…apache#773) ### What changes were proposed in this pull request? 1. To avoid memory leak by caching of proxy user UGI. ### Why are the changes needed? Fix: apache#772 The Hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak. As we know, the filesystem cache's key is built by the scheme、authority and UGI. The scheme and authority are not changed every time. But for UGI, if we invoke the createProxyUser, it will always create a new one, that means the every invoking `Filesystem.get()`, it will be cached due to different key. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? 1. Existing UTs 2. Added tests
jerqi
pushed a commit
that referenced
this issue
Apr 13, 2023
### What changes were proposed in this pull request? 1. To avoid memory leak by caching of proxy user UGI. ### Why are the changes needed? Fix: #772 The Hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak. As we know, the filesystem cache's key is built by the scheme、authority and UGI. The scheme and authority are not changed every time. But for UGI, if we invoke the createProxyUser, it will always create a new one, that means the every invoking `Filesystem.get()`, it will be cached due to different key. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? 1. Existing UTs 2. Added tests
jerqi
added a commit
that referenced
this issue
Apr 13, 2023
zuston
added a commit
that referenced
this issue
Apr 17, 2023
### What changes were proposed in this pull request? 1. To avoid memory leak by caching of proxy user UGI. ### Why are the changes needed? Fix: #772 The Hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak. As we know, the filesystem cache's key is built by the scheme、authority and UGI. The scheme and authority are not changed every time. But for UGI, if we invoke the createProxyUser, it will always create a new one, that means the every invoking `Filesystem.get()`, it will be cached due to different key. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? 1. Existing UTs 2. Added tests
jerqi
added a commit
that referenced
this issue
Apr 17, 2023
zuston
added a commit
to zuston/incubator-uniffle
that referenced
this issue
Apr 17, 2023
…apache#773) 1. To avoid memory leak by caching of proxy user UGI. Fix: apache#772 The Hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak. As we know, the filesystem cache's key is built by the scheme、authority and UGI. The scheme and authority are not changed every time. But for UGI, if we invoke the createProxyUser, it will always create a new one, that means the every invoking `Filesystem.get()`, it will be cached due to different key. No. 1. Existing UTs 2. Added tests
zuston
added a commit
that referenced
this issue
Apr 17, 2023
…773) (#824) ### What changes were proposed in this pull request? 1. To avoid memory leak by caching of proxy user UGI. ### Why are the changes needed? Fix: #772 The Hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak. As we know, the filesystem cache's key is built by the scheme、authority and UGI. The scheme and authority are not changed every time. But for UGI, if we invoke the createProxyUser, it will always create a new one, that means the every invoking `Filesystem.get()`, it will be cached due to different key. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? 1. Existing UTs 2. Added tests
xianjingfeng
pushed a commit
to xianjingfeng/incubator-uniffle
that referenced
this issue
Jun 20, 2023
…ory leak (apache#773)" This reverts commit 89c2b92.
xianjingfeng
pushed a commit
to xianjingfeng/incubator-uniffle
that referenced
this issue
Jun 20, 2023
…apache#773) 1. To avoid memory leak by caching of proxy user UGI. Fix: apache#772 The Hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak. As we know, the filesystem cache's key is built by the scheme、authority and UGI. The scheme and authority are not changed every time. But for UGI, if we invoke the createProxyUser, it will always create a new one, that means the every invoking `Filesystem.get()`, it will be cached due to different key. No. 1. Existing UTs 2. Added tests
xianjingfeng
pushed a commit
to xianjingfeng/incubator-uniffle
that referenced
this issue
Jun 20, 2023
…ory leak (apache#773)" This reverts commit 42d9b0a.
xianjingfeng
pushed a commit
to xianjingfeng/incubator-uniffle
that referenced
this issue
Jun 20, 2023
… leak (apache#773) (apache#824) 1. To avoid memory leak by caching of proxy user UGI. Fix: apache#772 The Hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak. As we know, the filesystem cache's key is built by the scheme、authority and UGI. The scheme and authority are not changed every time. But for UGI, if we invoke the createProxyUser, it will always create a new one, that means the every invoking `Filesystem.get()`, it will be cached due to different key. No. 1. Existing UTs 2. Added tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code of Conduct
Search before asking
Describe the bug
The hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak.
As we know, the filesystem cache's key is built by the scheme、authority and ugi. The scheme and authority are not changed everytime. But for ugi, if we invoke the createProxyUser, it will always create a new one, that means the every invoking
Filesystem.get(), it will be cached due to different key.We should cache the proxy user ugi to avoid caching every time.
Affects Version(s)
master
Uniffle Server Log Output
No response
Uniffle Engine Log Output
No response
Uniffle Server Configurations
No response
Uniffle Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: