-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Address concurrency issues in DequeRecycler close #41695
Conversation
This change addresses some concurrency issues that can occur when closing a DequeRecycler. The first issue that is addressed is a ConcurrentModificationException that can occur when using a locked DequeRecycler that is backed by an ArrayDeque. The ArrayDeque is not thread safe and requires external synchronization. In most cases the locked DequeRecycler handles this correctly but closing the DequeRecycler is not protected by the lock, which can lead to the ConcurrentModificationException if other threads are calling the obtain method. Additionally, the DequeRecycler close method used an iterator to go over all entries in the Deque and then later cleared them. The close method now uses pollFirst in a loop to empty the Deque and still execute the destroy method. Finally, the ConcurrentDequeRecycler had an overzealous assertion in the close method that the size of the Deque is equivalent to the externally tracked size. This assertion is not always going to be true due to the nature of the implementation. There is no lock guarding both the deque and the size value, so there is always a chance that the two could be wrong depending on ongoing requests. This assertion has been removed and a comment has been added that mentions there can be some discrepancies between the actual size of the deque and the externally tracked size. These issues may have always been present, but I believe the changes in Closes elastic#41683
Pinging @elastic/es-core-infra |
Thanks @jaymode for looking into it. I think #39317 introduced problems indeed by closing the recycler before making sure that there are no more ongoing tasks in the threadpool. I tried to think about how we could make sure that |
@jpountz that makes sense to me. I'll close this PR and open another to remove the close method |
The changes in elastic#39317 brought to light some concurrency issues in the close method of Recyclers as we do not wait for threads running in the threadpool to be finished prior to the closing of the PageCacheRecycler and the Recyclers that are used internally. elastic#41695 was opened to address the concurrent close issues but upon review, the closing of these classes is not really needed as the instances should be become available for garbage collection once there is no longer a reference to the closed node. Closes elastic#41683
The changes in #39317 brought to light some concurrency issues in the close method of Recyclers as we do not wait for threads running in the threadpool to be finished prior to the closing of the PageCacheRecycler and the Recyclers that are used internally. #41695 was opened to address the concurrent close issues but upon review, the closing of these classes is not really needed as the instances should be become available for garbage collection once there is no longer a reference to the closed node. Closes #41683
The changes in #39317 brought to light some concurrency issues in the close method of Recyclers as we do not wait for threads running in the threadpool to be finished prior to the closing of the PageCacheRecycler and the Recyclers that are used internally. #41695 was opened to address the concurrent close issues but upon review, the closing of these classes is not really needed as the instances should be become available for garbage collection once there is no longer a reference to the closed node. Closes #41683
The changes in elastic#39317 brought to light some concurrency issues in the close method of Recyclers as we do not wait for threads running in the threadpool to be finished prior to the closing of the PageCacheRecycler and the Recyclers that are used internally. elastic#41695 was opened to address the concurrent close issues but upon review, the closing of these classes is not really needed as the instances should be become available for garbage collection once there is no longer a reference to the closed node. Closes elastic#41683
This change addresses some concurrency issues that can occur when
closing a DequeRecycler. The first issue that is addressed is a
ConcurrentModificationException that can occur when using a locked
DequeRecycler that is backed by an ArrayDeque. The ArrayDeque is not
thread safe and requires external synchronization. In most cases the
locked DequeRecycler handles this correctly but closing the
DequeRecycler is not protected by the lock, which can lead to the
ConcurrentModificationException if other threads are calling the obtain
method. Additionally, the DequeRecycler close method used an iterator
to go over all entries in the Deque and then later cleared them. The
close method now uses pollFirst in a loop to empty the Deque and still
execute the destroy method.
Finally, the ConcurrentDequeRecycler had an overzealous assertion in
the close method that the size of the Deque is equivalent to the
externally tracked size. This assertion is not always going to be true
due to the nature of the implementation. There is no lock guarding both
the deque and the size value, so there is always a chance that the two
could be wrong depending on ongoing requests. This assertion has been
removed and a comment has been added that mentions there can be some
discrepancies between the actual size of the deque and the externally
tracked size.
These issues may have always been present, but I believe the changes in
#39317 has made their presence known.
Closes #41683