#6336 causes overlord console page loading to be extreme slow

[QiuMM]

As discussed in #6336 (comment), I think we should revert #6336 first and find a better way to complete #6336. How do you think @FaxianZhao @jihoonson.

[vogievetsky]

Is this still an issue in the latest code?

[QiuMM]

Yes, it is. I have made some improvement in my druid code, I'll open a PR to patch this when I'm not busy.

[FaxianZhao]

I suggest use the Fine-Grained Lock resolve this kind issues.
I think my previous draft code won't cause this, because I use synchronized (lockResult.getTaskLock()) instead of toolbox.getTaskLockbox().doInCriticalSection, but it looks bad for committed code.

I think we could push down lock action to the TaskLock level. Giant lock is used to approve TaskLock, and TaskLock is work for the real business. We could implement TaskLockType with ReadWriteLock, but I haven't go through the whole picture.

[jihoonson]

@QiuMM thanks. I'm looking forward to your PR.

@FaxianZhao yeah, fine-grained locking would mitigate this issue. But, synchronizing taskLock isn't enough to fix the issue of #6336, because taskLock is shared by the tasks of the same groupId but the tasks of different groupIds should be able to request segment allocation for the same dataSource and interval at the same time. Check out #7491.

I think the best way to handle this is synchronizing segmentId allocation per dataSource per interval as @QiuMM did here.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the [email protected] list.
Thank you for your contributions.