Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KYUUBI #5832] Always perform closing action in OperationLog to avoid fd leak #5854

Closed
wants to merge 1 commit into from

Conversation

Flyangz
Copy link
Contributor

@Flyangz Flyangz commented Dec 14, 2023

🔍 Description

Issue References 🔗

This pull request fixes #5832

Describe Your Solution 🔧

We should always close writer regardless of whether OperationLog is initialized or not, to prevent other threads from continuing to write.

Types of changes 🔖

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests


Checklists

📝 Author Self Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • This patch was not authored or co-authored using Generative Tooling

📝 Committer Pre-Merge Checklist

  • Pull request title is okay.
  • No license issues.
  • Milestone correctly set?
  • Test coverage is ok
  • Assignees are selected.
  • Minimum number of approvals
  • No changes are requested

Be nice. Be informative.

@Flyangz
Copy link
Contributor Author

Flyangz commented Dec 14, 2023

@pan3793 Would you mind reviewing this pr or do you know who can help?

@bowenliang123
Copy link
Contributor

LGTM overall.
Could you attach some screenshots as evidence showing the fd leaks properly fixed by this PR? Eg. comparing opened fd numbers, or the server logs.

@pan3793
Copy link
Member

pan3793 commented Dec 14, 2023

would you like to try 1.7.3 or 1.8.0? this issue looks like was solved by #5211

@Flyangz
Copy link
Contributor Author

Flyangz commented Dec 14, 2023

would you like to try 1.7.3 or 1.8.0? this issue looks like was solved by #5211

@pan3793 My online test in issue #5832 had already included this #5211 patch, but the operation logs still leaked.

@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (4c029f9) 61.23% compared to head (2549928) 61.21%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #5854      +/-   ##
============================================
- Coverage     61.23%   61.21%   -0.03%     
  Complexity       23       23              
============================================
  Files           609      609              
  Lines         36281    36280       -1     
  Branches       4976     4976              
============================================
- Hits          22217    22207      -10     
- Misses        11669    11673       +4     
- Partials       2395     2400       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pan3793 pan3793 changed the title [BUG] fix fd leak when closing KyuubiSession but OperationLog not initialized [KYUUBI #5832] Always perform closing action in OperationLog to avoid fd leak Dec 14, 2023
Copy link
Member

@pan3793 pan3793 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point, the change makes sense to me. would be better to attach your production testing report to prove the patch takes effect as @bowenliang123 suggested.

@Flyangz
Copy link
Contributor Author

Flyangz commented Dec 14, 2023

Ok, I'll provide it as soon as possible. But I am busy with another more important project recently, and the upgrade of Kyuubi has been stoped. It will probably take a week or two before I can get back to it.

@bowenliang123
Copy link
Contributor

bowenliang123 commented Dec 14, 2023

Looking forward to your feedback and verification.

@pan3793 pan3793 added this to the v1.7.4 milestone Dec 15, 2023
@pan3793
Copy link
Member

pan3793 commented Dec 15, 2023

Anyway, this fix looks good, let me merge it first, looking forward to your testing result once you have time @Flyangz

@pan3793 pan3793 closed this in 64ee629 Dec 15, 2023
pan3793 pushed a commit that referenced this pull request Dec 15, 2023
… fd leak

# 🔍 Description
## Issue References 🔗

This pull request fixes #5832

## Describe Your Solution 🔧

We should always close writer regardless of whether OperationLog is initialized or not, to prevent other threads from continuing to write.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5854 from Flyangz/fix-op-log-leak.

Closes #5832

2549928 [liuyang] fix fd leak when closing KyuubiSession but OperationLog not initialized

Authored-by: liuyang <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 64ee629)
Signed-off-by: Cheng Pan <[email protected]>
pan3793 pushed a commit that referenced this pull request Dec 15, 2023
… fd leak

# 🔍 Description
## Issue References 🔗

This pull request fixes #5832

## Describe Your Solution 🔧

We should always close writer regardless of whether OperationLog is initialized or not, to prevent other threads from continuing to write.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5854 from Flyangz/fix-op-log-leak.

Closes #5832

2549928 [liuyang] fix fd leak when closing KyuubiSession but OperationLog not initialized

Authored-by: liuyang <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 64ee629)
Signed-off-by: Cheng Pan <[email protected]>
@pan3793
Copy link
Member

pan3793 commented Dec 15, 2023

Merged to master/1.8/1.7

zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this pull request Mar 21, 2024
… avoid fd leak

# 🔍 Description
## Issue References 🔗

This pull request fixes apache#5832

## Describe Your Solution 🔧

We should always close writer regardless of whether OperationLog is initialized or not, to prevent other threads from continuing to write.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes apache#5854 from Flyangz/fix-op-log-leak.

Closes apache#5832

2549928 [liuyang] fix fd leak when closing KyuubiSession but OperationLog not initialized

Authored-by: liuyang <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
beryllw pushed a commit to beryllw/incubator-kyuubi that referenced this pull request Jun 7, 2024
… avoid fd leak

# 🔍 Description
## Issue References 🔗

This pull request fixes apache#5832

## Describe Your Solution 🔧

We should always close writer regardless of whether OperationLog is initialized or not, to prevent other threads from continuing to write.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes apache#5854 from Flyangz/fix-op-log-leak.

Closes apache#5832

2549928 [liuyang] fix fd leak when closing KyuubiSession but OperationLog not initialized

Authored-by: liuyang <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 64ee629)
Signed-off-by: Cheng Pan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Close KyuubiSession when OperationLog not initialized may lead log file leak
5 participants