Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: possible memory leak in Streamer #76471

Closed
nvanbenschoten opened this issue Feb 12, 2022 · 1 comment · Fixed by #76480
Closed

sql: possible memory leak in Streamer #76471

nvanbenschoten opened this issue Feb 12, 2022 · 1 comment · Fixed by #76480
Assignees
Labels
A-sql-execution Relating to SQL execution. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-investigation Further steps needed to qualify. C-label will change. T-sql-queries SQL Queries Team

Comments

@nvanbenschoten
Copy link
Member

In each call to NewStreamer, we register an OnChange handler on the streamerConcurrencyLimit:

streamerConcurrencyLimit.SetOnChange(&st.SV, func(ctx context.Context) {
s.coordinator.asyncSem.UpdateCapacity(uint64(streamerConcurrencyLimit.Get(&st.SV)))
})

This handler captures the Streamer object s. The handler is then never unregistered. I believe this leaks the Streamer object and all references. This is a problem because, unlike most static objects that are leaked due to calls to SetOnChange, we create a new Streamer for each SQL query that uses one.

Two ways to address this are to:

  1. unregister the OnChange handler when tearing down the Streamer. We don't currently have a way to unregister an OnChange handler, but adding one should not be difficult.
  2. skip the OnChange handler. Do we need a single instance of a Streamer to react dynamically to cluster setting changes? Is it enough for new instances to use the new limit?

Note: I haven't actually confirmed that this leaks memory, so determining whether it's real would be the first step here.

@nvanbenschoten nvanbenschoten added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-investigation Further steps needed to qualify. C-label will change. A-sql-execution Relating to SQL execution. labels Feb 12, 2022
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Feb 12, 2022
@yuzefovich
Copy link
Member

yuzefovich commented Feb 13, 2022

Thanks Nathan for spotting this! I believe you're correct that we have a memory leak (although it's not huge because Streamer objects are nil-ed out in Streamer.Close, so we're only leaking the object itself and nothing else). I agree with your point that we don't need to dynamically react to a change in the cluster setting, so I'm removing that in #76480.

@craig craig bot closed this as completed in 07a3683 Feb 14, 2022
@mgartner mgartner moved this to Done in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-sql-execution Relating to SQL execution. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-investigation Further steps needed to qualify. C-label will change. T-sql-queries SQL Queries Team
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants