opt: share catalog objects between connections #40922
Labels
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-sql-queries
SQL Queries Team
Milestone
Comments
RaduBerinde
added a commit
to RaduBerinde/cockroach
that referenced
this issue
Sep 30, 2019
Currently the stats cache exposes the encoded histogram data, and the opt catalog decodes it. The problem is that opt catalog objects are not shared across connections so this is inefficient (especially in memory usage). This change moves the decoding into the stats cache. The opt catalog objects now simply point to the data computed by the cache. There are still inefficiencies to improve on in the future: the opt catalog can hold on to multiple versions of tables, so we will keep many versions of histograms "alive". Informs cockroachdb#41206. Informs cockroachdb#40922. Release justification: fix for new functionality. Release note: None
craig bot
pushed a commit
that referenced
this issue
Oct 1, 2019
41220: sql: move histogram decoding into the stats cache r=RaduBerinde a=RaduBerinde Currently the stats cache exposes the encoded histogram data, and the opt catalog decodes it. The problem is that opt catalog objects are not shared across connections so this is inefficient (especially in memory usage). This change moves the decoding into the stats cache. The opt catalog objects now simply point to the data computed by the cache. There are still inefficiencies to improve on in the future: the opt catalog can hold on to multiple versions of tables, so we will keep many versions of histograms "alive". Informs #41206. Informs #40922. Release justification: fix for new functionality. Release note: None 41231: util/log: ensure that secondary loggers do not leak memory r=petermattis,nvanbenschoten a=knz Fixes #41230. Note: the primary cause of this issue is removed by #40993 but that PR is blocked until 19.2 is out. I'm cherry-picking the subset of those changes sufficient to solve issue #41230, here. Prior to this patch, logging via a secondary logger would allocate a buffer, then add it to the buffer free list of the secondary logger. This was causing a memory leak because only the free list from the main logger is used to allocate buffers (even in secondary loggers), so all the now-unused buffers from secondary logs would remain unused and accumulate, locked from Go's GC attention because they are referenced somewhere. Release justification: bug fix Release note (bug fix): A memory leak was fixed that affected secondary logging (SQL audit logs, statement execution, and RocksDB logging). Co-authored-by: Radu Berinde <[email protected]> Co-authored-by: Raphael 'kena' Poss <[email protected]>
|
We have marked this issue as stale because it has been inactive for |
|
We have marked this issue as stale because it has been inactive for |
yuzefovich
added
the
C-enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
#40566 shows a case where we are using a lot of memory for the opt catalog objects. That was a case with 10,000 connections, each having their own object. We should add a per-server cache so we can reuse the objects between connections.
Jira issue: CRDB-5500
The text was updated successfully, but these errors were encountered: