Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: Set IngestAsFlushable to true by default in the Pebble Options #97194

Closed
bananabrick opened this issue Feb 15, 2023 · 2 comments · Fixed by #98873
Closed

storage: Set IngestAsFlushable to true by default in the Pebble Options #97194

bananabrick opened this issue Feb 15, 2023 · 2 comments · Fixed by #98873
Assignees
@jbowens
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-storage Storage Team

Comments

@bananabrick
Copy link
Contributor

bananabrick commented Feb 15, 2023 •
edited by cockroach-jira-scripts
Loading

We introduced the ability ingest sstables as a flushable in cockroachdb/pebble#2313. Enable this by default, and add a cluster setting to turn it off in the worst case.

We need cockroachdb/pebble#2266, and cockroachdb/pebble#2292 done to turn it on.

Jira issue: CRDB-24567
@bananabrick bananabrick added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Feb 15, 2023
@sumeerbhola sumeerbhola added A-storage Relating to our storage engine (Pebble) on-disk storage. T-storage Storage Team labels Feb 15, 2023
@jbowens jbowens self-assigned this Mar 14, 2023
@bananabrick
Copy link
Contributor Author

Should be enabled by default. Do we just need a cluster setting here?

@jbowens
Copy link
Collaborator

jbowens commented Mar 14, 2023

We need a Cockroach cluster version to ratchet the format major version up to the point that it can be enabled, and then a cluster setting as well.

@jbowens jbowens mentioned this issue Mar 15, 2023
Merged
jbowens added a commit to jbowens/cockroach that referenced this issue Mar 15, 2023
@jbowens
storage: enable flushable ingestion
d22420f 
Add a new cluster version that bumps the Pebble format major version to
FormatFlushableIngest, enabling the use of flushable ingestions. Future work
will introduce a cluster setting allowing disabling flushable ingestions.

Epic: None
Release note: None
Informs cockroachdb#97194.
jbowens added a commit to jbowens/pebble that referenced this issue Mar 15, 2023
@jbowens
db: use function for Experimental.DisableIngestAsFlushable
8d3e1c8 
Convert the Experimental.DisableIngestAsFlushable setting to a function,
allowing the client to disable ingesting sstables as flushable at runtime. This
will be used to tie the setting to a cluster setting in CockroachDB.

This will be backported to 23.1.

Informs cockroachdb/cockroach#97194.
@jbowens jbowens mentioned this issue Mar 15, 2023
Merged
craig bot pushed a commit that referenced this issue Mar 15, 2023
@msbutler @ajwerner @jbowens
Merge #98399 #98593 #98625 #98675
125aa4a 
98399: roachtest: move HealthChecker, DiskUsageTracker/Logger to roachtestutil pkg r=benbardin a=msbutler

 Epic: none

 Release note: none

98593: sql: use lower admission control priority in schema changes r=ajwerner a=ajwerner

This will change the transaction priority for schema changes and the corresponding validation transactions to be run at the BulkPri priority level.

Epic: none

Fixes: #98592

Release note: None

98625: skip-test: add 'Epic: None' to commit msg r=smg260 a=smg260

.. to satisfy `lint-epic-issue-refs ` when a skip test PR is created

Epic: none

Release note: None

98675: storage: enable flushable ingestion r=nicktrav a=jbowens

Add a new cluster version that bumps the Pebble format major version to FormatFlushableIngest, enabling the use of flushable ingestions. Future work will introduce a cluster setting allowing disabling flushable ingestions.

Epic: None
Release note: None
Informs #97194.

Co-authored-by: Michael Butler <[email protected]>
Co-authored-by: ajwerner <[email protected]>
Co-authored-by: Miral Gadani <[email protected]>
Co-authored-by: Jackson Owens <[email protected]>
jbowens added a commit to cockroachdb/pebble that referenced this issue Mar 15, 2023
@jbowens
db: use function for Experimental.DisableIngestAsFlushable
c9f3b74 
Convert the Experimental.DisableIngestAsFlushable setting to a function,
allowing the client to disable ingesting sstables as flushable at runtime. This
will be used to tie the setting to a cluster setting in CockroachDB.

This will be backported to 23.1.

Informs cockroachdb/cockroach#97194.
jbowens added a commit to jbowens/pebble that referenced this issue Mar 16, 2023
@jbowens
db: use function for Experimental.DisableIngestAsFlushable
232d7d3 
Convert the Experimental.DisableIngestAsFlushable setting to a function,
allowing the client to disable ingesting sstables as flushable at runtime. This
will be used to tie the setting to a cluster setting in CockroachDB.

This will be backported to 23.1.

Informs cockroachdb/cockroach#97194.
@jbowens jbowens mentioned this issue Mar 16, 2023
Merged
jbowens added a commit to cockroachdb/pebble that referenced this issue Mar 16, 2023
@jbowens
db: use function for Experimental.DisableIngestAsFlushable
d44c908 
Convert the Experimental.DisableIngestAsFlushable setting to a function,
allowing the client to disable ingesting sstables as flushable at runtime. This
will be used to tie the setting to a cluster setting in CockroachDB.

This will be backported to 23.1.

Informs cockroachdb/cockroach#97194.
@jbowens jbowens mentioned this issue Mar 17, 2023
Merged
craig bot pushed a commit that referenced this issue Mar 20, 2023
@jbowens @DrewKimball
Merge #97789 #98873 #98874
3427f11 
97789: docs: Pass in Jira tickets in the body of product change issues r=nickvigilante a=nickvigilante

Epic and issue references are now pointing to the associated Jira
tickets.

Fixes DOC-7019

Release note: None

98873: storage: add storage.ingest_as_flushable.enabled setting r=nicktrav a=jbowens

Add a cluster setting controlling whether to allow lazy ingestion of sstables that overlap with a memtable. This cluster setting defaults to enabling lazy ingestion, but is only respected by Pebble if the cluster version is ratcheted appropriately high.

Epic: None
Close #97194.
Release note: None

98874: opt: set udf CalledOnNullInput correctly for scalar UDFs r=DrewKimball a=DrewKimball

Previously, the `UDFPrivate.CalledOnNullInput` field would be set to true for scalar UDFs, since strict behavior is handled using a CASE statement. However, this is confusing and it is also correct to just set `CalledOnNullInput` to the value implied by the function's definition.

Informs CRDB-20535

Release note: None

Co-authored-by: Nick Vigilante <[email protected]>
Co-authored-by: Jackson Owens <[email protected]>
Co-authored-by: Drew Kimball <[email protected]>
@craig craig bot closed this as completed in fad6842 Mar 20, 2023
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jbowens jbowens

Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-storage Storage Team
Projects
No open projects
[Deprecated] Storage
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

storage: add storage.ingest_as_flushable.enabled setting jbowens/cockroach
3 participants
@jbowens @bananabrick @sumeerbhola