admission: roachtest-ify index/column backfills impacting foreground traffic #83826
Labels
A-admission-control
A-kv
Anything in KV that doesn't belong in a more specific category.
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-kv
KV Team
Comments
irfansharif
added
C-enhancement
irfansharif
changed the title
This was referenced Oct 3, 2022
|
Some data from the last run |
10 tasks
15 tasks
craig bot
pushed a commit
that referenced
this issue
May 26, 2023
103757: roach{prod,test}: add first-class support for disk snapshots r=irfansharif a=irfansharif
Part of #89978. Pre-cursor to #83826. Part of #98703.
Long-lived disk snapshots can drastically reduce testing time for scale tests. Tests, whether run by hand or through CI, need only run the long running fixture generating code (importing some dataset, generating it organically through workload, etc.) once snapshot fingerprints are changed, fingerprints that incorporate the major crdb version that generated them.
Here's an example run that freshly generates disk snapshots:
=== RUN admission-control/index-backfill
no existing snapshots found for admission-control/index-backfill (ac-index-backfill), doing pre-work
created volume snapshot ac-index-backfill-0001-vunknown-1-n2-standard-8 for volume irfansharif-snapshot-0001-1 on irfansharif-snapshot-0001-1/n1
using 1 newly created snapshot(s) with prefix "ac-index-backfill"
detached and deleted volume irfansharif-snapshot-0001-1 from irfansharif-snapshot-0001
created volume irfansharif-snapshot-0001-1
attached volume irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
mounted irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
--- PASS: admission-control/index-backfill (79.14s)
Here's a subsequent run that makes use of the aforementioned disk snapshots:
=== RUN admission-control/index-backfill
using 1 pre-existing snapshot(s) with prefix "ac-index-backfill"
detached and deleted volume irfansharif-snapshot-0001-1 from irfansharif-snapshot-0001
created volume irfansharif-snapshot-0001-1
attached volume irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
mounted irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
--- PASS: admission-control/index-backfill (43.47s)
We add the following APIs to the roachtest.Cluster interface, for tests to interact with disk snapshots. admission-control/index-backfill is an example test making use of these APIs.
```go
type Cluster interface {
// ...
// CreateSnapshot creates volume snapshots of the cluster using
// the given prefix. These snapshots can later be retrieved,
// deleted or applied to already instantiated clusters.
CreateSnapshot(ctx context.Context, snapshotPrefix string) error
// ListSnapshots lists the individual volume snapshots that
// satisfy the search criteria.
ListSnapshots(
ctx context.Context, vslo vm.VolumeSnapshotListOpts,
) ([]vm.VolumeSnapshot, error)
// DeleteSnapshots permanently deletes the given snapshots.
DeleteSnapshots(
ctx context.Context, snapshots ...vm.VolumeSnapshot,
) error
// ApplySnapshots applies the given volume snapshots to the
// underlying cluster. This is a destructive operation as far as
// existing state is concerned - all already-attached volumes are
// detached and deleted to make room for new snapshot-derived
// volumes. The new volumes are created using the same specs
// (size, disk type, etc.) as the original cluster.
ApplySnapshots(
ctx context.Context, snapshots []vm.VolumeSnapshot,
) error
}
```
These Cluster APIs are in turn is powered by the following additions to the vm.Provider interface, implemented by each cloud provider. GCE is the fully spec-ed out one for now.
```go
type Provider interface {
// ...
// CreateVolume creates a new volume using the given options.
CreateVolume(l *logger.Logger, vco VolumeCreateOpts) (Volume, error)
// ListVolumes lists all volumes already attached to the given VM.
ListVolumes(l *logger.Logger, vm *VM) ([]Volume, error)
// DeleteVolume detaches and deletes the given volume from the
// given VM.
DeleteVolume(l *logger.Logger, volume Volume, vm *VM) error
// AttachVolume attaches the given volume to the given VM.
AttachVolume(l *logger.Logger, volume Volume, vm *VM) (string, error)
// CreateVolumeSnapshot creates a snapshot of the given volume,
// using the given options.
CreateVolumeSnapshot(
l *logger.Logger, volume Volume, vsco VolumeSnapshotCreateOpts,
) (VolumeSnapshot, error)
// ListVolumeSnapshots lists the individual volume snapshots that
// satisfy the search criteria.
ListVolumeSnapshots(
l *logger.Logger, vslo VolumeSnapshotListOpts,
) ([]VolumeSnapshot, error)
// DeleteVolumeSnapshot permanently deletes the given snapshot.
DeleteVolumeSnapshot(l *logger.Logger, snapshot VolumeSnapshot) error
}
```
Since these snapshots necessarily outlive the tests, and we don't want them dangling perpetually, we introduce a prune-dangling roachtest that acts as a poor man's cron job, sifting through expired snapshots (>30days) and deleting them. For GCE at least it's not obvious to me how to create these snapshots in cloud buckets with a TTL built in, hence this hack. It looks like this (with change to the TTL):
=== RUN prune-dangling
pruned old snapshot ac-index-backfill-0001-vunknown-1-n2-standard-8
--- PASS: prune-dangling (8.59s)
---
We add expose some of these APIs through the roachprod binary directly.
```
$ roachprod snapshot --help
snapshot enables creating/listing/deleting/applying cluster snapshots
Usage:
roachprod snapshot [command]
Available Commands:
create snapshot a named cluster, using the given snapshot name and description
list list all snapshots for the given cloud provider, optionally filtering by the given name
delete delete all snapshots for the given cloud provider optionally filtering by the given name
apply apply the named snapshots from the given cloud provider to the named cluster
```
---
About admission-control/index-backfill. It's a fully featured test that uses the TPC-C 100k dataset and runs a foreground load for 20k customers. It takes >4hrs to import this data set; with disk snapshots this step is skipped entirely and takes a few minutes. The actual test is trivial, we run the foreground load for 1hr and run a large index backfill concurrently. Before #98308, this results in wild performance oscillations. It's still a bit wild after flow control, but less so.
We slightly extend the tpc-e harness to make this happen, adding a few smarts: exposing a 'during' helper to run backfills concurrently with foreground load, integrate with --skip-init, estimated setup times, prometheus, and disk snapshots of course.
Release note: None
Co-authored-by: irfan sharif <[email protected]>
|
This was done in #103757. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
In #82700 we have a manual reproduction for column backfills resulting in diskbandwidth starvation, and severely impacting tail latencies. This issue tracks roachtest-ifying the reproduction, possibly after evaluating fixes through admission control: #82898.
Jira issue: CRDB-17285
Epic CRDB-25348
The text was updated successfully, but these errors were encountered: