Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
100739: go.mod: bump Pebble to 16bf77ee4012 r=RaduBerinde a=RaduBerinde

16bf77ee shared: add IsNotExistError method to shared.Storage
93eff0a7 manifest: remove struct Atomic from FileBacking
6bf568d8 manifest: use atomic types in FileMetadata
67895e1d manifest: don't copy FileMetadata by value
7d5a3b7c db: use atomic types in Db
a664f69f replay: use atomic types
9de87a35 vfs: use atomic.Uint32 for disk full generation
75b78212 pebble: use atomic.Int32 for memTable.writerRefs
44af3316 cmd/pebble: use atomic types
5e9ccb46 pebble: use atomic type in commitQueue
41fce88e *: use atomic types in tests
e0a52f16 pebble: add TODO for FilterMetrics
18af1e35 pebble: use atomic types in table cache
49137e40 pebble: use atomic.Uint64 for sequence numbers
e808d80d pebble: use atomic.Int32 for iterCount in tableCacheOpts
37f6c795 pebble: use atomic.Bool for Batch.applied
c865bc3c pebble: use atomic.Int64 for TableCache refs

Release note: none
Epic: none

100814: upgrade: waitForDelRangeInGCJob should only wait for running jobs r=fqazi a=fqazi

Previously, the logic to determine that all GC jobs were using DelRange was no-oped because in our testing environments, we always had: storage.mvcc.range_tombstones.enabled set. The mixed version tests for schema would always create GC jobs with DelRange and be fine. Once this default was changed we started to observe the behaviour of this migration was that it will wait till the GC interval before we would move to the next version. To address this, this patch fixes the check inside waitForDelRangeInGCJob to exclude GC jobs that haven't started any work.

Fixes: #100732
Release note: None

Co-authored-by: Radu Berinde <[email protected]>
Co-authored-by: Faizan Qazi <[email protected]>
  • Loading branch information
3 people committed Apr 6, 2023
3 parents 8e532bb + 71bbed7 + 28e727b commit 659b90b
Show file tree
Hide file tree
Showing 7 changed files with 16 additions and 9 deletions.
6 changes: 3 additions & 3 deletions DEPS.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -1555,10 +1555,10 @@ def go_deps():
patches = [
"@com_github_cockroachdb_cockroach//build/patches:com_github_cockroachdb_pebble.patch",
],
sha256 = "f282ddeea7d1c18f2acc37c252e6673b6d11c046a1c78bb28106bd7b8feea319",
strip_prefix = "github.com/cockroachdb/[email protected]20230403163348-b84a7ec7d8dc",
sha256 = "4ba6205e0829b82c4c0ece96e38c7be860684ae0dab725d745b0d5fd62fd93c8",
strip_prefix = "github.com/cockroachdb/[email protected]20230405180252-16bf77ee4012",
urls = [
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/pebble/com_github_cockroachdb_pebble-v0.0.0-20230403163348-b84a7ec7d8dc.zip",
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/pebble/com_github_cockroachdb_pebble-v0.0.0-20230405180252-16bf77ee4012.zip",
],
)
go_repository(
Expand Down
2 changes: 1 addition & 1 deletion build/bazelutil/distdir_files.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ DISTDIR_FILES = {
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/go-test-teamcity/com_github_cockroachdb_go_test_teamcity-v0.0.0-20191211140407-cff980ad0a55.zip": "bac30148e525b79d004da84d16453ddd2d5cd20528e9187f1d7dac708335674b",
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/gostdlib/com_github_cockroachdb_gostdlib-v1.19.0.zip": "c4d516bcfe8c07b6fc09b8a9a07a95065b36c2855627cb3514e40c98f872b69e",
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/logtags/com_github_cockroachdb_logtags-v0.0.0-20230118201751-21c54148d20b.zip": "ca7776f47e5fecb4c495490a679036bfc29d95bd7625290cfdb9abb0baf97476",
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/pebble/com_github_cockroachdb_pebble-v0.0.0-20230403163348-b84a7ec7d8dc.zip": "f282ddeea7d1c18f2acc37c252e6673b6d11c046a1c78bb28106bd7b8feea319",
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/pebble/com_github_cockroachdb_pebble-v0.0.0-20230405180252-16bf77ee4012.zip": "4ba6205e0829b82c4c0ece96e38c7be860684ae0dab725d745b0d5fd62fd93c8",
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/redact/com_github_cockroachdb_redact-v1.1.3.zip": "7778b1e4485e4f17f35e5e592d87eb99c29e173ac9507801d000ad76dd0c261e",
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/returncheck/com_github_cockroachdb_returncheck-v0.0.0-20200612231554-92cdbca611dd.zip": "ce92ba4352deec995b1f2eecf16eba7f5d51f5aa245a1c362dfe24c83d31f82b",
"https://storage.googleapis.com/cockroach-godeps/gomod/github.com/cockroachdb/sentry-go/com_github_cockroachdb_sentry_go-v0.6.1-cockroachdb.2.zip": "fbb2207d02aecfdd411b1357efe1192dbb827959e36b7cab7491731ac55935c9",
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ require (
github.com/cockroachdb/go-test-teamcity v0.0.0-20191211140407-cff980ad0a55
github.com/cockroachdb/gostdlib v1.19.0
github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b
github.com/cockroachdb/pebble v0.0.0-20230403163348-b84a7ec7d8dc
github.com/cockroachdb/pebble v0.0.0-20230405180252-16bf77ee4012
github.com/cockroachdb/redact v1.1.3
github.com/cockroachdb/returncheck v0.0.0-20200612231554-92cdbca611dd
github.com/cockroachdb/stress v0.0.0-20220803192808-1806698b1b7b
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -479,8 +479,8 @@ github.com/cockroachdb/gostdlib v1.19.0/go.mod h1:+dqqpARXbE/gRDEhCak6dm0l14AaTy
github.com/cockroachdb/logtags v0.0.0-20211118104740-dabe8e521a4f/go.mod h1:Vz9DsVWQQhf3vs21MhPMZpMGSht7O/2vFW2xusFUVOs=
github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b h1:r6VH0faHjZeQy818SGhaone5OnYfxFR/+AzdY3sf5aE=
github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b/go.mod h1:Vz9DsVWQQhf3vs21MhPMZpMGSht7O/2vFW2xusFUVOs=
github.com/cockroachdb/pebble v0.0.0-20230403163348-b84a7ec7d8dc h1:JvaHl6Zd/1rLIJ/sJAkEGCsyFRRp5Lh5nMAZzUnftZc=
github.com/cockroachdb/pebble v0.0.0-20230403163348-b84a7ec7d8dc/go.mod h1:9lRMC4XN3/BLPtIp6kAKwIaHu369NOf2rMucPzipz50=
github.com/cockroachdb/pebble v0.0.0-20230405180252-16bf77ee4012 h1:ZHyejsA4M/E+4fcR3OKQytkIK95yvYaqT4yOpJKFhJQ=
github.com/cockroachdb/pebble v0.0.0-20230405180252-16bf77ee4012/go.mod h1:9lRMC4XN3/BLPtIp6kAKwIaHu369NOf2rMucPzipz50=
github.com/cockroachdb/redact v1.1.3 h1:AKZds10rFSIj7qADf0g46UixK8NNLwWTNdCIGS5wfSQ=
github.com/cockroachdb/redact v1.1.3/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg=
github.com/cockroachdb/returncheck v0.0.0-20200612231554-92cdbca611dd h1:KFOt5I9nEKZgCnOSmy8r4Oykh8BYQO8bFOTgHDS8YZA=
Expand Down
1 change: 0 additions & 1 deletion pkg/sql/schemachanger/sctest/cumulative.go
Original file line number Diff line number Diff line change
Expand Up @@ -1718,7 +1718,6 @@ WHERE
}

func BackupMixedVersionElements(t *testing.T, path string, newCluster NewMixedClusterFunc) {
skip.WithIssue(t, 100732)
testVersion := clusterversion.ClusterVersion{
Version: clusterversion.ByKey(clusterversion.V23_1_SchemaChangerDeprecatedIndexPredicates - 1),
}
Expand Down
5 changes: 5 additions & 0 deletions pkg/storage/shared_storage.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (

"github.com/cockroachdb/cockroach/pkg/cloud"
"github.com/cockroachdb/cockroach/pkg/util/ioctx"
"github.com/cockroachdb/errors"
"github.com/cockroachdb/pebble/objstorage/shared"
)

Expand Down Expand Up @@ -122,3 +123,7 @@ func (e *externalStorageWrapper) Delete(basename string) error {
func (e *externalStorageWrapper) Size(basename string) (int64, error) {
return e.es.Size(e.ctx, basename)
}

func (e *externalStorageWrapper) IsNotExistError(err error) bool {
return errors.Is(err, cloud.ErrFileDoesNotExist)
}
5 changes: 4 additions & 1 deletion pkg/upgrade/upgrades/wait_for_del_range_in_gc_job.go
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,10 @@ func waitForDelRangeInGCJob(
elements AS (SELECT * FROM tables UNION ALL SELECT * FROM indexes)
SELECT id
FROM elements
WHERE COALESCE(progress->>'status' NOT IN ('WAITING_FOR_MVCC_GC', 'CLEARED'), true)
-- While we are waiting for the GC TTL, the status will be WAITING_FOR_CLEAR because omitempty
-- set on this field it will not exist in the JSON output. Because tombstone adoption unconditionally
-- enabled by an earlier version, we should be safe to skip any job that hasn't started GCing yet.
WHERE COALESCE(progress->>'status' NOT IN ('WAITING_FOR_MVCC_GC', 'CLEARED'), false)
GROUP BY id;
`)
if err != nil || len(jobIDs) == 0 {
Expand Down

0 comments on commit 659b90b

Please sign in to comment.