Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv: replan rangefeeds with chronic closed ts lag #8

Merged
merged 397 commits into from
Dec 19, 2024
Merged
Changes from 2 commits
Commits
Show all changes
397 commits
Select commit Hold shift + click to select a range
c3d2c23
vecstore: add in-memory lock
andy-kimball Dec 5, 2024
471c13d
vecstore: add per-partition locking to InMemoryStore
andy-kimball Dec 5, 2024
972c1d5
Merge #136784
Dec 7, 2024
9744e5f
Merge #136809
Dec 7, 2024
236472e
clusterversion: remove old gates
RaduBerinde Dec 7, 2024
ccaed97
Merge #136076
Dec 7, 2024
8dbb632
kv: delete TestSendToOneClient
nvanbenschoten Dec 7, 2024
ab104ff
roachtest: update regression.diffs
mw5h Nov 15, 2024
f0d009e
Merge #135313
Dec 7, 2024
d7ea854
Merge #135335
Dec 7, 2024
ed0ae97
vecstore: move methods from Store to Txn
andy-kimball Dec 7, 2024
8763093
vecstore: move Txn code into separate file
andy-kimball Dec 7, 2024
bf6cbf6
catalog/descs: avoid log.Fatal for invalid descriptor error
rafiss Dec 8, 2024
714529c
kvserver: deflake TestLeaseRequestFromExpirationToEpochOrLeaderDoesNo…
iskettaneh Dec 8, 2024
683618f
Merge #136849
Dec 9, 2024
5d83c6c
pkg/util/debugutil: introduce debug.Stack() wrapper
arjunmahishi Nov 28, 2024
5ff2ff2
pkg/*: replace uses of debug.Stack() with debugutil.Stack()
arjunmahishi Nov 28, 2024
d4ea2d5
pkg/testutils/lint: add a linter rule to block call to debug.Stack()
arjunmahishi Dec 4, 2024
d29db81
Merge #136869
Dec 9, 2024
74e83fa
Merge #136288
Dec 9, 2024
c341c35
roachtest: fix resize operation in run-operation command
shailendra-patel Dec 5, 2024
c83ac56
Merge #136787
Dec 9, 2024
b7331e4
Merge #136797
Dec 9, 2024
f3c0952
pkg/util/allstacks: wrap the stack traces with debugutil.SafeStack
arjunmahishi Dec 9, 2024
4a8eb1d
Merge #136926
Dec 9, 2024
3b2d6d8
kvserver/rangefeed: change sync.Locker to sync.Mutex
wenyihu6 Dec 7, 2024
03cb1f1
Merge #137007
Dec 9, 2024
68f0f85
pkg/cli: elide table names in tested zip output
dt Dec 9, 2024
5f7af75
Merge #136942 #136968
Dec 9, 2024
da8dc48
kvserver/rangefeed: add unbuffered registration
wenyihu6 Aug 30, 2024
0dfde15
Merge #136855
Dec 9, 2024
36491c4
sql/externalcatalog: extract type descriptors
msbutler Dec 6, 2024
4b1193b
crosscluster/logical: use external catalog to plan LDR job
msbutler Dec 6, 2024
96d0840
sql/externalcatalog: rewrite table descriptor ids in IngestExternalCa…
msbutler Dec 9, 2024
27fd7f9
Merge #137014
Dec 9, 2024
dec11f5
Merge #136310 #136545 #136919
Dec 9, 2024
bd17f89
cli: include new jobs tables in debug zips
dt Nov 19, 2024
c651d65
Merge #136740
Dec 9, 2024
7b563ad
cliccl: use latest version for the TestList
RaduBerinde Dec 9, 2024
128c952
Merge #135963
Dec 9, 2024
a247ae5
Merge #136785
Dec 9, 2024
3a65456
testutils/list: add NoEnumeratingAllTables linter
dt Dec 9, 2024
e382842
sqlstatsccl: speed up and deflake TestSQLStatsRegions
dhartunian Dec 4, 2024
b0667c1
Merge #136840
Dec 9, 2024
0b43622
Merge #137028
Dec 9, 2024
e20a781
tracing: consolidate regexp bench into BenchmarkTracer_StartSpanCtx
dhartunian Dec 9, 2024
ac41722
tracing: add startSpanFast for optimized path when disabled
dhartunian Dec 9, 2024
5e7a5b1
sql/tests: add `ANALYZE` to sysbench microbenchmark
mgartner Dec 9, 2024
7f75f3e
Merge #136978
Dec 9, 2024
e17d713
tree: make nil DatumAlloc act as a no-op
yuzefovich Dec 5, 2024
2831511
stats: use no-op DatumAlloc when decoding EncDatums
yuzefovich Dec 5, 2024
42e1e79
teamcity: remove `bench` scripts
rickystewart Dec 9, 2024
46a3e88
storage: fix incorrect mount association
RaduBerinde Dec 9, 2024
4be77ab
Merge #136780 #136901 #136994 #137039 #137049
Dec 9, 2024
67b1c9c
catalog/lease: deflake TestDescriptorRefreshOnRetry
rafiss Dec 9, 2024
660522e
docgen: update BEGIN and SET TRANSACTION diagrams
taroface Dec 9, 2024
4c1602e
Merge #136820 #136911
Dec 9, 2024
139bbcd
roachtest: update mt-upgrade test owner to db-server
rimadeodhar Dec 9, 2024
5e1e1a1
server/license: Test license enforcement stays disabled after single-…
spilchen Dec 9, 2024
ff5a897
sql/catalog: Address late review comment from #136488
spilchen Dec 10, 2024
bc23142
kvserver: eagerly unquiesce ranges that use leader leases
arulajmani Nov 26, 2024
f1c04dd
rpc: avoid allocation of TracingInternalClient when tracing disabled
nvanbenschoten Dec 6, 2024
de3b122
Merge #136941
Dec 10, 2024
f20164e
rpc: reuse gRPC streams across unary BatchRequest RPCs
nvanbenschoten Dec 3, 2024
a8c78e5
rpc: avoid heap allocations of InternalClient objects
nvanbenschoten Dec 9, 2024
db408d2
rpc: make batch stream pool general over Conn
tbg Dec 9, 2024
af6f42f
workflow: remove cockroach-microbench-ci
sambhav-jain-16 Dec 10, 2024
5f83714
Merge #137075
Dec 10, 2024
8c810eb
kvserver: fix bad expectation in TestReplicaRangefeedErrrors
stevendanna Dec 10, 2024
2d215ad
rpc: add BenchmarkGRPCPing
tbg Nov 29, 2024
bb4d357
Merge #136558
Dec 10, 2024
69b1dcf
kvnemesis: increase the retry backoff for aborted txns
miraradeva Dec 10, 2024
87b463a
roachtest: increase the token return time with disk bandwidth limit
andrewbaptist Dec 9, 2024
28d9b53
Merge #136232
Dec 10, 2024
b0632b9
roachtest: better triage schema workload flakes in backup-restore tests
msbutler Dec 10, 2024
ec5a185
roachtest: disable rate limiters in mixedversion
DarrylWong Dec 6, 2024
790ae95
Merge #136930
Dec 10, 2024
cd6aca1
raft: add counters for flow state changes
pav-kv Dec 10, 2024
4fdc203
rac2: improve TestRangeController wrt match and inflight-bytes
sumeerbhola Dec 10, 2024
d7c5dc0
schemaexpr: add assignment cast to computed column if types differ
rafiss Dec 10, 2024
1efb255
Merge #136931
Dec 10, 2024
8a87993
kvserver: fix missing RaftTruncatedState in RangeInfo
pav-kv Dec 10, 2024
202bb43
kvserver: add TestFlowControlSendQueueRangeSplitMerge test
kvoli Nov 15, 2024
ca0365c
kvcoord: deflake TestDistSenderReplicaStall
miraradeva Dec 10, 2024
35a018c
kvserver: deflake TestConsistencyQueueRecomputeStats
miraradeva Dec 9, 2024
2926980
kvserver/rangefeed: rename *.buffered_stream_sender.* to *.buffered_s…
wenyihu6 Dec 9, 2024
b2795d8
changefeedccl: skip some sql errors in random expr changefeed test
rharding6373 Dec 9, 2024
4c67ac5
raft: count flow states when leader steps up
pav-kv Dec 10, 2024
4b31b16
Merge #136927 #137024 #137058
Dec 10, 2024
18e6931
release: released CockroachDB version 25.1.0-alpha.0. Next version: 2…
cockroach-teamcity Dec 10, 2024
50f8fe0
crosscluster/logical: update udf test to expect at-least-once
dt Dec 10, 2024
452161e
backup: print total testcluster setup time
dt Dec 10, 2024
95d95a6
Merge #136258 #136648 #137019 #137044 #137059 #137099 #137118
Dec 10, 2024
f36c509
sql/externalcatalog: return written descriptors in IngestExternalCatalog
msbutler Dec 9, 2024
f05cedf
crosscluster/logical: create new tables if specified
msbutler Dec 9, 2024
46e8e0f
Merge #137067 #137094
Dec 10, 2024
34e3abf
kv: allocate BatchRequest header and first request together
nvanbenschoten Dec 10, 2024
0165b50
kv: allocate BatchResponse header and responses together
nvanbenschoten Dec 10, 2024
8a55611
build: publish build metadata as a part of Make and Publish Build
rail Dec 10, 2024
a3957e4
Merge #137134
Dec 10, 2024
fadec71
roachtest: remove transaction rate verification from cdc/ledger test
aerfrei Dec 9, 2024
9f28bc0
Merge #136782 #137032 #137070
Dec 10, 2024
4bd1fbc
loq: clear LeadEpoch when re-writing the RangeDescriptor
iskettaneh Dec 9, 2024
1efc73e
changefeedccl: add sql smith to TestChangefeedNemeses
wenyihu6 Nov 19, 2024
993a878
ui: use error body over status text for err messages
xinhaoz Dec 4, 2024
80e5238
schemaexpr: remove unused evalCtx parameter
rafiss Dec 10, 2024
e439d73
roachtest: use updated update_tenant_resource_limits signature in 24.…
DarrylWong Dec 10, 2024
2ebc611
Merge #137097
Dec 10, 2024
024cce3
tracker: add InflightBytes to BasicProgress
sumeerbhola Dec 7, 2024
6391681
kvserver,rac2: use RaftMaxInflightBytes when force-flushing
sumeerbhola Dec 7, 2024
b2c7360
sql: fix failing tests for 25.1
celiala Dec 10, 2024
57273c7
Merge #137055 #137137
Dec 10, 2024
fa56750
Merge #137107
Dec 10, 2024
df024bc
tests/cdc: remove initial scan from changefeed assume role tests
rharding6373 Dec 10, 2024
17e7e5e
cmd/smith: add an option to only generate UDFs
yuzefovich Dec 10, 2024
a3fd8ba
sqlsmith: extend PLpgSQL support
yuzefovich Dec 11, 2024
d18eb68
Merge #137108
Dec 11, 2024
4455478
backup: rewrite UDT oids in histograms when restoring table stats
michae2 Dec 11, 2024
5310b9c
cli/interactive_tests: add `--logs-dir` for server started with `demo…
shubhamdhama Dec 11, 2024
355dd14
cli: update CLI output for tsdump upload
aa-joshi Dec 9, 2024
5b9cad9
Merge #137003
Dec 11, 2024
3e1c78a
Merge #137001
Dec 11, 2024
805fafd
Merge #137119
Dec 11, 2024
8ea3e13
roachtest: add groups to task manager
herkolategan Nov 20, 2024
6c69685
roachtest: add groups to mvt helper
herkolategan Nov 27, 2024
97578e0
roachtest: add groups to test framework
herkolategan Nov 27, 2024
9b40ff3
roachtest: use a task group in multitenant upgrade test
herkolategan Nov 27, 2024
7c758d4
roachtest: take a side-eye snapshot when an online restore test is slow
jeffswenson Dec 10, 2024
b687eef
Merge #137082
Dec 11, 2024
8195bbe
Merge #137068
Dec 11, 2024
d32b44d
release: generate cloud-only images automatically
rail Dec 11, 2024
bcd6c79
kvserver: run LoQ tests with leader leases
iskettaneh Dec 11, 2024
cd89447
roachtest: only run pcr tests on gce
msbutler Dec 11, 2024
cc2eb85
roachtest: only run ldr tests on gce
msbutler Dec 11, 2024
d422abf
roachtest: increase max accepted latency on c2c shutdown tests
msbutler Dec 11, 2024
9c5816a
roachtest: run sysbench nightly
tbg Dec 11, 2024
77c03da
kvserver: add TestFlowControlSendQueueRangeSplitMergeMixedVersion
kvoli Dec 4, 2024
c32b701
Merge #137277
Dec 11, 2024
bfdabf4
sql/tests: fix primary index value encoding
nvanbenschoten Dec 11, 2024
12a1804
sql/tests: fix secondary index key in sysbenchKV.DeleteInsert
nvanbenschoten Dec 11, 2024
98c7d8e
Merge #136772 #137245
Dec 11, 2024
0cd9351
sql: add `(*connExecutor).execStmtInOpenStateWithPausablePortal`
mgartner Dec 4, 2024
29be717
sql: remove `isPausablePortal` anonymous function in `execStmtInOpenS…
mgartner Dec 4, 2024
11f5bac
sql: remove pausable-portal-specific code paths in `execStmtInOpenState`
mgartner Dec 5, 2024
03492a6
sql: remove `processCleanupFunc` in `execStmtInOpenState`
mgartner Dec 5, 2024
d666566
sql: remove `portal` parameter from `execStmtInOpenState`
mgartner Dec 5, 2024
353856d
sql: remove `localVars` struct in `execStmtInOpenState`
mgartner Dec 6, 2024
e56661a
sql: refactor errors in `execStmtInOpenState`
mgartner Dec 9, 2024
ca13fec
sql: add metamorphic test variable to force pausable portal logic
mgartner Dec 9, 2024
a738718
privilege: verify that MODIFY privileges have related VIEW privileges
rafiss Dec 11, 2024
7701661
Merge #137267
Dec 11, 2024
7838a69
Merge #137242
Dec 11, 2024
9952005
roachtest: make disk bandwidth test manual only
aadityasondhi Dec 3, 2024
2b3e4f4
Merge #136225 #137136 #137216 #137275 #137281
Dec 11, 2024
b8c532e
sql: label latency histograms with statement fingerprint
MattWhelan Nov 15, 2024
7be5bfb
Merge #136902
Dec 11, 2024
0b676d9
ci: check that generated documentation is up-to-date
rickystewart Dec 10, 2024
e8aa398
Merge #137135 #137274
Dec 11, 2024
dca5db7
sql: pass session data to getPlanDistribution
yuzefovich Dec 11, 2024
8269527
sql: adjust physical planning heuristics around small sorts and aggre…
yuzefovich Dec 10, 2024
9d69824
sql: distribute large top K sorts
yuzefovich Dec 10, 2024
f6ff2fe
sql: adjust physical planning heuristic for scans
yuzefovich Dec 10, 2024
6916103
kvserver: deflake TestScanNodeVitalityFromKV
iskettaneh Dec 11, 2024
7575548
Merge #137072 #137234
Dec 11, 2024
51f5ba3
raft: remove fortification tests which don't run with prevote
arulajmani Dec 11, 2024
34373ea
Merge #136765
Dec 11, 2024
f8f539e
Merge #136922
Dec 11, 2024
7c9ac9f
raft: forget leader in testingStepDown
arulajmani Dec 11, 2024
e1673ec
raft: remove redundant special case in becomeFollower
arulajmani Dec 11, 2024
975c428
ci: bump dependency for `code-cover-pubish` GH action
srosenberg Dec 11, 2024
00264aa
Merge #137298
Dec 11, 2024
144cfad
raft: clean up getBasicStatus hack
arulajmani Dec 11, 2024
90cae56
ci: fix typo in `code-cover-publish` GH action
srosenberg Dec 11, 2024
53fadf4
Merge #136861 #136977 #137313
Dec 11, 2024
e8ee6f5
Merge #137318
Dec 12, 2024
e383180
sql: add max_prepared_transactions variable
nvanbenschoten Aug 18, 2024
c66a374
sql: add parser support for XA transactions
nvanbenschoten Aug 15, 2024
160db6f
cli: push tsdump upload failed metrics to datadog logs
aa-joshi Dec 11, 2024
fb7afda
cmd/dev: add short flag to lint during build
dt Dec 12, 2024
e273e83
kvserver/rangefeed: metamorphically enable RangefeedUseBufferedSender
wenyihu6 Dec 8, 2024
32bf42d
sql: create system.prepared_transactions table
nvanbenschoten Aug 16, 2024
1a038d8
Merge #129295 #129297 #129298
Dec 12, 2024
36df22d
Merge #137091
Dec 12, 2024
43fe07f
Merge #135924 #136995 #137288
Dec 12, 2024
e6e3287
backupccl: altering backup schedule no longer resumes schedule
kev-cao Nov 7, 2024
771b4f0
backupccl: add test for alter backup schedule into new URI
kev-cao Dec 12, 2024
cffa888
ccl/changefeedccl/cdcevent: Fix flake in TestEventColumnOrderingWithS…
spilchen Dec 12, 2024
497656e
Merge #137334
Dec 12, 2024
90678e7
Merge #137300
Dec 12, 2024
6cb15dc
Merge #137333
Dec 12, 2024
3ad8c0f
sql: do not limit set-returning UDF when it has OUT parameters
DrewKimball Dec 11, 2024
074371d
sql: implement RETURNS TABLE syntax
DrewKimball Dec 11, 2024
60c5e14
opt: add method to add strict dependency to FuncDepSet
DrewKimball Jan 8, 2024
4f464d5
Merge #137089
Dec 12, 2024
ab9a800
kvserver: make SubsumeRequest do a write
sumeerbhola Dec 11, 2024
0e4cb73
Merge #137031
Dec 12, 2024
ae3299b
kvserver,rac2: turn off raftReceiveQueue.maxLen enforcement in apply_…
sumeerbhola Dec 7, 2024
b3aba93
opt: add implicit SELECT FOR UPDATE to initial scan of DELETE
yuzefovich Dec 10, 2024
6a3221f
opt: infer functional dependencies for window functions
DrewKimball Jan 8, 2024
147054d
opt: add rule to merge GroupBy and Window
DrewKimball Jan 8, 2024
f4b41a8
optbuilder: always select tombstone index using the index ordinal
mw5h Dec 12, 2024
cccfc7c
Merge #137361
Dec 12, 2024
32886df
Merge #117499 #137069
Dec 12, 2024
bb0e758
ui,metric,server: fix custom chart to include histogram metrics
kyle-a-wong Dec 6, 2024
02da6b2
Merge #137251
Dec 13, 2024
ddbb08e
changefeedccl/kvfeed: add log for table that encountered schema change
andyyang890 Dec 12, 2024
a55295b
Merge #136969 #137308
Dec 13, 2024
603ff88
Merge #136900
Dec 13, 2024
fada042
Merge #137250
Dec 13, 2024
476b683
roachtest: use new ruby version for activerecord test
rafiss Dec 13, 2024
ada0ea7
Merge #135831
Dec 13, 2024
4ace523
roachprod: respect AWS_PROFILE env var
herkolategan Dec 13, 2024
5590074
Merge #134829
Dec 13, 2024
c33f39c
kvserver: deflake TestReplicaCircuitBreaker_RangeFeed
iskettaneh Dec 13, 2024
047d1be
Merge #137092 #137393
Dec 13, 2024
04392dc
ci: deadmansnitch for code-cover-publish Github Action
golgeek Dec 13, 2024
bc94828
roachtest: bump up the acceptable change for intent perturbation
miraradeva Dec 13, 2024
e894ec7
Merge #137219
Dec 13, 2024
aefb582
roachtest: remove RequiresLicense test spec
DarrylWong Dec 11, 2024
40a187c
Merge #137405
Dec 13, 2024
8919614
Merge #137412
Dec 13, 2024
2e66bb2
jobspb: mark ResolvedSpan_BoundaryType as SafeValue
andyyang890 Dec 13, 2024
95025db
docgen: remove unused BEGIN diagram
taroface Dec 13, 2024
42593c0
Merge #137383
Dec 13, 2024
614464d
changefeedccl: eliminate race in test validation
rharding6373 Nov 5, 2024
12d3116
scbuild: add transient constraint to validate computed column
rafiss Dec 12, 2024
b7f7c56
optbuilder: use assignment cast for computed columns
rafiss Dec 12, 2024
1bcc653
scjob: mark assertion errors as permanent job errors
rafiss Dec 12, 2024
0bad78d
logictest: verify that ON UPDATE expressions' size is checked
rafiss Dec 13, 2024
3ad421a
schemaexpr: consolidate logic to wrap expression in assignment cast
rafiss Dec 13, 2024
ced8f94
Merge #137375 #137410
Dec 13, 2024
a34f999
kvserver: make TestRaftReceiveQueuesEnforceMaxLenConcurrency cheaper
sumeerbhola Dec 13, 2024
c586eda
sql/catalog: remove isMutable flag toggle in getObjectPrefix
mgartner Dec 13, 2024
3878bf6
Merge #134331 #137311
Dec 13, 2024
a4d45a4
Merge #137299
Dec 13, 2024
e188730
go.mod: update logtag dependency
RaduBerinde Dec 13, 2024
7f44c49
serverident: unexport ServerIdentificationContextKey
RaduBerinde Dec 13, 2024
c0b40de
rac2: ignore MsgApps when constructing RaftEvent in MsgAppPull mode
sumeerbhola Dec 13, 2024
882ecf9
ctxutil: introduce fast values
RaduBerinde Dec 13, 2024
57a7b11
Merge #137359 #137432
Dec 13, 2024
634c21a
opt: add implicit FOR UPDATE locks for generic query plans
mgartner Dec 12, 2024
6448bef
Merge #137447
Dec 13, 2024
81950d1
Merge #137265
Dec 13, 2024
31250b7
sql/tests: manually split ranges in BenchmarkSysbench/KV
nvanbenschoten Dec 14, 2024
8f401f7
Merge #137469
Dec 14, 2024
49cff91
Merge #137440
Dec 14, 2024
fe048f6
Merge #137360
Dec 15, 2024
9354770
Merge #137344
Dec 15, 2024
8abbe9b
Merge #137455
Dec 16, 2024
2f7df22
kv: replan rangefeeds with chronic closed ts lag
kvoli Dec 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions pkg/kv/kvserver/rangefeed/buffered_registration.go
Original file line number Diff line number Diff line change
@@ -7,7 +7,6 @@ package rangefeed

import (
"context"
"sync"
"sync/atomic"
"time"

@@ -48,7 +47,7 @@ type bufferedRegistration struct {
blockWhenFull bool // if true, block when buf is full (for tests)

mu struct {
sync.Locker
syncutil.Mutex
// True if this registration buffer has overflowed, dropping a live event.
// This will cause the registration to exit with an error once the buffer
// has been emptied.
@@ -100,7 +99,6 @@ func newBufferedRegistration(
buf: make(chan *sharedEvent, bufferSz),
blockWhenFull: blockWhenFull,
}
br.mu.Locker = &syncutil.Mutex{}
br.mu.catchUpIter = catchUpIter
return br
}
@@ -110,6 +108,8 @@ func newBufferedRegistration(
// indicating that live events were lost and a catch-up scan should be initiated.
// If overflowed is already set, events are ignored and not written to the
// buffer.
//
// nolint:deferunlockcheck
func (br *bufferedRegistration) publish(
ctx context.Context, event *kvpb.RangeFeedEvent, alloc *SharedBudgetAllocation,
) {
@@ -189,6 +189,8 @@ func (br *bufferedRegistration) Disconnect(pErr *kvpb.Error) {
// The loop exits with any error encountered, if the provided context is
// canceled, or when the buffer has overflowed and all pre-overflow entries
// have been emitted.
//
// nolint:deferunlockcheck
func (br *bufferedRegistration) outputLoop(ctx context.Context) error {
// If the registration has a catch-up scan, run it.
if err := br.maybeRunCatchUpScan(ctx); err != nil {
@@ -255,6 +257,7 @@ func (br *bufferedRegistration) outputLoop(ctx context.Context) error {
}
}

// nolint:deferunlockcheck
func (br *bufferedRegistration) runOutputLoop(ctx context.Context, _forStacks roachpb.RangeID) {
defer br.drainAllocations(ctx)