CBG-3764 update base implementation of XDCR in rosmar #6746

torcolvin · 2024-03-28T00:55:25Z

The basic implementation of bucket to bucket XDCR should behave identically to the existing Couchbase Server implementation with -mobile=Off settings but with a manual filter expression to filter all _sync: docs except attachments.

define common interfaces in sg-bucket for XDCR and related structs
move test to use NewXDCR to allow it to run the same test with CBS or rosmar

cleanup:

define DataStoreName on Collection object to avoid casting (this felt better in rosmar than it does in sync gateway)

couchbaselabs/rosmar#29
couchbase/sg-bucket#117

Next steps are in CBG-3861 where the we will probably move a JSON struct for _vv and _mou to sg-bucket so it can be used in rosmar implementation. That change can go into a version of rosmar that exists in 3.2 because it will only be used in rosmar code for beryllium, and then only by test-only xdcr package in sync gateway.

Pre-review checklist

Removed debug logging (fmt.Print, log.Print, ...)
Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Dependencies (if applicable)

Link upstream PRs
Update Go module dependencies when merged

Integration Tests

GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/2368/

(Tested xdcr package with CBS 7.2, 7.6.0, 7.6.1)

* Refactor makeHandler functions to use handlerOptions * Add /_ping API endpoint and silent handler functionality * Add API docs for /_ping * Publicly accessible and declare on metric router * Add test * Skip writing response body when request method is HEAD (RFC-9110) * Only honor handler's requested log level if the request was successful

* Add TestRevocationsWithQueryLimit2Channels and fix loop * Move lastSeq assignment before inner loop

* Avoid a data race on writing db.PurgeInterval during test allow specifying PurgeInterval in DatabaseContextOptions to override the value we might return from the server.

err was shadowed inside a goroutine, added scoping when available

* Silence metrics requests * Lower log level of successful admin auth and 'Recording snapshot of current debug variables' message

Co-authored-by: adamcfraser <[email protected]>

* Pass context into Query * Pass context into changes/channel cache * Fix missing ctx param in test ExplainQuery * Remove loggingCtx to force caller to pass a context and add context for test functions * Only Flush on gocb buckets --------- Co-authored-by: Tor Colvin <[email protected]>

Restores the 3.0.x read buffer size, to avoid memory growth in 3.1.1 associated with cbgt/import.

* Make SubdocXattrStore work with Rosmar collections * There's a new interface sgbucket.XattrStore2 that contains the methods of SubdocXattrStore that aren't redundant. * rosmar.Collection implements XattrStore2. * LeakyDataStore now implements SubdocXattrStore; the methods are either delegated to their existing DataStore counterparts, or to the new methods from XattrStore2. * AsSubdocXattrStore() isn't needed anymore, because LeakyDataStore now implements SubdocXattrStore. * Removed obsolete TestRequiresSubdocXattrStore()

Co-authored-by: Ben Brooks <[email protected]>

* CBG-3352 remove invalidation, just remove data from cache * Fix fieldalignment on structs

- drop a lock by not checking the value before we remove it from the cache. - switched name from Invalidate to Remove for clarity

* CBG-3292: force config update on corrupt db config

…achment (#6414)

* Update XattrStore, remove subdocXattrStore Remove subdocXattrStore (previously used to support working with multiple gocb versions), moving required functions not already included in XattrStore into XattrStore.

Co-authored-by: Ben Brooks <[email protected]>

* Spin master into 3.3 and continue building 4.0 on separate branch * Bump 3.2 start build * bump master API version * Move most 3.2 stats to 3.3 * Change 4.0 branch name

* Update dependabot.yml to pull from private registry * Update .github/dependabot.yml Co-authored-by: Ben Brooks <[email protected]> --------- Co-authored-by: Ben Brooks <[email protected]>

Bumps [github.com/gorilla/mux](https://github.com/gorilla/mux) from 1.8.0 to 1.8.1. - [Release notes](https://github.com/gorilla/mux/releases) - [Commits](gorilla/mux@v1.8.0...v1.8.1) --- updated-dependencies: - dependency-name: github.com/gorilla/mux dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.14.0 to 0.18.0. - [Commits](golang/oauth2@v0.14.0...v0.18.0) --- updated-dependencies: - dependency-name: golang.org/x/oauth2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps google.golang.org/protobuf from 1.32.0 to 1.33.0. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: adamcfraser <[email protected]>

- define common interfaces in sg-bucket for XDCR and related structs - move test to use NewXDCR to allow it to run the same test with CBS or rosmar cleanup: - define DataStoreName on Collection object to avoid casting

github-actions · 2024-04-01T15:50:00Z

Redocly previews

- xdcr tests have timing issues with creating an XDCR bucket to bucket replication and do not run 95% of the time. - fix an error with defer bucket.Close() where the tests would panic

Bumps [github.com/elastic/gosigar](https://github.com/elastic/gosigar) from 0.14.2 to 0.14.3. - [Release notes](https://github.com/elastic/gosigar/releases) - [Changelog](https://github.com/elastic/gosigar/blob/master/CHANGELOG.md) - [Commits](elastic/gosigar@v0.14.2...v0.14.3) --- updated-dependencies: - dependency-name: github.com/elastic/gosigar dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Cleanup tests - Create TB() helper functions to make it easier to pick up the right TB function since it can get reset or hard to pick up - use require.EventuallyT instead of testing.T.Fatalf to get tracebacks when they fail. Because these functions fail anyway, removed the return of boolean found. - switch panic to require statements

- changes defaults to enabled - update golangci-lint-action to v4 to avoid node deprecation warnings from action

* Combine MutationStore2, Collection, DataStoreName into DataStore This is cleanup from the introduction to rosmar. - rosmar.Collection and Collection already satisified the DataStoreName interface, so putting it as part of DataStore removes need for AsDataStoreName checks. - remove GetFeedType() checks since it will always return DCP - Remove StartTapFeed from leaky bucket because it was not being used. TestStopChangeCache was only running with xattrs disabled, but the test would pass even if the events arrived out of order, so the leaky bucket tap code was not being executed. - Reflect in documentation that DCP feed is always used. * Replace leaky bucket code for DCPMissingDocs

torcolvin · 2024-04-09T14:31:10Z

Superseded by #6761 which includes all the code in sg-bucket & rosmar into sync gateway.

Additionally this PR happened at a time when main branch was created and rebased from master, so this PR is stale in many ways.

bbrks and others added 30 commits August 23, 2023 09:39

CBG-3273: Fix revocation feed loop (#6378)

d1898fd

* Add TestRevocationsWithQueryLimit2Channels and fix loop * Move lastSeq assignment before inner loop

Avoid a data race on writing db.PurgeInterval during test (#6363)

18f7420

* Avoid a data race on writing db.PurgeInterval during test allow specifying PurgeInterval in DatabaseContextOptions to override the value we might return from the server.

CBG-2455 fix race detector issue (#6380)

36cf9a8

err was shadowed inside a goroutine, added scoping when available

CBG-3316: Silence metrics requests (#6376)

a3f83b5

* Silence metrics requests * Lower log level of successful admin auth and 'Recording snapshot of current debug variables' message

CBG-3288: Pass Context into NewSyncRunner for logging (#6382)

c585e48

CBG-3171: Fix rev cache multi node inconsistency (#6369)

a45339d

Co-authored-by: adamcfraser <[email protected]>

CBG-3324: Add startup cnf logging to DefaultDbConfig (#6381)

5288dbf

CBG-3289 handle walrus databases as in memory rosmar databases (#6379)

aff7f54

CBG-3288-import: Pass context to import filter function (#6383)

3b3d1a7

DOC-11444 fix key name and add defaults for log rotation (#6385)

9c7cad0

CBG-3283 Restore 3.0.x read buffer size for cbgt (#6386)

ca81fa7

Restores the 3.0.x read buffer size, to avoid memory growth in 3.1.1 associated with cbgt/import.

Create 3.0.8.2 branch (#6389)

9290abe

Co-authored-by: Ben Brooks <[email protected]>

CBG-3352 remove invalid entries from revcache (#6390)

246fde2

* CBG-3352 remove invalidation, just remove data from cache * Fix fieldalignment on structs

Bump golang.org/x/crypto to latest version (#6397)

8f49e36

CBG-3357: Change log levels for StartOnlineProcesses (#6388)

4c23519

CBG-3382: Catch invalid state transition in asyncDatabaseOnline (#6392)

cf74d2e

CBG-3352 switch Invalidate to Remove (#6399)

94071f8

- drop a lock by not checking the value before we remove it from the cache. - switched name from Invalidate to Remove for clarity

CBG-3381 update rosmar to avoid data race (#6400)

132ec4e

CBG-3332 increase log information for documents not imported (#6391)

2bde1ea

CBG-3377: Set up builds for 3.1.2 (#6393)

c660b5a

CBG-3393: Bump to latest Go version (1.21.1) (#6407)

3f1c066

CBG-3285 fix DefaultDbConfig CompactIntervalDays (#6409)

a5b25c3

CBG-3292: Handling for corrupt db config (#6377)

b35e52e

* CBG-3292: force config update on corrupt db config

Be more specific about errors during invalid response for BLIP GetAtt…

e823439

…achment (#6414)

Update XattrStore, remove subdocXattrStore (#6412)

a76bb57

* Update XattrStore, remove subdocXattrStore Remove subdocXattrStore (previously used to support working with multiple gocb versions), moving required functions not already included in XattrStore into XattrStore.

Warn -> Debug for 'gap in revision list' log (#6416)

14c0eb9

CBG-3379 add missing context logging (#6408)

2da8f9c

Co-authored-by: Ben Brooks <[email protected]>

torcolvin and others added 8 commits March 15, 2024 13:33

Skip failing tests on Couchbase 7.6 (#6731)

e3aed06

CBG-3804: Reset 3.2, make master 3.3 and rename 4.0 branch (#6732)

57caa78

* Spin master into 3.3 and continue building 4.0 on separate branch * Bump 3.2 start build * bump master API version * Move most 3.2 stats to 3.3 * Change 4.0 branch name

Update dependabot.yml to pull from private registry (#6733)

07367d2

* Update dependabot.yml to pull from private registry * Update .github/dependabot.yml Co-authored-by: Ben Brooks <[email protected]> --------- Co-authored-by: Ben Brooks <[email protected]>

CBG-3764 create multi-xattr subdoc APIs (#6739)

8b976f5

Co-authored-by: adamcfraser <[email protected]>

CBG-3764 update base implementation of XDCR in rosmar

3e3a268

- define common interfaces in sg-bucket for XDCR and related structs - move test to use NewXDCR to allow it to run the same test with CBS or rosmar cleanup: - define DataStoreName on Collection object to avoid casting

torcolvin requested review from adamcfraser and gregns1 March 28, 2024 00:55

torcolvin assigned adamcfraser and gregns1 Mar 28, 2024

torcolvin added 4 commits March 28, 2024 16:17

CBG-3857 Mark database state of starting more clearly (#6743)

cac16fb

refactor

9e53aac

update go.mod

58a0a91

Make sure the tests are collections aware

5110a10

torcolvin and others added 8 commits April 1, 2024 12:04

Fix test

2344e0e

make tests pass with 6.6 (#6751)

59c3286

- xdcr tests have timing issues with creating an XDCR bucket to bucket replication and do not run 95% of the time. - fix an error with defer bucket.Close() where the tests would panic

support golangci-lint 1.57.2 (#6754)

4316e75

- changes defaults to enabled - update golangci-lint-action to v4 to avoid node deprecation warnings from action

Merge remote-tracking branch 'origin/master' into CBG-3764

aa7d0ab

update go.mod

20840ba

bbrks force-pushed the main branch from a9811cd to 1879730 Compare April 8, 2024 11:02

torcolvin mentioned this pull request Apr 9, 2024

CBG-3703 create a basic bucket to bucket rosmar implementation #6761

Merged

4 tasks

torcolvin closed this Apr 9, 2024

torcolvin deleted the CBG-3764 branch April 9, 2024 14:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CBG-3764 update base implementation of XDCR in rosmar #6746

CBG-3764 update base implementation of XDCR in rosmar #6746

torcolvin commented Mar 28, 2024 •

edited

Loading

github-actions bot commented Apr 1, 2024 •

edited

Loading

torcolvin commented Apr 9, 2024

CBG-3764 update base implementation of XDCR in rosmar #6746

CBG-3764 update base implementation of XDCR in rosmar #6746

Conversation

torcolvin commented Mar 28, 2024 • edited Loading

Pre-review checklist

Dependencies (if applicable)

Integration Tests

github-actions bot commented Apr 1, 2024 • edited Loading

Redocly previews

torcolvin commented Apr 9, 2024

torcolvin commented Mar 28, 2024 •

edited

Loading

github-actions bot commented Apr 1, 2024 •

edited

Loading