Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update bulk_validation_referential_integrity_check notebook to concur with refscan (no false positives) #796

Merged
merged 8 commits into from
Nov 26, 2024
+97 −353
Merged

Update bulk_validation_referential_integrity_check notebook to concur with refscan (no false positives) #796

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
aae9723
todo: progress
dwinston Nov 21, 2024
3aa160f
todo: progress
dwinston Nov 21, 2024
700c27c
fix: ThreadPoolExecutor is evil?
dwinston Nov 26, 2024
5369b1a
Drop the old tests2 dir
shreddd Nov 20, 2024
39e0444
Update to the newer syntax for getting utcnow() (#788)
shreddd Nov 25, 2024
2da475c
Add "Translators" subsystem as option in PR template (#767)
eecavanna Nov 23, 2024
d79f6eb
Remove reference to nonexistent `tests2` directory
eecavanna Nov 25, 2024
0a3b6b9
Merge branch 'main' into 576-prefer-any_of-union-to-range
dwinston Nov 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ mongorestore-nmdc-db:
mkdir -p /tmp/remote-mongodump/nmdc
# SSH into the remote server, stream the dump directory as a gzipped tar archive, and extract it locally.
ssh -i ~/.ssh/nersc ${NERSC_USERNAME}@dtn01.nersc.gov \
'tar -czf - -C /global/cfs/projectdirs/m3408/nmdc-mongodumps/dump_nmdc-prod_2024-07-29_20-12-07/nmdc .' \
'tar -czf - -C /global/cfs/projectdirs/m3408/nmdc-mongodumps/dump_nmdc-prod_2024-11-25_20-12-02/nmdc .' \
| tar -xzv -C /tmp/remote-mongodump/nmdc
mongorestore -v -h localhost:27018 -u admin -p root --authenticationDatabase=admin \
--drop --nsInclude='nmdc.*' --dir /tmp/remote-mongodump
Comment on lines 103 to 107
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, @dwinston, as of about 24 hours ago (via this PR), the daily dumps are being created with the --gzip CLI option to mongodump. Before that, the --gzip CLI option was not being included. So, the mongorestore command here may require a change (e.g. the addition of its --gzip CLI option—assuming you're restoring a dump that was created after about 24 hours ago).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, the mongorestore works without an explicit --gzip. smart.

Expand Down
Loading