Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement refscan-based real-time referential integrity checking on /metadata/json:validate #835

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from
Draft

Implement refscan-based real-time referential integrity checking on /metadata/json:validate #835

wants to merge 12 commits into from

Conversation

eecavanna
Copy link
Collaborator

@eecavanna eecavanna commented Dec 12, 2024 •
edited
Loading

In this branch, I... ...will fill out this PR template later.

Details

...

Related issue(s)

...

Related subsystem(s)

  • Runtime API (except the Minter)
  • Minter
  • Dagster
  • Project documentation (in the docs directory)
  • Translators (metadata ingest pipelines)
  • MongoDB migrations
  • Other

Testing

  • I tested these changes (explain below)
  • I did not test these changes

I tested these changes by...

Documentation

  • I have not checked for relevant documentation yet (e.g. in the docs directory)
  • I have updated all relevant documentation so it will remain accurate
  • Other (explain below)

Maintainability

  • Every Python function I defined includes a docstring (test functions are exempt from this)
  • Every Python function parameter I introduced includes a type hint (e.g. study_id: str)
  • All "to do" or "fix me" Python comments I added begin with either # TODO or # FIXME
  • I used black to format all the Python files I created/modified
  • The PR title is in the imperative mood (e.g. "Do X") and not the declarative mood (e.g. "Does X" or "Did X")

@eecavanna eecavanna self-assigned this Dec 12, 2024
@eecavanna eecavanna linked an issue Dec 12, 2024 that may be closed by this pull request
Implement refscan-based real-time referential integrity validation in Runtime #831
Open
@eecavanna eecavanna mentioned this pull request Dec 13, 2024
Closed
eecavanna and others added 3 commits December 17, 2024 20:20
@eecavanna
Check referential integrity after processing _all_ specified collections
a7e73c9 
Previously, we were checking it after inserting documents into _each_
specified collection, which made it so we would not know whether a
referenced document would have been inserted into a later collection.
style: reformat
3389a4d
@eecavanna
Disable referential integrity checking in validate_json by default
bfd880f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@eecavanna eecavanna

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Implement refscan-based real-time referential integrity validation in Runtime
1 participant
@eecavanna