Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/pla 331 lift input directory creation into distinct step #106

Conversation

olaughter
Copy link
Contributor

@olaughter olaughter commented Nov 27, 2024

Replace execution data with db_state file. We already pass this in to the pipeline, like so:

{
  "db_state_file_key": "input/2024-11-25T09.41.24.302062/db_state.json"
}

Meaning we can use this to skip the execution data when looking for the updates file.

Copy link

linear bot commented Nov 27, 2024

@olaughter olaughter requested a review from a team November 27, 2024 14:25
@olaughter
Copy link
Contributor Author

I tried to update the integration tests here but seems they are already broken ¯_(ツ)_/¯

@jesse-c
Copy link
Contributor

jesse-c commented Nov 27, 2024

@olaughter: I had temporarily broken them by removing some 2 year old secrets. It's fixed now!

@olaughter
Copy link
Contributor Author

@olaughter: I had temporarily broken them by removing some 2 year old secrets. It's fixed now!

Aha okay! Rerunning 👍

@olaughter
Copy link
Contributor Author

Nope still busted

@jesse-c
Copy link
Contributor

jesse-c commented Nov 27, 2024

Ah sorry, I was thinking of the navigator-data-pipeline repository!

@olaughter
Copy link
Contributor Author

Ah no worries. The pipeline repo tests do actually cover the interface here, so I'm inclined to ignore the broken test...

Copy link
Contributor

@jesse-c jesse-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

It's nice that it just needs the presence, and thus is easily changed. I've got just an open, non-blocking question.

HOW_TO_UPDATE_TESTS.md Outdated Show resolved Hide resolved
HOW_TO_UPDATE_TESTS.md Outdated Show resolved Hide resolved
src/navigator_data_ingest/main.py Show resolved Hide resolved
@olaughter olaughter merged commit ac78da6 into main Nov 27, 2024
1 of 2 checks passed
@olaughter olaughter deleted the feature/pla-331-lift-input-directory-creation-into-distinct-step branch November 27, 2024 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants