Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Save huggingface checkpoint as artifact in mlflow callback #17686

Merged
merged 7 commits into from
Jun 17, 2022
Merged

Save huggingface checkpoint as artifact in mlflow callback #17686

merged 7 commits into from
Jun 17, 2022

Conversation

swethmandava
Copy link
Contributor

@swethmandava swethmandava commented Jun 13, 2022 •
edited
Loading

What does this PR do?

  1. Store model checkpoints including tokenizers that are needed to reload the model from mlflow as artifacts
  2. Allow model to be register-able. (they are not if log_artifacts is used to log the model)

Fixes # (issue)
#15495
#10881
#7698

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

@sgugger

@swethmandava swethmandava mentioned this pull request Jun 13, 2022
Closed
5 tasks
@swethmandava swethmandava changed the title Ckpt saving mlflow Save huggingface checkpoint as artifact in mlflow callback Jun 13, 2022
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Jun 13, 2022 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

sgugger
sgugger approved these changes Jun 14, 2022
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this! It's just missing a bit of documentation I believe.

src/transformers/integrations.py
@@ -878,6 +878,17 @@ def on_train_end(self, args, state, control, **kwargs):
if self._auto_end_run and self._ml_flow.active_run():
self._ml_flow.end_run()

def on_save(self, args, state, control, **kwargs):
if self._initialized and state.is_world_process_zero and self._log_artifacts:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this new behavior (uploading the saved model at each save) should be documented above with the HF_MLFLOW_LOG_ARTIFACTS env variable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed the documentation - thanks for your review!

@sam-h-bean sam-h-bean mentioned this pull request Jun 17, 2022
Closed
@sgugger sgugger merged commit 522a9ec into huggingface:main Jun 17, 2022
@sgugger
Copy link
Collaborator

sgugger commented Jun 17, 2022

Thanks again!

@ElefHead
Copy link

ElefHead commented Aug 5, 2022

Hi there! @swethmandava Thanks for adding this functionality. Quick question: because the artifact logging was removed, wouldn't the intermediate checkpoints not be tracked? Only the latest checkpoint would be logged as a model, right?

@swethmandava
Copy link
Contributor Author

Hi there! @swethmandava Thanks for adding this functionality. Quick question: because the artifact logging was removed, wouldn't the intermediate checkpoints not be tracked? Only the latest checkpoint would be logged as a model, right?

It should now save all the checkpoints. every time on_save is called

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgugger sgugger sgugger approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@swethmandava @HuggingFaceDocBuilderDev @sgugger @ElefHead