-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Better TQDM output #3654
Better TQDM output #3654
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is awesome thank you ! Looking forward to see this in action
It helps a lot to understand what's happening exactly, we have many progress bars now x)
@lhoestq I've created a notebook for you to see the difference: https://colab.research.google.com/drive/1by3EqnoKvC2p-yKW4lPDGOFOZHyGVyeQ?usp=sharing. Feel free to suggest better descriptions for the progress bars. If everything looks good, think we can merge. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's awesome ! Love it :)
This PR does the following:
dataset_infos.json
exists for a dataset, usesnum_examples
to print the total number of examples that needs to be generated (inbuilder.py
)tqdm
+ multiprocessing in Jupyter Notebook/Colab (the issue stems from this commit in thetqdm
repo: tqdm/tqdm@f7722ed)drop_last_batch
andwith_ranks
params toDatasetDict.map
map
and the CSV/JSON loader whenbatched=True
to fixtqdm
progress barsbool(logging.get_verbosity() == logging.NOTSET)
(or simplifiesbool(logging.get_verbosity() == logging.NOTSET) or not utils.is_progress_bar_enabled()
tonot utils.is_progress_bar_enabled()
) condition and usesutils.is_progress_bar_enabled
to check iftqdm
output is enabled (this comment from @stas00 explains why thebool(logging.get_verbosity() == logging.NOTSET)
check is problematic: [logging] unable to turn off tqdm logging transformers#14889 (comment))Fix #2630