Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

disable codecarbon as it's very unstable #152

Merged
merged 1 commit into from
Oct 22, 2021
Merged

disable codecarbon as it's very unstable #152

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions megatron/global_vars.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,11 @@ def _set_tensorboard_writer(args):
'no TensorBoard logs will be written.', flush=True)


# Important: the codecarbon is very unstable and its latest incarnation using the python scheduler interferes with the asyncio library we use in the test suite which breaks everything, so making this a no-op for now.
def _set_codecarbon_tracker(args):

return # turned off

global _GLOBAL_CODECARBON_TRACKER
if not hasattr(args, 'codecarbon_dir') or args.codecarbon_dir is None:
return
Expand Down Expand Up @@ -187,6 +191,9 @@ def _set_codecarbon_tracker(args):


def codecarbon_tracker_start():

return # turned off, see the notes above

global _GLOBAL_CODECARBON_TRACKER
if _GLOBAL_CODECARBON_TRACKER is None:
return
Expand All @@ -196,6 +203,9 @@ def codecarbon_tracker_start():


def codecarbon_tracker_stop():

return # turned off, see the notes above

global _GLOBAL_CODECARBON_TRACKER
if _GLOBAL_CODECARBON_TRACKER is None:
return
Expand All @@ -205,6 +215,9 @@ def codecarbon_tracker_stop():


def codecarbon_tracker_flush():

return # turned off, see the notes above

global _GLOBAL_CODECARBON_TRACKER
if _GLOBAL_CODECARBON_TRACKER is None:
return
Expand Down
2 changes: 0 additions & 2 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ tensorboard
torch
transformers
DeepSpeed @ git+https://github.com/microsoft/DeepSpeed.git
# at some point when it starts working freeze with ether min version or sha using the syntax codecarbon.git@deadbeaf
codecarbon @ git+https://github.com/mlco2/codecarbon.git
# versions from HF transformers
black==21.4b0
isort>=5.5.4
2 changes: 0 additions & 2 deletions tests/test_training.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,6 @@ def get_variation_config(self, variation, output_dir):
--save {output_dir}/checkpoints
--load {output_dir}/checkpoints
--data-path {data_dir}/meg-gpt2-openwebtext_text_document
--codecarbon-dir {output_dir}/codecarbon
--tensorboard-dir {output_dir}/tensorboard
--tensorboard-queue-size 5
--log-timers-to-tensorboard
Expand Down Expand Up @@ -314,7 +313,6 @@ def test_training_prefix_lm_all(self):
--save {output_dir}/checkpoints
--load {output_dir}/checkpoints
--data-path {data_dir}/meg-gpt2-openwebtext_text_document
--codecarbon-dir {output_dir}/codecarbon
--tensorboard-dir {output_dir}/tensorboard
--tensorboard-queue-size 5
--log-timers-to-tensorboard
Expand Down