Make DDP master port configurable

[danpovey]

No description provided.

[pzelasko]

LGTM

[pzelasko]

After the recent DDP changes to mmi_att_transformer_train.py, I no longer see any logs in the command line -- is that expected? I ran it like python ./mmi_att_transformer_train.py --max-duration 120 --num-epochs 6 --on-the-fly-feats true --full-libri true --world-size 2

[csukuangfj]

I no longer see any logs in the command line -- is that expected?

No, there should be logs printed to the console.

[YiwenShaoStephen]

I have the same issue after using DDP.
There are some discussion in PyTorch Forum (facebookresearch/hydra#1126). But I'm not very famaliar with this topic so hopefully it will give you some information.

[danpovey]

I can't see why it might not print to the console. I do notice that the valid objf in the two halves of the validation set are very different- perhaps division is very nonrandom?

…

On Thu, Apr 15, 2021 at 5:34 AM Yiwen Shao ***@***.***> wrote: I have the same issue after using DDP. There are some discussion in PyTorch Forum (facebookresearch/hydra#1126 <facebookresearch/hydra#1126>). But I'm not very famaliar with this topic so hopefully it will give you some information. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#158 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLOZM24QJVIQUGG7GZFTTIYC6BANCNFSM42YSD3SQ> .

[pzelasko]

Probably it's getting divided over test-clean and test-other... We could either pre-shuffle the validation cut sets; add shuffle=True to the validation set sampler; or aggregate their losses by syncing the GPUs.

I haven't looked further into the printing issue yet, if I find something I will let you guys know.

[csukuangfj]

I do notice that the valid objf in the two halves of the validation set are
very different- perhaps division is very nonrandom?

That should be fixed via #162.

[pzelasko]

I have no idea why the logs are not working and my only workaround is a desperate logging.info = print in the global namespace. FYI neither the files are created nor the console is getting any outputs.

[csukuangfj]

After the recent DDP changes to mmi_att_transformer_train.py, I no longer see any logs in the command line

I have the same issue after using DDP.

@pzelasko @YiwenShaoStephen

Were both of you using torch==1.8.0?

I am encountering the same issue when using torch 1.8.0. But if I switch to torch==1.7.1, the issue is gone without
any code changes.

[pzelasko]

My torch version was 1.8.1. Interesting...