[Question] Converting a Megatron-LM ckpt to Nemo

[abgoswam]

Following the GPT Pretraining section in the Megatron-LM repo, we are able to successfully train a model using Megatron-LM

I saw pointers on how to convert from HF to nemo. For example, this conversion script convert_llama_hf_to_nemo.py

However I did not see any examples of converting a ckpt saved using the Megatron-LM to nemo format . Are there any examples for this?

p.s. I am thinking of doing this conversion to nemo, so I can use tools like Nemo-Aligner for post-training

[aimarz]

There is a script for that: <NeMo_ROOT_FOLDER>/examples/nlp/language_modeling/megatron_lm_ckpt_to_nemo.py

Look up this guide: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/checkpoints/convert_mlm.html

Also this issue I opened might be relevant to you: #10480

[abgoswam]

thanks @aimarz . let me try it out on the Megatron-LM ckpts we pre-trained