question about how to reproduce BERT Pre-training

[weberxie]

Hi team, We are very impressive on your great work, especially about the ZeRO-Infinity!

I was trying to reproduce the BERT Pre-traning especially to enable the ZeRO-Infinity. But the tutorial confused me that, there is no declaration of the preprocesstion. It seems very different from the BERT of DeepLearningExamples. So, anyone can tell me how can I download and preprocess the data, to successfully reproduce the results?

Thanks.

[tjruwase]

@weberxie, unfortunately the bing_bert dataset is not publicly available. But there other options, please see here if any work for you.

[weberxie]

@tjruwase Thank you for your kind response.