[enhancement] improve the performace of bloom model conversion, reduce the memory and time cost

[Yangruipis]

what can i do

1. Load model from pytorch bin / safetensors file directory, instead of call from_pretrained function of transformers, cause the from_pretrained method may take a lot of time for weight initialization, and auto covnert bf16 weights to fp32 which doubles the memory.
2. Convert every single model file in subprocess with python multiprocessing. The origin code loads entire model in main process, and pass each parameters to subprocess, this may take a lot of memory and cost more time.

conversion benchmarks

code	model	format	by-shard	nproc	elapsed(s)	memory(G)
before	bloom-175b	safetensors x 72	/	8	3390.0 [1]	700
after	bloom-175b	safetensors x 72	NO	8	1516.66	350
after	bloom-175b	safetensors x 72	YES	8	1165.03	50
after	bloom-175b	safetensors x 72	YES	24	494.81	150

[1]: from_pretrained: 1910.47, convert: 1479.53

some screenshots

[Yangruipis]

output alignment

I compared the md5 of each wegiths of model bigscience/bloom-560m, including:

1. code before this pr
2. code after this pr, disable --by-shard
3. code after this pr, enable --by-shard

And the weight files md5 are ALL strictly alignmented

screenshots

[jaedeok-nvidia]

Thank @Yangruipis to propose a significant improvement of the bloom converter. It looks very good to me.
Related to the bloom converter there is a multiprocessing stuck issue, can you please help us to fix the issue in this PR (explained in the comments)?

[Yangruipis]

sure, I'll check it later

[Yangruipis]

@kjaedeok I couldn't locate the issue or any related comments. Could you please direct me to them?

[jaedeok-nvidia]

The issue was reported directly to us, and the above workaround solved it.

[Yangruipis]

glad to hear that