bug: RuntimeError when run sft training with bloom-1.7b

[Modas-Li]

RuntimeError: [enforce fail at inline_container.cc:325] . unexpected pos 128 vs
0
terminate called after throwing an instance of 'c10::Error'
what(): [enforce fail at inline_container.cc:325] . unexpected pos 128 vs 0
frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x55 (0x7fb8877232f5 in /data/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: + 0x3cbbe2c (0x7fb8b7a11e2c in /data/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #2: mz_zip_writer_add_mem_ex_v2 + 0x5c5 (0x7fb8b7a0b775 in /data/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #3: caffe2::serialize::PyTorchStreamWriter::writeRecord(std::string const&, void const*, unsigned long, bool) + 0xb9 (0x7fb8b7a13419 in /data/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: caffe2::serialize::PyTorchStreamWriter::writeEndOfFile() + 0x2c3 (0x7fb8b7a138e3 in /data/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: caffe2::serialize::PyTorchStreamWriter::~PyTorchStreamWriter() + 0x125 (0x7fb8b7a13b55 in /data/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: + 0x84ecf5 (0x7fb8dfa78cf5 in /data/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x3e4272 (0x7fb8df60e272 in /data/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #8: + 0x3e51af (0x7fb8df60f1af in /data/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #9: + 0x10b92c (0x55fb9925a92c in /data/anaconda3/bin/python)
frame #10: + 0xfaf31 (0x55fb99249f31 in /data/anaconda3/bin/python)
frame #11: + 0x10bff7 (0x55fb9925aff7 in /data/anaconda3/bin/python)
frame #12: + 0x10c00d (0x55fb9925b00d in /data/anaconda3/bin/python)
frame #13: + 0x10c00d (0x55fb9925b00d in /data/anaconda3/bin/python)
frame #14: + 0x10c00d (0x55fb9925b00d in /data/anaconda3/bin/python)
frame #15: PyDict_SetItem + 0x4d2 (0x55fb992b1492 in /data/anaconda3/bin/python)
frame #16: PyDict_SetItemString + 0x4f (0x55fb992b1e5f in /data/anaconda3/bin/python)
frame #17: PyImport_Cleanup + 0x9e (0x55fb992fbd9e in /data/anaconda3/bin/python)
frame #18: Py_FinalizeEx + 0x72 (0x55fb993755e2 in /data/anaconda3/bin/python)
frame #19: + 0x236d2b (0x55fb99385d2b in /data/anaconda3/bin/python)
frame #20: _Py_UnixMain + 0x80 (0x55fb993863f0 in /data/anaconda3/bin/python)
frame #21: __libc_start_main + 0xf5 (0x7fb90902b555 in /lib64/libc.so.6)
frame #22: + 0x1e3e32 (0x55fb99332e32 in /data/anaconda3/bin/python)

[Modas-Li]

disk is full .clean