Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

train_sft.py中加载tokenizer耗时太长，请问是正常的吗？ #3

Closed
70557dzqc opened this issue Jun 2, 2023 · 2 comments
Closed

train_sft.py中加载tokenizer耗时太长，请问是正常的吗？ #3

70557dzqc opened this issue Jun 2, 2023 · 2 comments
Labels
solved This problem has been already solved

Comments

@70557dzqc
Copy link

06/02/2023 06:16:54 - INFO - utils.common - Loaded tokenizer in 569.9713776111603 seconds. 接近10分钟了。
@hiyouga hiyouga added the pending This problem is yet to be addressed label Jun 2, 2023
@thaumstrial
Copy link

@panxb833 如果你的数据集很多而且cpu性能较差的话是正常的

@hiyouga hiyouga added solved This problem has been already solved and removed pending This problem is yet to be addressed labels Jun 4, 2023
@hiyouga hiyouga closed this as completed Jun 4, 2023
@hiyouga
Copy link
Owner

hiyouga commented Jun 4, 2023 •
edited
Loading

新版代码中修改了加载逻辑，应该变快了。

@DBtxy DBtxy mentioned this issue Jul 27, 2023
Closed
@godfly godfly mentioned this issue Aug 17, 2023
Closed
@HaimianYu HaimianYu mentioned this issue Nov 24, 2023
Closed
1 task
@xiechengmude xiechengmude mentioned this issue Feb 15, 2024
Closed
1 task
@alukanlp alukanlp mentioned this issue Feb 25, 2024
Closed
1 task
@feria-tu feria-tu mentioned this issue May 17, 2024
Closed
1 task
@zjxxsr zjxxsr mentioned this issue Jun 11, 2024
Closed
1 task
@Mr-Otaku-Lin Mr-Otaku-Lin mentioned this issue Jun 13, 2024
Closed
1 task
@zhoushaoxiang zhoushaoxiang mentioned this issue Jun 14, 2024
Closed
1 task
@ldknight ldknight mentioned this issue Jul 2, 2024
Closed
1 task
@xiao-liya xiao-liya mentioned this issue Jul 14, 2024
Closed
1 task
@fuqiang-benz fuqiang-benz mentioned this issue Jul 15, 2024
Closed
1 task
@hecheng64 hecheng64 mentioned this issue Sep 16, 2024
Closed
1 task
@Hansen06 Hansen06 mentioned this issue Sep 27, 2024
Closed
1 task
@alphanlp alphanlp mentioned this issue Oct 5, 2024
Closed
1 task
@hiennguyennq hiennguyennq mentioned this issue Oct 21, 2024
Open
This was referenced Nov 23, 2024
Closed
Closed
Closed
@bisque-qwe bisque-qwe mentioned this issue Dec 6, 2024
Closed
1 task
@asdksadsad asdksadsad mentioned this issue Dec 19, 2024
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
solved This problem has been already solved
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hiyouga @70557dzqc @thaumstrial