GPT2-Fineweb10B

[dustinwloring1988]

model hosted on huggingface : https://huggingface.co/AIGym/gpt2-fineweb10

The model I trained is hosted on huggingface and is available to download and use as you want. I also included the log file with it from its training.

The model was trained on 1 4060ti over 5 days. It was trained on the 10b sample from the fineweb dataset.

We have now began tokienizing the 350B sample and will be training a much bigger version with a custom bpe tokenizer.

Just thought I would share what I have so far.

[Zemulax]

Hi dustin,
would you be able to release some details on your training process? like what gpus/cpu resources you used, your hyperparameters, training period, Are you an individual or company/group?

Thank you

[dustinwloring1988]

@Zemulax yes no problem. I will post a link soon along with upload all the files to github and huggingface. I have 4 different models each with different parameters. I should be able to get this done tomorrow or the next day.
I also have 2 custom tokenizers one with 64K vocab and the other with 128K. I will be using the latter on the next model I train.

[Zemulax]

Thank you very much

[dustinwloring1988]

Here are the 2 repos I will add documentation soon.
https://huggingface.co/dustinwloring1988/GPT2-tests
https://github.com/dustinwloring1988/GPT2-Tests