Adding resource section to GPT-J docs #21270

adit299 · 2023-01-23T20:55:23Z

What does this PR do?

Adds resources section to the GPT-J documents.

Fixes #20055 (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @stevhliu @MKhalusova

HuggingFaceDocBuilderDev · 2023-01-23T21:09:27Z

The documentation is not available anymore as the PR was closed or merged.

adit299 · 2023-01-26T17:02:28Z

Hello,

I have been currently working on finding resources for GPT-J, and mainly I have been using the links mentioned in #20055 and searching GPT-J in each of the links. I found a few links, but I feel this is not the best way to find the resources. Can you share some tips for how you were able to find more resources? @stevhliu

What I have so far:

GPT-J Description:

https://huggingface.co/EleutherAI/gpt-j-6B

Blog Posts:

NielsRogge's Transformers Tutorials:

https://github.com/kingoflolz/mesh-transformer-jax

stevhliu · 2023-01-26T18:53:18Z

Thanks for your work, that's a great start and I think you have most of them! You can also add:

This GPT-J notebook from Niels Transformers Tutorials for inference.
A chapter in the Hugging Face Course for causal language modeling.
The example scripts and notebooks for causal language modeling and text generation (see the last three bullet points under the Resource section here for GPT-2).

stevhliu

Nice, thanks for adding these! Left a few comments on formatting and using the right GPT model, after which we should be good :)

docs/source/en/model_doc/gptj.mdx

stevhliu · 2023-01-27T21:34:00Z

docs/source/en/model_doc/gptj.mdx

+- A blog post introducing GPT-J [GPT-J-6B: 6B JAX-Based Transformer](https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/) 🌎
+- A notebook for [GPT-J-6B Inference Demo](https://colab.research.google.com/github/kingoflolz/mesh-transformer-jax/blob/master/colab_demo.ipynb)
+- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the 🤗 Hugging Face Course.
+- [`GPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).


Replace GPT2 with GPTJ, same for the TF and Flax implementations

Suggested change

- [`GPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).

- [`GPTJCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).

I have updated the labels. However, looking at the links, they mention GPT-2 but GPT-J is not mentioned as being supported. For example the script mentioned here: https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling.

My questions are:

(1) I am a bit confused about what these links are. Are these just scripts that make the modeling process easier? Clarification on what these are would be appreciated.

(2) Are there any scripts present which support GPT-J? I had a look but couldn't find anything.

These are example scripts for how to finetune a model for a certain task if you prefer to just run a script instead of a notebook.

The script above for causal language modeling (CLM) should support GPT-J and all models in general with a CLM pretraining objective (so anything in the GPT family).

docs/source/en/model_doc/gptj.mdx

adit299 · 2023-01-28T23:38:21Z

It looks like the formatting for the docs is still not correct..the bulletpoints are all jumbled up. Looking into this...

stevhliu

One last typo, and the formatting looks good on my end! 👍

docs/source/en/model_doc/gptj.mdx

Co-authored-by: Steven Liu <[email protected]>

adit299 · 2023-01-30T20:11:31Z

I have marked the pull request as ready to review 👍 @stevhliu

…transformers into Add_resources_for_GPT-J Fixed one of the labels

sgugger

Thanks for adding those!

* Added resource section to GPT-J docs * Added most of the links found * Addressing review comments * Fixing formatting * Update docs/source/en/model_doc/gptj.mdx Co-authored-by: Steven Liu <[email protected]> * Fixing one of the labels --------- Co-authored-by: Steven Liu <[email protected]>

Added resource section to GPT-J docs

d1612f6

Added most of the links found

051e149

stevhliu reviewed Jan 27, 2023

View reviewed changes

adit299 added 2 commits January 28, 2023 13:37

Addressing review comments

a4ec776

Fixing formatting

6035ead

stevhliu reviewed Jan 30, 2023

View reviewed changes

docs/source/en/model_doc/gptj.mdx Outdated Show resolved Hide resolved

Update docs/source/en/model_doc/gptj.mdx

6c5cb65

Co-authored-by: Steven Liu <[email protected]>

adit299 marked this pull request as ready for review January 30, 2023 20:10

adit299 added 2 commits January 30, 2023 15:16

Fixing one of the labels

4f80de7

Merge branch 'Add_resources_for_GPT-J' of https://github.com/adit299/…

b726af2

…transformers into Add_resources_for_GPT-J Fixed one of the labels

stevhliu requested a review from sgugger January 30, 2023 21:30

sgugger approved these changes Jan 30, 2023

View reviewed changes

sgugger merged commit 914e500 into huggingface:main Jan 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding resource section to GPT-J docs #21270

Adding resource section to GPT-J docs #21270

adit299 commented Jan 23, 2023

HuggingFaceDocBuilderDev commented Jan 23, 2023 •

edited

Loading

adit299 commented Jan 26, 2023 •

edited

Loading

stevhliu commented Jan 26, 2023

stevhliu left a comment

stevhliu Jan 27, 2023

adit299 Jan 28, 2023

stevhliu Jan 30, 2023

adit299 commented Jan 28, 2023

stevhliu left a comment

adit299 commented Jan 30, 2023

sgugger left a comment

Adding resource section to GPT-J docs #21270

Adding resource section to GPT-J docs #21270

Conversation

adit299 commented Jan 23, 2023

What does this PR do?

Before submitting

Who can review?

HuggingFaceDocBuilderDev commented Jan 23, 2023 • edited Loading

adit299 commented Jan 26, 2023 • edited Loading

stevhliu commented Jan 26, 2023

stevhliu left a comment

Choose a reason for hiding this comment

stevhliu Jan 27, 2023

Choose a reason for hiding this comment

adit299 Jan 28, 2023

Choose a reason for hiding this comment

stevhliu Jan 30, 2023

Choose a reason for hiding this comment

adit299 commented Jan 28, 2023

stevhliu left a comment

Choose a reason for hiding this comment

adit299 commented Jan 30, 2023

sgugger left a comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Jan 23, 2023 •

edited

Loading

adit299 commented Jan 26, 2023 •

edited

Loading