-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add functional transformer demo #971
base: main
Are you sure you want to change the base?
Conversation
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Hi @kiya00, thank you for writing thunder tutorials! I would be very keen to not give the impression that users need to convert their code to functional in order to run it through thunder. Is there a particular reason for starting at a functional transformer here? |
That is certainly not part of the plan. Do you have suggestions on what should be changed to avoid making this impression?
It's the simplest form of PyTorch code apart from the imperative style without any functions.
Of course, LitGPT is an example of that. |
I think it's mainly a wording thing. The initial wording looked a bit like the functional part was the key to get it to run with thunder, e.g.
seems quite odd to me. At the other end of the spectrum would be something like "Usually, you can just apply The other part is that, jitting a module and grabbing the |
Hi @t-vi @IvanYashchuk , I rephrased a bit, the main purpose of this notebook is to give an example of writing a simple functional python function for a pytorch module and thunder can also apply to this version. there's no implication that the function needs to be converted in any specific way to be compatible with Thunder. Sorry for the confusion of the initial draft. and I hope this revision more accurately conveys my intention, please help to take a look if I express what I meant |
If we run this notebook in CI using the hugging face weights, the HF_TOKEN is needed and the weight is needed to download in |
I think it would be OK to skip it in the CI. (We have not been running full models in it.) |
Is there any other popular model that is not behind a registration wall? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So first, it is a great tutorial.
I'm still not thrilled about the introduction:
These two things:
easily understood and optimized by both developers and compilers.
By the end of this notebook, you'll have a clear understanding of how functional programming principles can be leveraged to create more efficient and compiler-friendly transformer models.
are relatively dubious to me. How would a developer more easily understand and optimize a functional model? If that is so, why does PyTorch still do modular style?
Maybe we can put it more as
As part of compiling models such as LitGPT, Thunder produces a functional version - the computation trace and function - of the model. Here we implement such a functional model directly to understand what is usually done behind the scenes.
The other question I'd have is if our use of the code is OK here (did we ask the gist author, do we think that the notebook is affected by the copyright of the gist)?
I've left a message to the author on the gist, hopefully we'll get some feedback soon. |
In a v2 (absolutely not required in this PR), it might be interesting to compare functional version built here to the computation trace from jitting LitGPT. |
* Update the type signature of `rope` * Update the docstring of `rotate` Co-authored-by: beverlylytle <[email protected]>
This notebook demonstrates the acceleration of a transformer model, implemented in a functional style, using Thunder. Key highlights:
The primary objective is to explain the characteristics of Thunder-friendly code and verify functionality with loaded pre-trained weights.
The code used in the notebook is adapted from https://gist.github.com/nreHieW/a4ae05d216c5326c9fb9a70fcdda3274