Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[torch.compile] directly register custom op #9896

Merged
merged 25 commits into from
Nov 1, 2024
Merged

[torch.compile] directly register custom op #9896

merged 25 commits into from
Nov 1, 2024

Conversation

youkaichao
Copy link
Member

see https://gist.github.com/youkaichao/ecbea9ec9fc79a45d2adce1784d7a9a5 for the discussion on the perf implification of registering custom ops

@youkaichao
fix register of fused moe
29e77d9 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
fix marlin
ba4f3ad 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
parallel state
19b4bff 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
fix parallel state
a09f3cb 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
add toy llama
12cac28 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
add test simple
7f0fedb 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
flash attn
15c7728 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
flashinfer
e3ca1b6 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
v1.flash_attn
f7e2966 
Signed-off-by: youkaichao <[email protected]>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@youkaichao
Copy link
Member Author

the schema is generated by using torch.library.custom_op

@youkaichao
utils
f8fe3dd 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
flashinfer
8334099 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
simple
5ec48b9 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
toy llama
449121c 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
flash attn
eca0f28 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
parallel state
dc23b25 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
marlin
cb4f157 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
fused moe
3c5e7dd 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
v1.flash_attn
0547328 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
add comments
d87bfe0 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
Copy link
Member Author

need to solve pytorch/pytorch#139444 first

@youkaichao
hack fix library
d515d61 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
fix doc build
012533a 
Signed-off-by: youkaichao <[email protected]>
tlrmchlsmth
tlrmchlsmth reviewed Nov 1, 2024
View reviewed changes
vllm/utils.py Outdated Show resolved Hide resolved
@youkaichao
fix lifetime
2ef5e40 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
add lib arg
65945e0 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
fix
529b28a 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
fix doc build
d4b1399 
Signed-off-by: youkaichao <[email protected]>
@youkaichao youkaichao added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 1, 2024
@youkaichao youkaichao merged commit 96e0c9c into vllm-project:main Nov 1, 2024
34 of 49 checks passed
@youkaichao youkaichao deleted the register branch November 1, 2024 04:56
@youkaichao youkaichao mentioned this pull request Nov 2, 2024
Merged
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Nov 4, 2024
@youkaichao
[torch.compile] directly register custom op (vllm-project#9896)
cbdb2c6 
Signed-off-by: youkaichao <[email protected]>
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Nov 4, 2024
@youkaichao
[torch.compile] directly register custom op (vllm-project#9896)
d547ee8 
Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
richardsliu pushed a commit to richardsliu/vllm that referenced this pull request Nov 4, 2024
@youkaichao @richardsliu
[torch.compile] directly register custom op (vllm-project#9896)
0458bc6 
Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Richard Liu <[email protected]>
bigPYJ1151 pushed a commit to bigPYJ1151/vllm that referenced this pull request Nov 5, 2024
@youkaichao @bigPYJ1151
[torch.compile] directly register custom op (vllm-project#9896)
c5a966c 
Signed-off-by: youkaichao <[email protected]>
hissu-hyvarinen pushed a commit to ROCm/vllm that referenced this pull request Nov 6, 2024
@youkaichao @hissu-hyvarinen
[torch.compile] directly register custom op (vllm-project#9896)
0eab1f9 
Signed-off-by: youkaichao <[email protected]>
JC1DA pushed a commit to JC1DA/vllm that referenced this pull request Nov 11, 2024
@youkaichao @JC1DA
[torch.compile] directly register custom op (vllm-project#9896)
2151ed5 
Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Loc Huynh <[email protected]>
sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024
@youkaichao @sumitd2
[torch.compile] directly register custom op (vllm-project#9896)
a9c1021 
Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Sumit Dubey <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@youkaichao @tlrmchlsmth