[inductor] Don't specialize `split` on `sizes` parameter. #141077

ysiraichi · 2024-11-20T00:00:40Z

Stack from ghstack (oldest at bottom):

This PR modifies the lowering of split operation, so that it won't generate guards,
specializing on the sizes parameter. Instead, it specializes on the number of output
tensors being generated (i.e. function of the size of the base tensor, and the sizes
parameter).

As a result, operations such as chunk (whose number of output tensors usually is
constant given a static chunk number) won't trigger recompiles when varying the size of
the base tensor.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov

[ghstack-poisoned]

pytorch-bot · 2024-11-20T00:00:44Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141077

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

[DomainsOnly] Jobs fail with GLIBC version not found

✅ No Failures

As of commit 06644fb with merge base 12e95aa ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ezyang

nice

ezyang · 2024-11-20T00:49:27Z

@pytorchbot merge

pytorchmergebot · 2024-11-20T00:51:28Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[ghstack-poisoned]

pytorchmergebot · 2024-11-20T00:52:55Z

Rebased gh/ysiraichi/75/orig onto refs/remotes/origin/viable/strict because #141078 was rebased, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/141077)

pytorchmergebot · 2024-11-20T00:56:44Z

Merge failed

Reason: New commits were pushed while merging. Please rerun the merge command.

Details for Dev Infra team

Raised by workflow job

[ghstack-poisoned]

This PR turns clamping off for the `split` operation. By doing so, we generate less bound guards and reduce the number of recompilation when varying the input size. ```python @torch.compile(dynamic=True) def f(x): return x.chunk(4) >>> f(torch.arange(12)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([ 9, 10, 11])) >>> f(torch.arange(11)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([ 9, 10])) >>> f(torch.arange(10)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([9])) ``` Pull Request resolved: #141078 Approved by: https://github.com/ezyang ghstack dependencies: #141077

…1077) Fix: pytorch#139936 This PR modifies the lowering of `split` operation, so that it won't generate guards, specializing on the sizes parameter. Instead, it specializes on the number of output tensors being generated (i.e. function of the size of the base tensor, and the sizes parameter). As a result, operations such as `chunk` (whose number of output tensors usually is constant given a static chunk number) won't trigger recompiles when varying the size of the base tensor. Pull Request resolved: pytorch#141077 Approved by: https://github.com/ezyang

This PR turns clamping off for the `split` operation. By doing so, we generate less bound guards and reduce the number of recompilation when varying the input size. ```python @torch.compile(dynamic=True) def f(x): return x.chunk(4) >>> f(torch.arange(12)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([ 9, 10, 11])) >>> f(torch.arange(11)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([ 9, 10])) >>> f(torch.arange(10)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([9])) ``` Pull Request resolved: pytorch#141078 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#141077

…1077) Fix: pytorch#139936 This PR modifies the lowering of `split` operation, so that it won't generate guards, specializing on the sizes parameter. Instead, it specializes on the number of output tensors being generated (i.e. function of the size of the base tensor, and the sizes parameter). As a result, operations such as `chunk` (whose number of output tensors usually is constant given a static chunk number) won't trigger recompiles when varying the size of the base tensor. Pull Request resolved: pytorch#141077 Approved by: https://github.com/ezyang

This PR turns clamping off for the `split` operation. By doing so, we generate less bound guards and reduce the number of recompilation when varying the input size. ```python @torch.compile(dynamic=True) def f(x): return x.chunk(4) >>> f(torch.arange(12)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([ 9, 10, 11])) >>> f(torch.arange(11)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([ 9, 10])) >>> f(torch.arange(10)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([9])) ``` Pull Request resolved: pytorch#141078 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#141077

…1077) Fix: pytorch#139936 This PR modifies the lowering of `split` operation, so that it won't generate guards, specializing on the sizes parameter. Instead, it specializes on the number of output tensors being generated (i.e. function of the size of the base tensor, and the sizes parameter). As a result, operations such as `chunk` (whose number of output tensors usually is constant given a static chunk number) won't trigger recompiles when varying the size of the base tensor. Pull Request resolved: pytorch#141077 Approved by: https://github.com/ezyang

This PR turns clamping off for the `split` operation. By doing so, we generate less bound guards and reduce the number of recompilation when varying the input size. ```python @torch.compile(dynamic=True) def f(x): return x.chunk(4) >>> f(torch.arange(12)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([ 9, 10, 11])) >>> f(torch.arange(11)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([ 9, 10])) >>> f(torch.arange(10)) (tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([9])) ``` Pull Request resolved: pytorch#141078 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#141077

Fix: #139936 This PR modifies the lowering of `split` operation, so that it won't generate guards, specializing on the sizes parameter. Instead, it specializes on the number of output tensors being generated (i.e. function of the size of the base tensor, and the sizes parameter). As a result, operations such as `chunk` (whose number of output tensors usually is constant given a static chunk number) won't trigger recompiles when varying the size of the base tensor. ghstack-source-id: 791bb9b7265fc455db5f7d4d99dcef5f2e919e87 Pull Request resolved: pytorch/pytorch#141077

Update

b2514de

[ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: inductor labels Nov 20, 2024

ysiraichi mentioned this pull request Nov 20, 2024

[inductor] Don't clamp on split operation. #141078

Closed

ysiraichi added the topic: not user facing topic category label Nov 20, 2024

pytorchbot added the open source label Nov 20, 2024

ysiraichi requested review from ezyang, anijain2305 and bdhirsh November 20, 2024 00:17

ezyang approved these changes Nov 20, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 20, 2024

pytorchmergebot added the merging label Nov 20, 2024

Update

5873c7f

[ghstack-poisoned]

pytorchmergebot removed the merging label Nov 20, 2024

Rebased.

06644fb

[ghstack-poisoned]

pytorchmergebot closed this in 154f90f Nov 21, 2024

pytorchmergebot added the Merged label Nov 21, 2024

ezyang mentioned this pull request Nov 28, 2024

Not all values of RelaxedUnspecConstraint(L['old_my_residual_1'].size()[0]) are valid because L['old_my_residual_1'].size()[0] was inferred to be a constant (256). #141251

Closed

github-actions bot deleted the gh/ysiraichi/74/head branch December 22, 2024 02:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[inductor] Don't specialize `split` on `sizes` parameter. #141077

[inductor] Don't specialize `split` on `sizes` parameter. #141077

ysiraichi commented Nov 20, 2024 •

edited by pytorchmergebot

Loading

pytorch-bot bot commented Nov 20, 2024 •

edited

Loading

ezyang left a comment

ezyang commented Nov 20, 2024

pytorchmergebot commented Nov 20, 2024

pytorchmergebot commented Nov 20, 2024

pytorchmergebot commented Nov 20, 2024

[inductor] Don't specialize split on sizes parameter. #141077

[inductor] Don't specialize split on sizes parameter. #141077

Conversation

ysiraichi commented Nov 20, 2024 • edited by pytorchmergebot Loading

pytorch-bot bot commented Nov 20, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141077

❗ 1 Active SEVs

✅ No Failures

ezyang left a comment

Choose a reason for hiding this comment

ezyang commented Nov 20, 2024

pytorchmergebot commented Nov 20, 2024

Merge started

pytorchmergebot commented Nov 20, 2024

pytorchmergebot commented Nov 20, 2024

Merge failed

[inductor] Don't specialize `split` on `sizes` parameter. #141077

[inductor] Don't specialize `split` on `sizes` parameter. #141077

ysiraichi commented Nov 20, 2024 •

edited by pytorchmergebot

Loading

pytorch-bot bot commented Nov 20, 2024 •

edited

Loading