Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Depend on torch nightly for nightly releases #599

Merged
merged 1 commit into from
Dec 15, 2023
Merged

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Dec 15, 2023

No description provided.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 15, 2023
@vmoens vmoens added the CI label Dec 15, 2023
@vmoens vmoens marked this pull request as ready for review December 15, 2023 14:19
@vmoens vmoens merged commit 68e8f40 into main Dec 15, 2023
25 of 33 checks passed
Copy link

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 120. Improved: $\large\color{#35bf28}8$. Worsened: $\large\color{#d91a1a}10$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_plain_set_nested 63.2970μs 17.0623μs 58.6088 KOps/s 61.7908 KOps/s $\textbf{\color{#d91a1a}-5.15\%}$
test_plain_set_stack_nested 0.1976ms 0.1496ms 6.6841 KOps/s 7.0844 KOps/s $\textbf{\color{#d91a1a}-5.65\%}$
test_plain_set_nested_inplace 45.2240μs 19.1572μs 52.1997 KOps/s 53.8341 KOps/s $\color{#d91a1a}-3.04\%$
test_plain_set_stack_nested_inplace 0.2615ms 0.1797ms 5.5635 KOps/s 5.7417 KOps/s $\color{#d91a1a}-3.10\%$
test_items 15.3790μs 2.3863μs 419.0614 KOps/s 413.4569 KOps/s $\color{#35bf28}+1.36\%$
test_items_nested 0.4634ms 0.2667ms 3.7490 KOps/s 3.7787 KOps/s $\color{#d91a1a}-0.78\%$
test_items_nested_locked 1.4075ms 0.2680ms 3.7309 KOps/s 3.7598 KOps/s $\color{#d91a1a}-0.77\%$
test_items_nested_leaf 0.5486ms 0.1660ms 6.0242 KOps/s 6.0793 KOps/s $\color{#d91a1a}-0.91\%$
test_items_stack_nested 2.2228ms 1.4745ms 678.2053 Ops/s 691.8689 Ops/s $\color{#d91a1a}-1.97\%$
test_items_stack_nested_leaf 2.1284ms 1.3348ms 749.1814 Ops/s 764.6562 Ops/s $\color{#d91a1a}-2.02\%$
test_items_stack_nested_locked 1.7948ms 0.7659ms 1.3056 KOps/s 1.3304 KOps/s $\color{#d91a1a}-1.86\%$
test_keys 34.1430μs 3.8838μs 257.4796 KOps/s 251.6424 KOps/s $\color{#35bf28}+2.32\%$
test_keys_nested 0.5828ms 0.1404ms 7.1205 KOps/s 6.7693 KOps/s $\textbf{\color{#35bf28}+5.19\%}$
test_keys_nested_locked 0.1969ms 0.1390ms 7.1962 KOps/s 7.1264 KOps/s $\color{#35bf28}+0.98\%$
test_keys_nested_leaf 0.2470ms 0.1393ms 7.1763 KOps/s 7.2142 KOps/s $\color{#d91a1a}-0.52\%$
test_keys_stack_nested 1.5036ms 1.3942ms 717.2773 Ops/s 701.7859 Ops/s $\color{#35bf28}+2.21\%$
test_keys_stack_nested_leaf 1.5051ms 1.3910ms 718.9208 Ops/s 728.2543 Ops/s $\color{#d91a1a}-1.28\%$
test_keys_stack_nested_locked 0.7922ms 0.6688ms 1.4952 KOps/s 1.5185 KOps/s $\color{#d91a1a}-1.54\%$
test_values 8.0550μs 1.0876μs 919.4492 KOps/s 881.6514 KOps/s $\color{#35bf28}+4.29\%$
test_values_nested 88.7250μs 49.3824μs 20.2501 KOps/s 20.4909 KOps/s $\color{#d91a1a}-1.18\%$
test_values_nested_locked 0.1343ms 49.4576μs 20.2194 KOps/s 20.3164 KOps/s $\color{#d91a1a}-0.48\%$
test_values_nested_leaf 87.0620μs 44.7426μs 22.3500 KOps/s 22.7380 KOps/s $\color{#d91a1a}-1.71\%$
test_values_stack_nested 1.2890ms 1.1943ms 837.2774 Ops/s 855.5124 Ops/s $\color{#d91a1a}-2.13\%$
test_values_stack_nested_leaf 1.3077ms 1.1807ms 846.9325 Ops/s 864.1747 Ops/s $\color{#d91a1a}-2.00\%$
test_values_stack_nested_locked 0.7172ms 0.5079ms 1.9688 KOps/s 2.0074 KOps/s $\color{#d91a1a}-1.92\%$
test_membership 14.0960μs 1.3242μs 755.1596 KOps/s 738.6643 KOps/s $\color{#35bf28}+2.23\%$
test_membership_nested 41.4470μs 2.7569μs 362.7298 KOps/s 364.2488 KOps/s $\color{#d91a1a}-0.42\%$
test_membership_nested_leaf 20.8790μs 2.7652μs 361.6398 KOps/s 356.7572 KOps/s $\color{#35bf28}+1.37\%$
test_membership_stacked_nested 33.3820μs 11.7472μs 85.1269 KOps/s 86.7990 KOps/s $\color{#d91a1a}-1.93\%$
test_membership_stacked_nested_leaf 48.7010μs 11.7677μs 84.9784 KOps/s 86.8596 KOps/s $\color{#d91a1a}-2.17\%$
test_membership_nested_last 22.0710μs 5.8008μs 172.3901 KOps/s 171.9734 KOps/s $\color{#35bf28}+0.24\%$
test_membership_nested_leaf_last 37.5900μs 5.8156μs 171.9513 KOps/s 170.1294 KOps/s $\color{#35bf28}+1.07\%$
test_membership_stacked_nested_last 0.2603ms 0.1636ms 6.1122 KOps/s 6.0833 KOps/s $\color{#35bf28}+0.48\%$
test_membership_stacked_nested_leaf_last 59.2400μs 13.8027μs 72.4497 KOps/s 73.4514 KOps/s $\color{#d91a1a}-1.36\%$
test_nested_getleaf 32.3600μs 10.6404μs 93.9812 KOps/s 93.7584 KOps/s $\color{#35bf28}+0.24\%$
test_nested_get 46.4560μs 10.0452μs 99.5500 KOps/s 99.2130 KOps/s $\color{#35bf28}+0.34\%$
test_stacked_getleaf 0.9973ms 0.6374ms 1.5690 KOps/s 1.5960 KOps/s $\color{#d91a1a}-1.70\%$
test_stacked_get 0.6862ms 0.6102ms 1.6387 KOps/s 1.6757 KOps/s $\color{#d91a1a}-2.21\%$
test_nested_getitemleaf 36.7780μs 10.5979μs 94.3582 KOps/s 94.1947 KOps/s $\color{#35bf28}+0.17\%$
test_nested_getitem 37.4590μs 10.1140μs 98.8727 KOps/s 99.7682 KOps/s $\color{#d91a1a}-0.90\%$
test_stacked_getitemleaf 0.8067ms 0.6382ms 1.5669 KOps/s 1.5857 KOps/s $\color{#d91a1a}-1.18\%$
test_stacked_getitem 0.6856ms 0.6089ms 1.6423 KOps/s 1.6473 KOps/s $\color{#d91a1a}-0.30\%$
test_lock_nested 0.8977ms 0.4098ms 2.4403 KOps/s 2.4172 KOps/s $\color{#35bf28}+0.95\%$
test_lock_stack_nested 59.2521ms 5.9878ms 167.0073 Ops/s 166.8523 Ops/s $\color{#35bf28}+0.09\%$
test_unlock_nested 0.9933ms 0.4190ms 2.3867 KOps/s 2.1202 KOps/s $\textbf{\color{#35bf28}+12.57\%}$
test_unlock_stack_nested 56.1245ms 5.7480ms 173.9735 Ops/s 175.8826 Ops/s $\color{#d91a1a}-1.09\%$
test_flatten_speed 0.3803ms 0.2682ms 3.7287 KOps/s 3.7793 KOps/s $\color{#d91a1a}-1.34\%$
test_unflatten_speed 0.7411ms 0.4558ms 2.1941 KOps/s 2.2604 KOps/s $\color{#d91a1a}-2.94\%$
test_common_ops 1.1655ms 0.6903ms 1.4487 KOps/s 1.4949 KOps/s $\color{#d91a1a}-3.09\%$
test_creation 58.4190μs 2.1242μs 470.7719 KOps/s 508.4185 KOps/s $\textbf{\color{#d91a1a}-7.40\%}$
test_creation_empty 29.5650μs 9.7485μs 102.5798 KOps/s 107.2530 KOps/s $\color{#d91a1a}-4.36\%$
test_creation_nested_1 44.7530μs 12.7158μs 78.6421 KOps/s 82.9485 KOps/s $\textbf{\color{#d91a1a}-5.19\%}$
test_creation_nested_2 90.5180μs 18.1611μs 55.0627 KOps/s 58.0617 KOps/s $\textbf{\color{#d91a1a}-5.17\%}$
test_clone 0.1233ms 12.2795μs 81.4368 KOps/s 80.5024 KOps/s $\color{#35bf28}+1.16\%$
test_getitem[int] 32.3700μs 11.9733μs 83.5193 KOps/s 83.4314 KOps/s $\color{#35bf28}+0.11\%$
test_getitem[slice_int] 80.3300μs 24.1435μs 41.4190 KOps/s 42.7272 KOps/s $\color{#d91a1a}-3.06\%$
test_getitem[range] 0.1091ms 43.2300μs 23.1321 KOps/s 23.4295 KOps/s $\color{#d91a1a}-1.27\%$
test_getitem[tuple] 45.9250μs 18.9989μs 52.6345 KOps/s 52.9909 KOps/s $\color{#d91a1a}-0.67\%$
test_getitem[list] 0.3399ms 37.6157μs 26.5847 KOps/s 26.5029 KOps/s $\color{#35bf28}+0.31\%$
test_setitem_dim[int] 58.3490μs 30.5514μs 32.7317 KOps/s 34.1268 KOps/s $\color{#d91a1a}-4.09\%$
test_setitem_dim[slice_int] 91.0290μs 55.6661μs 17.9643 KOps/s 18.3656 KOps/s $\color{#d91a1a}-2.19\%$
test_setitem_dim[range] 0.1201ms 73.8495μs 13.5411 KOps/s 13.8769 KOps/s $\color{#d91a1a}-2.42\%$
test_setitem_dim[tuple] 86.0600μs 44.2077μs 22.6205 KOps/s 23.7005 KOps/s $\color{#d91a1a}-4.56\%$
test_setitem 0.1320ms 18.5200μs 53.9958 KOps/s 55.1371 KOps/s $\color{#d91a1a}-2.07\%$
test_set 0.1250ms 17.7032μs 56.4868 KOps/s 56.7279 KOps/s $\color{#d91a1a}-0.42\%$
test_set_shared 3.1718ms 0.1383ms 7.2289 KOps/s 7.4439 KOps/s $\color{#d91a1a}-2.89\%$
test_update 0.1274ms 20.6610μs 48.4004 KOps/s 50.8985 KOps/s $\color{#d91a1a}-4.91\%$
test_update_nested 0.1298ms 28.0099μs 35.7017 KOps/s 37.0641 KOps/s $\color{#d91a1a}-3.68\%$
test_set_nested 0.1096ms 19.6104μs 50.9934 KOps/s 52.1310 KOps/s $\color{#d91a1a}-2.18\%$
test_set_nested_new 73.2670μs 24.0997μs 41.4943 KOps/s 42.7534 KOps/s $\color{#d91a1a}-2.95\%$
test_select 93.0430μs 47.7579μs 20.9389 KOps/s 21.2020 KOps/s $\color{#d91a1a}-1.24\%$
test_unbind_speed 0.7704ms 0.3398ms 2.9431 KOps/s 2.9600 KOps/s $\color{#d91a1a}-0.57\%$
test_unbind_speed_stack0 61.2575ms 4.1993ms 238.1323 Ops/s 224.5936 Ops/s $\textbf{\color{#35bf28}+6.03\%}$
test_unbind_speed_stack1 2.0041μs 0.6234μs 1.6042 MOps/s 1.5807 MOps/s $\color{#35bf28}+1.49\%$
test_split 1.9102ms 1.5221ms 656.9687 Ops/s 638.3296 Ops/s $\color{#35bf28}+2.92\%$
test_chunk 55.9309ms 1.6154ms 619.0352 Ops/s 609.9745 Ops/s $\color{#35bf28}+1.49\%$
test_creation[device0] 0.3875ms 0.2887ms 3.4642 KOps/s 3.0636 KOps/s $\textbf{\color{#35bf28}+13.07\%}$
test_creation_from_tensor 3.3240ms 0.3301ms 3.0290 KOps/s 3.0730 KOps/s $\color{#d91a1a}-1.43\%$
test_add_one[memmap_tensor0] 71.1820μs 25.3252μs 39.4864 KOps/s 39.2346 KOps/s $\color{#35bf28}+0.64\%$
test_contiguous[memmap_tensor0] 31.5490μs 5.9169μs 169.0064 KOps/s 169.7921 KOps/s $\color{#d91a1a}-0.46\%$
test_stack[memmap_tensor0] 49.6020μs 19.3525μs 51.6730 KOps/s 50.6133 KOps/s $\color{#35bf28}+2.09\%$
test_memmaptd_index 0.4446ms 0.2054ms 4.8684 KOps/s 5.1206 KOps/s $\color{#d91a1a}-4.93\%$
test_memmaptd_index_astensor 0.6280ms 0.2629ms 3.8038 KOps/s 3.9084 KOps/s $\color{#d91a1a}-2.68\%$
test_memmaptd_index_op 0.6187ms 0.5362ms 1.8650 KOps/s 1.9233 KOps/s $\color{#d91a1a}-3.03\%$
test_serialize_model 0.1586s 0.1040s 9.6130 Ops/s 9.1293 Ops/s $\textbf{\color{#35bf28}+5.30\%}$
test_serialize_model_filesystem 0.1485s 97.6733ms 10.2382 Ops/s 10.7272 Ops/s $\color{#d91a1a}-4.56\%$
test_serialize_model_pickle 0.4483s 0.3762s 2.6582 Ops/s 2.6232 Ops/s $\color{#35bf28}+1.33\%$
test_serialize_weights 0.1594s 0.1023s 9.7789 Ops/s 10.3650 Ops/s $\textbf{\color{#d91a1a}-5.65\%}$
test_serialize_weights_filesystem 95.8406ms 88.2599ms 11.3302 Ops/s 10.1129 Ops/s $\textbf{\color{#35bf28}+12.04\%}$
test_serialize_weights_returnearly 0.3958s 0.1553s 6.4402 Ops/s 8.2386 Ops/s $\textbf{\color{#d91a1a}-21.83\%}$
test_serialize_weights_pickle 1.1534s 0.6318s 1.5829 Ops/s 1.4099 Ops/s $\textbf{\color{#35bf28}+12.27\%}$
test_reshape_pytree 57.7370μs 22.9683μs 43.5383 KOps/s 44.1765 KOps/s $\color{#d91a1a}-1.44\%$
test_reshape_td 59.3100μs 30.8636μs 32.4007 KOps/s 34.1904 KOps/s $\textbf{\color{#d91a1a}-5.23\%}$
test_view_pytree 72.4950μs 22.8709μs 43.7237 KOps/s 43.4817 KOps/s $\color{#35bf28}+0.56\%$
test_view_td 43.0090μs 4.8269μs 207.1713 KOps/s 206.5001 KOps/s $\color{#35bf28}+0.33\%$
test_unbind_pytree 61.7450μs 26.4338μs 37.8304 KOps/s 38.0942 KOps/s $\color{#d91a1a}-0.69\%$
test_unbind_td 0.1332ms 54.4398μs 18.3689 KOps/s 18.1867 KOps/s $\color{#35bf28}+1.00\%$
test_split_pytree 89.9470μs 26.3778μs 37.9106 KOps/s 38.7049 KOps/s $\color{#d91a1a}-2.05\%$
test_split_td 0.6035ms 42.9253μs 23.2963 KOps/s 23.5096 KOps/s $\color{#d91a1a}-0.91\%$
test_add_pytree 90.7590μs 32.0767μs 31.1753 KOps/s 31.5263 KOps/s $\color{#d91a1a}-1.11\%$
test_add_td 0.1555ms 48.5922μs 20.5794 KOps/s 21.9557 KOps/s $\textbf{\color{#d91a1a}-6.27\%}$
test_distributed 18.2540μs 6.1322μs 163.0743 KOps/s 166.6194 KOps/s $\color{#d91a1a}-2.13\%$
test_tdmodule 0.9251ms 23.4201μs 42.6984 KOps/s 42.9610 KOps/s $\color{#d91a1a}-0.61\%$
test_tdmodule_dispatch 0.2046ms 41.6318μs 24.0201 KOps/s 25.3479 KOps/s $\textbf{\color{#d91a1a}-5.24\%}$
test_tdseq 54.4410μs 26.2703μs 38.0658 KOps/s 39.8991 KOps/s $\color{#d91a1a}-4.59\%$
test_tdseq_dispatch 0.1404ms 46.2986μs 21.5989 KOps/s 22.4264 KOps/s $\color{#d91a1a}-3.69\%$
test_instantiation_functorch 1.6228ms 1.3157ms 760.0794 Ops/s 775.9173 Ops/s $\color{#d91a1a}-2.04\%$
test_instantiation_td 1.4808ms 1.0108ms 989.2986 Ops/s 937.5960 Ops/s $\textbf{\color{#35bf28}+5.51\%}$
test_exec_functorch 0.2389ms 0.1563ms 6.3995 KOps/s 6.2787 KOps/s $\color{#35bf28}+1.92\%$
test_exec_functional_call 0.2347ms 0.1437ms 6.9601 KOps/s 6.7173 KOps/s $\color{#35bf28}+3.62\%$
test_exec_td 0.2232ms 0.1407ms 7.1088 KOps/s 6.9938 KOps/s $\color{#35bf28}+1.64\%$
test_exec_td_decorator 0.6376ms 0.1714ms 5.8334 KOps/s 5.8113 KOps/s $\color{#35bf28}+0.38\%$
test_vmap_mlp_speed[True-True] 1.0292ms 0.8798ms 1.1366 KOps/s 1.1277 KOps/s $\color{#35bf28}+0.79\%$
test_vmap_mlp_speed[True-False] 0.6304ms 0.4702ms 2.1269 KOps/s 2.1167 KOps/s $\color{#35bf28}+0.48\%$
test_vmap_mlp_speed[False-True] 1.4913ms 0.7992ms 1.2512 KOps/s 1.2752 KOps/s $\color{#d91a1a}-1.88\%$
test_vmap_mlp_speed[False-False] 0.6173ms 0.3827ms 2.6130 KOps/s 2.5939 KOps/s $\color{#35bf28}+0.74\%$
test_vmap_mlp_speed_decorator[True-True] 3.8345ms 1.7750ms 563.3866 Ops/s 569.3227 Ops/s $\color{#d91a1a}-1.04\%$
test_vmap_mlp_speed_decorator[True-False] 0.7949ms 0.5190ms 1.9268 KOps/s 1.9296 KOps/s $\color{#d91a1a}-0.15\%$
test_vmap_mlp_speed_decorator[False-True] 1.9058ms 1.4683ms 681.0480 Ops/s 679.8040 Ops/s $\color{#35bf28}+0.18\%$
test_vmap_mlp_speed_decorator[False-False] 0.6816ms 0.3954ms 2.5292 KOps/s 2.5234 KOps/s $\color{#35bf28}+0.23\%$

@vmoens vmoens deleted the fix-nightly-torch branch December 15, 2023 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants