Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Refactor] Refactor functional calls in losses #1707

Merged
merged 8 commits into from
Nov 24, 2023
Merged

[Refactor] Refactor functional calls in losses #1707

merged 8 commits into from
Nov 24, 2023

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Nov 21, 2023 •
edited
Loading

Refactors functional calls in losses to

with params_and_buffers.to_module(module):
    module(data)

The main advantage is that it does not require any modification of the module anymore, which will for instance solve #1613.

cc @matteobettini @skandermoalla @albertbou92 @BY571

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Nov 21, 2023 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1707

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (8 Unrelated Failures)

As of commit 3c2e007 with merge base fa149e4 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 21, 2023
@vmoens vmoens added the Refactoring Refactoring of an existing feature label Nov 22, 2023
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 22, 2023 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}9$. Worsened: $\large\color{#d91a1a}13$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 63.3544ms 62.6579ms 15.9597 Ops/s 15.3488 Ops/s $\color{#35bf28}+3.98\%$
test_sync 40.5161ms 34.1434ms 29.2882 Ops/s 27.7791 Ops/s $\textbf{\color{#35bf28}+5.43\%}$
test_async 47.8996ms 32.0428ms 31.2082 Ops/s 30.9770 Ops/s $\color{#35bf28}+0.75\%$
test_simple 0.4837s 0.4325s 2.3124 Ops/s 2.3297 Ops/s $\color{#d91a1a}-0.75\%$
test_transformed 0.6314s 0.5920s 1.6891 Ops/s 1.6958 Ops/s $\color{#d91a1a}-0.40\%$
test_serial 1.3423s 1.2956s 0.7718 Ops/s 0.7773 Ops/s $\color{#d91a1a}-0.70\%$
test_parallel 1.3421s 1.2970s 0.7710 Ops/s 0.7542 Ops/s $\color{#35bf28}+2.23\%$
test_step_mdp_speed[True-True-True-True-True] 0.1339ms 22.6870μs 44.0781 KOps/s 43.4046 KOps/s $\color{#35bf28}+1.55\%$
test_step_mdp_speed[True-True-True-True-False] 54.7420μs 13.7655μs 72.6456 KOps/s 72.5102 KOps/s $\color{#35bf28}+0.19\%$
test_step_mdp_speed[True-True-True-False-True] 39.6540μs 14.0702μs 71.0721 KOps/s 71.9249 KOps/s $\color{#d91a1a}-1.19\%$
test_step_mdp_speed[True-True-True-False-False] 31.7900μs 8.3401μs 119.9026 KOps/s 119.7041 KOps/s $\color{#35bf28}+0.17\%$
test_step_mdp_speed[True-True-False-True-True] 57.8280μs 24.0198μs 41.6324 KOps/s 41.2584 KOps/s $\color{#35bf28}+0.91\%$
test_step_mdp_speed[True-True-False-True-False] 60.3200μs 14.9604μs 66.8433 KOps/s 66.0042 KOps/s $\color{#35bf28}+1.27\%$
test_step_mdp_speed[True-True-False-False-True] 56.4670μs 15.0768μs 66.3272 KOps/s 65.6781 KOps/s $\color{#35bf28}+0.99\%$
test_step_mdp_speed[True-True-False-False-False] 57.3670μs 9.7672μs 102.3835 KOps/s 103.3892 KOps/s $\color{#d91a1a}-0.97\%$
test_step_mdp_speed[True-False-True-True-True] 67.0650μs 25.7146μs 38.8883 KOps/s 38.9500 KOps/s $\color{#d91a1a}-0.16\%$
test_step_mdp_speed[True-False-True-True-False] 65.5930μs 17.3185μs 57.7416 KOps/s 60.8041 KOps/s $\textbf{\color{#d91a1a}-5.04\%}$
test_step_mdp_speed[True-False-True-False-True] 47.6380μs 15.1195μs 66.1396 KOps/s 64.2512 KOps/s $\color{#35bf28}+2.94\%$
test_step_mdp_speed[True-False-True-False-False] 55.2730μs 9.5909μs 104.2652 KOps/s 103.7093 KOps/s $\color{#35bf28}+0.54\%$
test_step_mdp_speed[True-False-False-True-True] 63.6790μs 26.9422μs 37.1165 KOps/s 36.9628 KOps/s $\color{#35bf28}+0.42\%$
test_step_mdp_speed[True-False-False-True-False] 44.7240μs 17.4325μs 57.3641 KOps/s 56.7181 KOps/s $\color{#35bf28}+1.14\%$
test_step_mdp_speed[True-False-False-False-True] 49.0110μs 16.5356μs 60.4757 KOps/s 60.3719 KOps/s $\color{#35bf28}+0.17\%$
test_step_mdp_speed[True-False-False-False-False] 36.8390μs 10.7846μs 92.7248 KOps/s 92.7006 KOps/s $\color{#35bf28}+0.03\%$
test_step_mdp_speed[False-True-True-True-True] 0.1089ms 25.8475μs 38.6884 KOps/s 38.5702 KOps/s $\color{#35bf28}+0.31\%$
test_step_mdp_speed[False-True-True-True-False] 54.6620μs 16.4373μs 60.8374 KOps/s 60.7207 KOps/s $\color{#35bf28}+0.19\%$
test_step_mdp_speed[False-True-True-False-True] 60.8530μs 17.5315μs 57.0403 KOps/s 56.3269 KOps/s $\color{#35bf28}+1.27\%$
test_step_mdp_speed[False-True-True-False-False] 32.4000μs 10.9155μs 91.6126 KOps/s 92.3593 KOps/s $\color{#d91a1a}-0.81\%$
test_step_mdp_speed[False-True-False-True-True] 80.3580μs 26.7954μs 37.3198 KOps/s 36.7932 KOps/s $\color{#35bf28}+1.43\%$
test_step_mdp_speed[False-True-False-True-False] 44.2120μs 17.6302μs 56.7208 KOps/s 56.3968 KOps/s $\color{#35bf28}+0.57\%$
test_step_mdp_speed[False-True-False-False-True] 50.1830μs 18.6442μs 53.6360 KOps/s 52.6929 KOps/s $\color{#35bf28}+1.79\%$
test_step_mdp_speed[False-True-False-False-False] 40.2650μs 12.0134μs 83.2401 KOps/s 83.4612 KOps/s $\color{#d91a1a}-0.26\%$
test_step_mdp_speed[False-False-True-True-True] 66.2930μs 28.0213μs 35.6871 KOps/s 35.2003 KOps/s $\color{#35bf28}+1.38\%$
test_step_mdp_speed[False-False-True-True-False] 42.9000μs 18.9249μs 52.8404 KOps/s 52.8632 KOps/s $\color{#d91a1a}-0.04\%$
test_step_mdp_speed[False-False-True-False-True] 73.1860μs 18.7621μs 53.2989 KOps/s 52.5684 KOps/s $\color{#35bf28}+1.39\%$
test_step_mdp_speed[False-False-True-False-False] 58.9500μs 12.0344μs 83.0954 KOps/s 83.4055 KOps/s $\color{#d91a1a}-0.37\%$
test_step_mdp_speed[False-False-False-True-True] 67.5360μs 28.9052μs 34.5958 KOps/s 33.7026 KOps/s $\color{#35bf28}+2.65\%$
test_step_mdp_speed[False-False-False-True-False] 68.4870μs 20.0181μs 49.9549 KOps/s 49.5999 KOps/s $\color{#35bf28}+0.72\%$
test_step_mdp_speed[False-False-False-False-True] 55.0130μs 19.5721μs 51.0931 KOps/s 49.8158 KOps/s $\color{#35bf28}+2.56\%$
test_step_mdp_speed[False-False-False-False-False] 51.3760μs 13.2075μs 75.7147 KOps/s 75.3650 KOps/s $\color{#35bf28}+0.46\%$
test_values[generalized_advantage_estimate-True-True] 12.5728ms 12.2282ms 81.7783 Ops/s 81.5674 Ops/s $\color{#35bf28}+0.26\%$
test_values[vec_generalized_advantage_estimate-True-True] 35.0759ms 26.4636ms 37.7877 Ops/s 37.6916 Ops/s $\color{#35bf28}+0.26\%$
test_values[td0_return_estimate-False-False] 0.9078ms 0.2027ms 4.9338 KOps/s 5.5901 KOps/s $\textbf{\color{#d91a1a}-11.74\%}$
test_values[td1_return_estimate-False-False] 27.5159ms 26.1928ms 38.1784 Ops/s 39.4200 Ops/s $\color{#d91a1a}-3.15\%$
test_values[vec_td1_return_estimate-False-False] 33.8212ms 26.7074ms 37.4428 Ops/s 37.3953 Ops/s $\color{#35bf28}+0.13\%$
test_values[td_lambda_return_estimate-True-False] 38.6641ms 36.0636ms 27.7288 Ops/s 28.1621 Ops/s $\color{#d91a1a}-1.54\%$
test_values[vec_td_lambda_return_estimate-True-False] 34.0609ms 26.5706ms 37.6356 Ops/s 37.1114 Ops/s $\color{#35bf28}+1.41\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 8.4218ms 8.2214ms 121.6340 Ops/s 124.6551 Ops/s $\color{#d91a1a}-2.42\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 3.2025ms 1.9703ms 507.5448 Ops/s 531.3416 Ops/s $\color{#d91a1a}-4.48\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 7.9911ms 0.4393ms 2.2766 KOps/s 2.3709 KOps/s $\color{#d91a1a}-3.98\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 45.9765ms 39.3389ms 25.4201 Ops/s 24.5393 Ops/s $\color{#35bf28}+3.59\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 10.0811ms 2.5008ms 399.8694 Ops/s 397.3265 Ops/s $\color{#35bf28}+0.64\%$
test_dqn_speed 64.1915ms 1.8736ms 533.7431 Ops/s 696.7229 Ops/s $\textbf{\color{#d91a1a}-23.39\%}$
test_ddpg_speed 11.3024ms 3.9468ms 253.3718 Ops/s 330.5113 Ops/s $\textbf{\color{#d91a1a}-23.34\%}$
test_sac_speed 19.5163ms 10.9023ms 91.7236 Ops/s 114.0478 Ops/s $\textbf{\color{#d91a1a}-19.57\%}$
test_redq_speed 24.8237ms 17.6000ms 56.8182 Ops/s 66.6276 Ops/s $\textbf{\color{#d91a1a}-14.72\%}$
test_redq_deprec_speed 24.3701ms 15.9693ms 62.6203 Ops/s 74.2060 Ops/s $\textbf{\color{#d91a1a}-15.61\%}$
test_td3_speed 20.1727ms 11.3523ms 88.0882 Ops/s 104.7451 Ops/s $\textbf{\color{#d91a1a}-15.90\%}$
test_cql_speed 47.9391ms 41.9490ms 23.8385 Ops/s 25.6680 Ops/s $\textbf{\color{#d91a1a}-7.13\%}$
test_a2c_speed 17.1457ms 8.9578ms 111.6346 Ops/s 122.9191 Ops/s $\textbf{\color{#d91a1a}-9.18\%}$
test_ppo_speed 17.3824ms 9.2756ms 107.8092 Ops/s 121.0411 Ops/s $\textbf{\color{#d91a1a}-10.93\%}$
test_reinforce_speed 15.8485ms 7.9903ms 125.1517 Ops/s 141.9873 Ops/s $\textbf{\color{#d91a1a}-11.86\%}$
test_iql_speed 43.2668ms 36.0386ms 27.7480 Ops/s 27.5380 Ops/s $\color{#35bf28}+0.76\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 2.3127ms 1.8191ms 549.7112 Ops/s 536.4043 Ops/s $\color{#35bf28}+2.48\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2.9646ms 1.9386ms 515.8268 Ops/s 415.6728 Ops/s $\textbf{\color{#35bf28}+24.09\%}$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2.6782ms 1.9632ms 509.3750 Ops/s 492.1566 Ops/s $\color{#35bf28}+3.50\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 2.5658ms 1.8449ms 542.0388 Ops/s 530.0365 Ops/s $\color{#35bf28}+2.26\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3.0660ms 1.9895ms 502.6309 Ops/s 494.4595 Ops/s $\color{#35bf28}+1.65\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3.0901ms 1.9577ms 510.8127 Ops/s 498.1987 Ops/s $\color{#35bf28}+2.53\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 2.0563ms 1.8351ms 544.9156 Ops/s 538.2095 Ops/s $\color{#35bf28}+1.25\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2.9427ms 1.9341ms 517.0245 Ops/s 435.2448 Ops/s $\textbf{\color{#35bf28}+18.79\%}$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2.9288ms 1.9502ms 512.7704 Ops/s 488.2919 Ops/s $\textbf{\color{#35bf28}+5.01\%}$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 0.1900s 2.1853ms 457.6004 Ops/s 523.0743 Ops/s $\textbf{\color{#d91a1a}-12.52\%}$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3.2220ms 1.9516ms 512.3905 Ops/s 486.6483 Ops/s $\textbf{\color{#35bf28}+5.29\%}$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2.6593ms 1.9601ms 510.1695 Ops/s 496.5075 Ops/s $\color{#35bf28}+2.75\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 2.7995ms 1.8417ms 542.9747 Ops/s 532.1219 Ops/s $\color{#35bf28}+2.04\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2.9877ms 1.9436ms 514.5196 Ops/s 494.0665 Ops/s $\color{#35bf28}+4.14\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3.1678ms 1.9787ms 505.3797 Ops/s 506.6271 Ops/s $\color{#d91a1a}-0.25\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 2.7417ms 1.8646ms 536.2996 Ops/s 449.2304 Ops/s $\textbf{\color{#35bf28}+19.38\%}$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2.7963ms 1.9649ms 508.9196 Ops/s 491.0668 Ops/s $\color{#35bf28}+3.64\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 3.1689ms 1.9648ms 508.9475 Ops/s 502.4576 Ops/s $\color{#35bf28}+1.29\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1957s 20.1875ms 49.5355 Ops/s 48.5298 Ops/s $\color{#35bf28}+2.07\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 0.1116s 20.3016ms 49.2573 Ops/s 47.5363 Ops/s $\color{#35bf28}+3.62\%$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 0.1089s 16.5009ms 60.6027 Ops/s 53.0951 Ops/s $\textbf{\color{#35bf28}+14.14\%}$
test_populate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1120s 18.5078ms 54.0311 Ops/s 53.1619 Ops/s $\color{#35bf28}+1.64\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 0.1078s 20.1923ms 49.5238 Ops/s 47.0938 Ops/s $\textbf{\color{#35bf28}+5.16\%}$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 0.1119s 18.4042ms 54.3355 Ops/s 53.0961 Ops/s $\color{#35bf28}+2.33\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1177s 18.8177ms 53.1415 Ops/s 52.1033 Ops/s $\color{#35bf28}+1.99\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 0.1122s 20.3038ms 49.2519 Ops/s 46.6908 Ops/s $\textbf{\color{#35bf28}+5.49\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 0.1068s 18.3434ms 54.5156 Ops/s 53.2190 Ops/s $\color{#35bf28}+2.44\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 22, 2023 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}9$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1272s 0.1231s 8.1214 Ops/s 8.1309 Ops/s $\color{#d91a1a}-0.12\%$
test_sync 0.1037s 0.1027s 9.7384 Ops/s 9.5025 Ops/s $\color{#35bf28}+2.48\%$
test_async 0.2731s 99.6338ms 10.0368 Ops/s 9.9256 Ops/s $\color{#35bf28}+1.12\%$
test_single_pixels 0.1329s 0.1326s 7.5411 Ops/s 6.8889 Ops/s $\textbf{\color{#35bf28}+9.47\%}$
test_sync_pixels 97.0390ms 94.8917ms 10.5383 Ops/s 10.4690 Ops/s $\color{#35bf28}+0.66\%$
test_async_pixels 0.2460s 91.3364ms 10.9485 Ops/s 10.8105 Ops/s $\color{#35bf28}+1.28\%$
test_simple 0.9520s 0.8876s 1.1266 Ops/s 1.1395 Ops/s $\color{#d91a1a}-1.13\%$
test_transformed 1.1949s 1.1350s 0.8810 Ops/s 0.8775 Ops/s $\color{#35bf28}+0.41\%$
test_serial 2.4901s 2.4226s 0.4128 Ops/s 0.4097 Ops/s $\color{#35bf28}+0.76\%$
test_parallel 2.5797s 2.5495s 0.3922 Ops/s 0.3953 Ops/s $\color{#d91a1a}-0.79\%$
test_step_mdp_speed[True-True-True-True-True] 76.2210μs 35.0757μs 28.5098 KOps/s 27.9919 KOps/s $\color{#35bf28}+1.85\%$
test_step_mdp_speed[True-True-True-True-False] 70.5120μs 20.9536μs 47.7246 KOps/s 48.9186 KOps/s $\color{#d91a1a}-2.44\%$
test_step_mdp_speed[True-True-True-False-True] 48.4700μs 20.9809μs 47.6624 KOps/s 47.1918 KOps/s $\color{#35bf28}+1.00\%$
test_step_mdp_speed[True-True-True-False-False] 59.9710μs 12.5896μs 79.4307 KOps/s 82.4768 KOps/s $\color{#d91a1a}-3.69\%$
test_step_mdp_speed[True-True-False-True-True] 84.9420μs 37.3664μs 26.7620 KOps/s 27.6452 KOps/s $\color{#d91a1a}-3.19\%$
test_step_mdp_speed[True-True-False-True-False] 65.2310μs 22.5868μs 44.2737 KOps/s 45.2628 KOps/s $\color{#d91a1a}-2.19\%$
test_step_mdp_speed[True-True-False-False-True] 46.6710μs 22.8901μs 43.6870 KOps/s 44.4730 KOps/s $\color{#d91a1a}-1.77\%$
test_step_mdp_speed[True-True-False-False-False] 41.1400μs 14.2141μs 70.3529 KOps/s 71.8772 KOps/s $\color{#d91a1a}-2.12\%$
test_step_mdp_speed[True-False-True-True-True] 68.6210μs 39.0542μs 25.6055 KOps/s 25.6233 KOps/s $\color{#d91a1a}-0.07\%$
test_step_mdp_speed[True-False-True-True-False] 61.3520μs 24.4638μs 40.8767 KOps/s 41.1953 KOps/s $\color{#d91a1a}-0.77\%$
test_step_mdp_speed[True-False-True-False-True] 47.7110μs 22.7722μs 43.9133 KOps/s 43.6129 KOps/s $\color{#35bf28}+0.69\%$
test_step_mdp_speed[True-False-True-False-False] 48.8210μs 14.0991μs 70.9263 KOps/s 72.2184 KOps/s $\color{#d91a1a}-1.79\%$
test_step_mdp_speed[True-False-False-True-True] 81.8410μs 40.4815μs 24.7027 KOps/s 24.3687 KOps/s $\color{#35bf28}+1.37\%$
test_step_mdp_speed[True-False-False-True-False] 75.4520μs 26.4014μs 37.8768 KOps/s 38.3704 KOps/s $\color{#d91a1a}-1.29\%$
test_step_mdp_speed[True-False-False-False-True] 61.0010μs 24.6843μs 40.5116 KOps/s 41.0449 KOps/s $\color{#d91a1a}-1.30\%$
test_step_mdp_speed[True-False-False-False-False] 57.5310μs 15.8795μs 62.9743 KOps/s 63.4572 KOps/s $\color{#d91a1a}-0.76\%$
test_step_mdp_speed[False-True-True-True-True] 63.2410μs 39.4315μs 25.3604 KOps/s 25.7230 KOps/s $\color{#d91a1a}-1.41\%$
test_step_mdp_speed[False-True-True-True-False] 72.8110μs 24.6572μs 40.5561 KOps/s 41.4871 KOps/s $\color{#d91a1a}-2.24\%$
test_step_mdp_speed[False-True-True-False-True] 53.7310μs 26.2438μs 38.1043 KOps/s 37.7341 KOps/s $\color{#35bf28}+0.98\%$
test_step_mdp_speed[False-True-True-False-False] 40.8600μs 16.0709μs 62.2242 KOps/s 62.1816 KOps/s $\color{#35bf28}+0.07\%$
test_step_mdp_speed[False-True-False-True-True] 79.7110μs 41.2104μs 24.2657 KOps/s 24.3228 KOps/s $\color{#d91a1a}-0.23\%$
test_step_mdp_speed[False-True-False-True-False] 46.7810μs 26.6399μs 37.5377 KOps/s 37.9680 KOps/s $\color{#d91a1a}-1.13\%$
test_step_mdp_speed[False-True-False-False-True] 49.9510μs 28.1807μs 35.4853 KOps/s 35.0300 KOps/s $\color{#35bf28}+1.30\%$
test_step_mdp_speed[False-True-False-False-False] 37.2700μs 18.0771μs 55.3187 KOps/s 56.7110 KOps/s $\color{#d91a1a}-2.46\%$
test_step_mdp_speed[False-False-True-True-True] 71.2300μs 42.8682μs 23.3273 KOps/s 23.1857 KOps/s $\color{#35bf28}+0.61\%$
test_step_mdp_speed[False-False-True-True-False] 47.0210μs 28.1905μs 35.4729 KOps/s 35.4543 KOps/s $\color{#35bf28}+0.05\%$
test_step_mdp_speed[False-False-True-False-True] 46.3510μs 28.0485μs 35.6525 KOps/s 35.2657 KOps/s $\color{#35bf28}+1.10\%$
test_step_mdp_speed[False-False-True-False-False] 38.4600μs 17.8056μs 56.1620 KOps/s 56.6203 KOps/s $\color{#d91a1a}-0.81\%$
test_step_mdp_speed[False-False-False-True-True] 97.1320μs 44.0555μs 22.6986 KOps/s 22.8766 KOps/s $\color{#d91a1a}-0.78\%$
test_step_mdp_speed[False-False-False-True-False] 55.8710μs 30.4904μs 32.7972 KOps/s 33.2206 KOps/s $\color{#d91a1a}-1.27\%$
test_step_mdp_speed[False-False-False-False-True] 48.0810μs 29.6222μs 33.7585 KOps/s 32.8509 KOps/s $\color{#35bf28}+2.76\%$
test_step_mdp_speed[False-False-False-False-False] 56.0910μs 19.6521μs 50.8852 KOps/s 51.3098 KOps/s $\color{#d91a1a}-0.83\%$
test_values[generalized_advantage_estimate-True-True] 27.0504ms 26.5452ms 37.6717 Ops/s 37.9039 Ops/s $\color{#d91a1a}-0.61\%$
test_values[vec_generalized_advantage_estimate-True-True] 85.2130ms 3.2775ms 305.1128 Ops/s 300.6841 Ops/s $\color{#35bf28}+1.47\%$
test_values[td0_return_estimate-False-False] 0.1021ms 66.8122μs 14.9673 KOps/s 14.7810 KOps/s $\color{#35bf28}+1.26\%$
test_values[td1_return_estimate-False-False] 57.1508ms 56.6800ms 17.6429 Ops/s 17.6930 Ops/s $\color{#d91a1a}-0.28\%$
test_values[vec_td1_return_estimate-False-False] 1.9914ms 1.7477ms 572.1717 Ops/s 571.9985 Ops/s $\color{#35bf28}+0.03\%$
test_values[td_lambda_return_estimate-True-False] 93.5276ms 90.8014ms 11.0130 Ops/s 11.1376 Ops/s $\color{#d91a1a}-1.12\%$
test_values[vec_td_lambda_return_estimate-True-False] 2.0141ms 1.7395ms 574.8634 Ops/s 571.5513 Ops/s $\color{#35bf28}+0.58\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 25.7999ms 25.3741ms 39.4102 Ops/s 39.5230 Ops/s $\color{#d91a1a}-0.29\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.8911ms 0.7402ms 1.3509 KOps/s 1.3485 KOps/s $\color{#35bf28}+0.18\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7539ms 0.6968ms 1.4352 KOps/s 1.4428 KOps/s $\color{#d91a1a}-0.53\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5629ms 1.4862ms 672.8390 Ops/s 670.6397 Ops/s $\color{#35bf28}+0.33\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.9792ms 0.7262ms 1.3771 KOps/s 1.3884 KOps/s $\color{#d91a1a}-0.82\%$
test_dqn_speed 7.9806ms 1.5954ms 626.8205 Ops/s 729.6053 Ops/s $\textbf{\color{#d91a1a}-14.09\%}$
test_ddpg_speed 4.8492ms 3.5638ms 280.5958 Ops/s 353.6062 Ops/s $\textbf{\color{#d91a1a}-20.65\%}$
test_sac_speed 10.8326ms 9.7829ms 102.2193 Ops/s 122.8423 Ops/s $\textbf{\color{#d91a1a}-16.79\%}$
test_redq_speed 16.8623ms 14.9231ms 67.0102 Ops/s 79.9370 Ops/s $\textbf{\color{#d91a1a}-16.17\%}$
test_redq_deprec_speed 15.0597ms 13.5837ms 73.6174 Ops/s 86.6342 Ops/s $\textbf{\color{#d91a1a}-15.02\%}$
test_td3_speed 19.9202ms 10.1746ms 98.2837 Ops/s 121.9447 Ops/s $\textbf{\color{#d91a1a}-19.40\%}$
test_cql_speed 33.3472ms 32.0871ms 31.1652 Ops/s 31.1017 Ops/s $\color{#35bf28}+0.20\%$
test_a2c_speed 8.5552ms 7.2928ms 137.1220 Ops/s 141.7925 Ops/s $\color{#d91a1a}-3.29\%$
test_ppo_speed 9.0670ms 7.5575ms 132.3194 Ops/s 135.1994 Ops/s $\color{#d91a1a}-2.13\%$
test_reinforce_speed 7.6003ms 6.3184ms 158.2674 Ops/s 164.2887 Ops/s $\color{#d91a1a}-3.67\%$
test_iql_speed 29.1471ms 27.5878ms 36.2479 Ops/s 38.4647 Ops/s $\textbf{\color{#d91a1a}-5.76\%}$
test_sample_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.0569ms 2.5682ms 389.3836 Ops/s 396.2351 Ops/s $\color{#d91a1a}-1.73\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3.6479ms 2.7365ms 365.4281 Ops/s 367.9993 Ops/s $\color{#d91a1a}-0.70\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.1316s 3.1128ms 321.2511 Ops/s 327.4765 Ops/s $\color{#d91a1a}-1.90\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.0253ms 2.5820ms 387.3041 Ops/s 392.5308 Ops/s $\color{#d91a1a}-1.33\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3.6976ms 2.7363ms 365.4576 Ops/s 323.4995 Ops/s $\textbf{\color{#35bf28}+12.97\%}$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 4.1525ms 2.7474ms 363.9827 Ops/s 366.3377 Ops/s $\color{#d91a1a}-0.64\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.0553ms 2.5739ms 388.5099 Ops/s 392.9060 Ops/s $\color{#d91a1a}-1.12\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 3.6817ms 2.7508ms 363.5303 Ops/s 365.4967 Ops/s $\color{#d91a1a}-0.54\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 4.4678ms 2.7514ms 363.4459 Ops/s 365.3626 Ops/s $\color{#d91a1a}-0.52\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.2962ms 2.5823ms 387.2456 Ops/s 390.6897 Ops/s $\color{#d91a1a}-0.88\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3.8021ms 2.7406ms 364.8771 Ops/s 364.4343 Ops/s $\color{#35bf28}+0.12\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 4.1400ms 2.7412ms 364.8099 Ops/s 364.5611 Ops/s $\color{#35bf28}+0.07\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.1952ms 2.5565ms 391.1581 Ops/s 391.6125 Ops/s $\color{#d91a1a}-0.12\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3.7823ms 2.7370ms 365.3681 Ops/s 364.8298 Ops/s $\color{#35bf28}+0.15\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3.5433ms 2.7385ms 365.1661 Ops/s 366.6265 Ops/s $\color{#d91a1a}-0.40\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 2.8898ms 2.5671ms 389.5370 Ops/s 392.0344 Ops/s $\color{#d91a1a}-0.64\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.1331s 3.1304ms 319.4441 Ops/s 367.8268 Ops/s $\textbf{\color{#d91a1a}-13.15\%}$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 3.6601ms 2.7487ms 363.8019 Ops/s 363.9698 Ops/s $\color{#d91a1a}-0.05\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.2692s 23.3351ms 42.8538 Ops/s 42.6279 Ops/s $\color{#35bf28}+0.53\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 0.1299s 20.3319ms 49.1839 Ops/s 48.9236 Ops/s $\color{#35bf28}+0.53\%$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 0.1318s 22.7712ms 43.9152 Ops/s 48.8884 Ops/s $\textbf{\color{#d91a1a}-10.17\%}$
test_populate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1315s 20.3955ms 49.0304 Ops/s 43.8264 Ops/s $\textbf{\color{#35bf28}+11.87\%}$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 0.1299s 20.3389ms 49.1669 Ops/s 48.8678 Ops/s $\color{#35bf28}+0.61\%$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 0.1308s 20.3759ms 49.0776 Ops/s 48.7240 Ops/s $\color{#35bf28}+0.73\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1300s 22.7471ms 43.9617 Ops/s 43.7433 Ops/s $\color{#35bf28}+0.50\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 0.1300s 20.3672ms 49.0985 Ops/s 48.6855 Ops/s $\color{#35bf28}+0.85\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 0.1314s 20.3736ms 49.0832 Ops/s 48.8280 Ops/s $\color{#35bf28}+0.52\%$

@vmoens vmoens changed the title [WIP] Refactor functional calls in losses [Refactor] Refactor functional calls in losses Nov 24, 2023
@vmoens vmoens marked this pull request as ready for review November 24, 2023 11:19
@vmoens vmoens merged commit bc7595f into main Nov 24, 2023
53 of 61 checks passed
@vmoens vmoens deleted the refactor-losses-functional branch November 24, 2023 14:44
@matteobettini
Copy link
Contributor

matteobettini commented Nov 25, 2023 •
edited
Loading

Was this supposed to solve #1576 and #1613?

Because with the script from #1576 i still get

Traceback (most recent call last):
  File "/Users/Matteo/PycharmProjects/BenchMARL/benchmarl/prova.py", line 15, in <module>
    loss_1 = DQNLoss(value_network=model, action_space="one_hot")
  File "/Users/Matteo/PycharmProjects/torchrl/torchrl/objectives/dqn.py", line 182, in __init__
    self.convert_to_functional(
  File "/Users/Matteo/PycharmProjects/torchrl/torchrl/objectives/common.py", line 294, in convert_to_functional
    self.__dict__[module_name] = deepcopy(module)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 296, in _reconstruct
    value = deepcopy(value, memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 296, in _reconstruct
    value = deepcopy(value, memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/copy.py", line 153, in deepcopy
    y = copier(memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/site-packages/torch_geometric/nn/dense/linear.py", line 115, in __deepcopy__
    out.weight = copy.deepcopy(self.weight, memo)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/torchrl/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1715, in __setattr__
    raise TypeError(f"cannot assign '{torch.typename(value)}' as parameter '{name}' "
TypeError: cannot assign 'torch.meta.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

Also #1593 is still present

@vmoens
Copy link
Contributor Author

vmoens commented Nov 26, 2023 •
edited
Loading

Is this on main or on a previous version of that branch (cannot judge based on the error stack)?
I had this error message with RNNs but solved it, not sure if this is the same issue.

Edit
Oh it's during deepcopy, not param assignment... but I don't get why the tensor isn't a param though. I'll fix this tmrw

@matteobettini
Copy link
Contributor

It is on main, can you not reproduce with the script in #1576 ?

This was referenced Nov 27, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Refactoring Refactoring of an existing feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vmoens @matteobettini @facebook-github-bot