[Feature] PyTrees in replay buffers

[vmoens]

No description provided.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1831

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ 1 Pending, 5 Unrelated Failures

As of commit 0a742fc with merge base c390cf6 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Libs Tests on Linux / unittests-vd4rl (3.9, 12.1) / linux-job (gh)
  test/test_libs.py::TestVD4RL::test_load[None]
• Unit-tests on Linux / tests-cpu (3.8) / linux-job (gh)
  test/test_libs.py::TestGym::test_vecenvs_env[CartPole-v1]
• Unit-tests on MacOS CPU / tests (3.11) / macos-job (gh)
  test/test_libs.py::TestGym::test_vecenvs_env[CartPole-v1]

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Habitat Tests on Linux / tests (3.9, 11.6) / linux-job (gh)
  test/test_libs.py::TestHabitat::test_habitat_render[False-HabitatPick-v0]
• Unit-tests on Windows / unittests-gpu / windows-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}4$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	61.6525ms	60.5969ms	16.5025 Ops/s	15.4357 Ops/s	$\textbf{\color{#35bf28}+6.91\%}$
test_sync	34.1104ms	32.6153ms	30.6605 Ops/s	30.0774 Ops/s	$\color{#35bf28}+1.94\%$
test_async	76.7126ms	32.5098ms	30.7600 Ops/s	31.0421 Ops/s	$\color{#d91a1a}-0.91\%$
test_simple	0.5013s	0.4338s	2.3055 Ops/s	2.2528 Ops/s	$\color{#35bf28}+2.34\%$
test_transformed	0.6501s	0.5932s	1.6858 Ops/s	1.6824 Ops/s	$\color{#35bf28}+0.20\%$
test_serial	1.4144s	1.3535s	0.7388 Ops/s	0.7312 Ops/s	$\color{#35bf28}+1.05\%$
test_parallel	1.3005s	1.2348s	0.8099 Ops/s	0.8286 Ops/s	$\color{#d91a1a}-2.26\%$
test_step_mdp_speed[True-True-True-True-True]	0.1721ms	21.1306μs	47.3248 KOps/s	46.8473 KOps/s	$\color{#35bf28}+1.02\%$
test_step_mdp_speed[True-True-True-True-False]	40.1950μs	12.8998μs	77.5204 KOps/s	76.6741 KOps/s	$\color{#35bf28}+1.10\%$
test_step_mdp_speed[True-True-True-False-True]	38.7630μs	12.4188μs	80.5234 KOps/s	77.7448 KOps/s	$\color{#35bf28}+3.57\%$
test_step_mdp_speed[True-True-True-False-False]	27.2310μs	7.5340μs	132.7310 KOps/s	129.4162 KOps/s	$\color{#35bf28}+2.56\%$
test_step_mdp_speed[True-True-False-True-True]	55.8950μs	22.5073μs	44.4301 KOps/s	43.4805 KOps/s	$\color{#35bf28}+2.18\%$
test_step_mdp_speed[True-True-False-True-False]	38.1210μs	14.2634μs	70.1098 KOps/s	69.9862 KOps/s	$\color{#35bf28}+0.18\%$
test_step_mdp_speed[True-True-False-False-True]	48.7720μs	13.6117μs	73.4660 KOps/s	72.6113 KOps/s	$\color{#35bf28}+1.18\%$
test_step_mdp_speed[True-True-False-False-False]	60.2930μs	8.6814μs	115.1884 KOps/s	113.3780 KOps/s	$\color{#35bf28}+1.60\%$
test_step_mdp_speed[True-False-True-True-True]	57.0580μs	23.9037μs	41.8344 KOps/s	41.5047 KOps/s	$\color{#35bf28}+0.79\%$
test_step_mdp_speed[True-False-True-True-False]	54.8130μs	15.5460μs	64.3253 KOps/s	63.7270 KOps/s	$\color{#35bf28}+0.94\%$
test_step_mdp_speed[True-False-True-False-True]	34.1640μs	13.6578μs	73.2182 KOps/s	71.7854 KOps/s	$\color{#35bf28}+2.00\%$
test_step_mdp_speed[True-False-True-False-False]	29.5350μs	8.8493μs	113.0039 KOps/s	113.1850 KOps/s	$\color{#d91a1a}-0.16\%$
test_step_mdp_speed[True-False-False-True-True]	51.7770μs	24.9795μs	40.0328 KOps/s	38.3851 KOps/s	$\color{#35bf28}+4.29\%$
test_step_mdp_speed[True-False-False-True-False]	46.7680μs	16.4886μs	60.6479 KOps/s	60.0423 KOps/s	$\color{#35bf28}+1.01\%$
test_step_mdp_speed[True-False-False-False-True]	39.2330μs	14.7542μs	67.7775 KOps/s	67.2374 KOps/s	$\color{#35bf28}+0.80\%$
test_step_mdp_speed[True-False-False-False-False]	34.9350μs	9.9618μs	100.3832 KOps/s	100.0713 KOps/s	$\color{#35bf28}+0.31\%$
test_step_mdp_speed[False-True-True-True-True]	0.2893ms	25.8348μs	38.7074 KOps/s	41.6245 KOps/s	$\textbf{\color{#d91a1a}-7.01\%}$
test_step_mdp_speed[False-True-True-True-False]	44.4230μs	15.5990μs	64.1069 KOps/s	64.4197 KOps/s	$\color{#d91a1a}-0.49\%$
test_step_mdp_speed[False-True-True-False-True]	51.4160μs	15.9639μs	62.6414 KOps/s	62.5237 KOps/s	$\color{#35bf28}+0.19\%$
test_step_mdp_speed[False-True-True-False-False]	34.5650μs	10.0452μs	99.5502 KOps/s	99.5549 KOps/s	$-0.00\%$
test_step_mdp_speed[False-True-False-True-True]	51.7170μs	25.2275μs	39.6394 KOps/s	39.5327 KOps/s	$\color{#35bf28}+0.27\%$
test_step_mdp_speed[False-True-False-True-False]	36.8990μs	16.7874μs	59.5686 KOps/s	59.7584 KOps/s	$\color{#d91a1a}-0.32\%$
test_step_mdp_speed[False-True-False-False-True]	60.5430μs	17.0466μs	58.6628 KOps/s	58.1345 KOps/s	$\color{#35bf28}+0.91\%$
test_step_mdp_speed[False-True-False-False-False]	34.7950μs	11.2646μs	88.7734 KOps/s	89.3139 KOps/s	$\color{#d91a1a}-0.61\%$
test_step_mdp_speed[False-False-True-True-True]	55.1030μs	26.1299μs	38.2703 KOps/s	37.1307 KOps/s	$\color{#35bf28}+3.07\%$
test_step_mdp_speed[False-False-True-True-False]	42.3600μs	18.0126μs	55.5167 KOps/s	53.1169 KOps/s	$\color{#35bf28}+4.52\%$
test_step_mdp_speed[False-False-True-False-True]	60.0730μs	17.1595μs	58.2768 KOps/s	58.3510 KOps/s	$\color{#d91a1a}-0.13\%$
test_step_mdp_speed[False-False-True-False-False]	35.3870μs	11.2493μs	88.8943 KOps/s	88.2858 KOps/s	$\color{#35bf28}+0.69\%$
test_step_mdp_speed[False-False-False-True-True]	0.2333ms	27.2649μs	36.6771 KOps/s	35.6783 KOps/s	$\color{#35bf28}+2.80\%$
test_step_mdp_speed[False-False-False-True-False]	0.1156ms	19.0178μs	52.5824 KOps/s	52.1758 KOps/s	$\color{#35bf28}+0.78\%$
test_step_mdp_speed[False-False-False-False-True]	52.4880μs	18.0162μs	55.5056 KOps/s	55.3164 KOps/s	$\color{#35bf28}+0.34\%$
test_step_mdp_speed[False-False-False-False-False]	36.5180μs	12.2429μs	81.6797 KOps/s	81.7318 KOps/s	$\color{#d91a1a}-0.06\%$
test_values[generalized_advantage_estimate-True-True]	14.5558ms	12.2512ms	81.6248 Ops/s	83.0874 Ops/s	$\color{#d91a1a}-1.76\%$
test_values[vec_generalized_advantage_estimate-True-True]	34.4304ms	26.8863ms	37.1937 Ops/s	35.6914 Ops/s	$\color{#35bf28}+4.21\%$
test_values[td0_return_estimate-False-False]	0.2717ms	0.1908ms	5.2415 KOps/s	5.5860 KOps/s	$\textbf{\color{#d91a1a}-6.17\%}$
test_values[td1_return_estimate-False-False]	27.2310ms	26.0388ms	38.4043 Ops/s	39.1936 Ops/s	$\color{#d91a1a}-2.01\%$
test_values[vec_td1_return_estimate-False-False]	38.4456ms	27.2224ms	36.7344 Ops/s	35.1032 Ops/s	$\color{#35bf28}+4.65\%$
test_values[td_lambda_return_estimate-True-False]	37.0529ms	36.2434ms	27.5912 Ops/s	28.2421 Ops/s	$\color{#d91a1a}-2.30\%$
test_values[vec_td_lambda_return_estimate-True-False]	34.9086ms	26.9509ms	37.1045 Ops/s	35.6867 Ops/s	$\color{#35bf28}+3.97\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	9.3550ms	8.2626ms	121.0275 Ops/s	125.0423 Ops/s	$\color{#d91a1a}-3.21\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	11.0361ms	1.7992ms	555.7921 Ops/s	525.4710 Ops/s	$\textbf{\color{#35bf28}+5.77\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	1.5810ms	0.4339ms	2.3045 KOps/s	2.3053 KOps/s	$\color{#d91a1a}-0.03\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	45.5880ms	38.7169ms	25.8285 Ops/s	27.1599 Ops/s	$\color{#d91a1a}-4.90\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	11.0208ms	2.6883ms	371.9805 Ops/s	374.1281 Ops/s	$\color{#d91a1a}-0.57\%$
test_dqn_speed	80.6370ms	8.1942ms	122.0371 Ops/s	133.6568 Ops/s	$\textbf{\color{#d91a1a}-8.69\%}$
test_ddpg_speed	16.6257ms	14.5747ms	68.6121 Ops/s	69.1993 Ops/s	$\color{#d91a1a}-0.85\%$
test_sac_speed	36.4980ms	29.4082ms	34.0041 Ops/s	34.6416 Ops/s	$\color{#d91a1a}-1.84\%$
test_redq_speed	49.9633ms	45.5970ms	21.9313 Ops/s	22.0298 Ops/s	$\color{#d91a1a}-0.45\%$
test_redq_deprec_speed	35.0172ms	25.7682ms	38.8075 Ops/s	39.4063 Ops/s	$\color{#d91a1a}-1.52\%$
test_td3_speed	28.8708ms	20.3352ms	49.1757 Ops/s	50.2609 Ops/s	$\color{#d91a1a}-2.16\%$
test_cql_speed	90.9609ms	88.2187ms	11.3355 Ops/s	11.5441 Ops/s	$\color{#d91a1a}-1.81\%$
test_a2c_speed	31.9213ms	27.6637ms	36.1484 Ops/s	37.5241 Ops/s	$\color{#d91a1a}-3.67\%$
test_ppo_speed	28.1693ms	27.5140ms	36.3452 Ops/s	37.3817 Ops/s	$\color{#d91a1a}-2.77\%$
test_reinforce_speed	27.3453ms	26.4564ms	37.7980 Ops/s	38.7832 Ops/s	$\color{#d91a1a}-2.54\%$
test_iql_speed	73.3895ms	64.5566ms	15.4903 Ops/s	15.9749 Ops/s	$\color{#d91a1a}-3.03\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	1.9692ms	1.4073ms	710.5717 Ops/s	709.7896 Ops/s	$\color{#35bf28}+0.11\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	8.7853ms	0.5312ms	1.8827 KOps/s	1.9289 KOps/s	$\color{#d91a1a}-2.40\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	1.5781ms	0.4968ms	2.0130 KOps/s	2.0266 KOps/s	$\color{#d91a1a}-0.67\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.5173ms	1.3555ms	737.7098 Ops/s	713.4323 Ops/s	$\color{#35bf28}+3.40\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.9728ms	0.5220ms	1.9158 KOps/s	1.9080 KOps/s	$\color{#35bf28}+0.41\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	8.8377ms	0.4981ms	2.0077 KOps/s	2.0568 KOps/s	$\color{#d91a1a}-2.39\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.3549ms	1.5743ms	635.2050 Ops/s	621.4699 Ops/s	$\color{#35bf28}+2.21\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	9.0461ms	0.6723ms	1.4874 KOps/s	1.5315 KOps/s	$\color{#d91a1a}-2.88\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	8.9284ms	0.6368ms	1.5703 KOps/s	1.5907 KOps/s	$\color{#d91a1a}-1.28\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	1.7186ms	1.3882ms	720.3405 Ops/s	711.5131 Ops/s	$\color{#35bf28}+1.24\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.6317ms	0.5242ms	1.9078 KOps/s	1.8772 KOps/s	$\color{#35bf28}+1.63\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	8.8858ms	0.5064ms	1.9747 KOps/s	2.0144 KOps/s	$\color{#d91a1a}-1.97\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	1.7955ms	1.3414ms	745.4875 Ops/s	650.9492 Ops/s	$\textbf{\color{#35bf28}+14.52\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	8.8408ms	0.5335ms	1.8744 KOps/s	1.8727 KOps/s	$\color{#35bf28}+0.09\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6248ms	0.4904ms	2.0390 KOps/s	1.2698 KOps/s	$\textbf{\color{#35bf28}+60.58\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.0382ms	1.6309ms	613.1764 Ops/s	586.5676 Ops/s	$\color{#35bf28}+4.54\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	9.7081ms	0.6760ms	1.4792 KOps/s	1.5262 KOps/s	$\color{#d91a1a}-3.08\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	8.9777ms	0.6633ms	1.5075 KOps/s	1.5653 KOps/s	$\color{#d91a1a}-3.69\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1172s	12.4062ms	80.6052 Ops/s	79.1910 Ops/s	$\color{#35bf28}+1.79\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	15.7941ms	13.6523ms	73.2476 Ops/s	74.0184 Ops/s	$\color{#d91a1a}-1.04\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	6.4822ms	3.2665ms	306.1393 Ops/s	306.4485 Ops/s	$\color{#d91a1a}-0.10\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1025s	10.0763ms	99.2433 Ops/s	97.5384 Ops/s	$\color{#35bf28}+1.75\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	20.2485ms	13.7252ms	72.8588 Ops/s	73.2177 Ops/s	$\color{#d91a1a}-0.49\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	6.8194ms	3.3056ms	302.5130 Ops/s	308.8684 Ops/s	$\color{#d91a1a}-2.06\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1133s	12.4783ms	80.1390 Ops/s	79.6432 Ops/s	$\color{#35bf28}+0.62\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	21.2616ms	14.0605ms	71.1212 Ops/s	72.3907 Ops/s	$\color{#d91a1a}-1.75\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	5.0674ms	3.4897ms	286.5547 Ops/s	286.7561 Ops/s	$\color{#d91a1a}-0.07\%$

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}21$. Worsened: $\large\color{#d91a1a}5$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1121s	0.1118s	8.9443 Ops/s	8.6651 Ops/s	$\color{#35bf28}+3.22\%$
test_sync	95.9686ms	95.6861ms	10.4508 Ops/s	10.4131 Ops/s	$\color{#35bf28}+0.36\%$
test_async	0.2552s	91.7366ms	10.9008 Ops/s	10.8848 Ops/s	$\color{#35bf28}+0.15\%$
test_single_pixels	0.1371s	0.1364s	7.3312 Ops/s	7.1636 Ops/s	$\color{#35bf28}+2.34\%$
test_sync_pixels	78.3381ms	76.8779ms	13.0076 Ops/s	13.0891 Ops/s	$\color{#d91a1a}-0.62\%$
test_async_pixels	0.1997s	72.2450ms	13.8418 Ops/s	13.8762 Ops/s	$\color{#d91a1a}-0.25\%$
test_simple	0.8876s	0.8165s	1.2248 Ops/s	1.1956 Ops/s	$\color{#35bf28}+2.44\%$
test_transformed	1.1097s	1.0453s	0.9566 Ops/s	0.9278 Ops/s	$\color{#35bf28}+3.10\%$
test_serial	2.3210s	2.2558s	0.4433 Ops/s	0.4321 Ops/s	$\color{#35bf28}+2.60\%$
test_parallel	1.9302s	1.8626s	0.5369 Ops/s	0.5334 Ops/s	$\color{#35bf28}+0.65\%$
test_step_mdp_speed[True-True-True-True-True]	83.0450μs	32.2956μs	30.9640 KOps/s	29.9564 KOps/s	$\color{#35bf28}+3.36\%$
test_step_mdp_speed[True-True-True-True-False]	43.3520μs	19.2157μs	52.0409 KOps/s	50.4943 KOps/s	$\color{#35bf28}+3.06\%$
test_step_mdp_speed[True-True-True-False-True]	32.8420μs	17.8914μs	55.8927 KOps/s	53.6493 KOps/s	$\color{#35bf28}+4.18\%$
test_step_mdp_speed[True-True-True-False-False]	29.1120μs	10.8895μs	91.8314 KOps/s	88.2284 KOps/s	$\color{#35bf28}+4.08\%$
test_step_mdp_speed[True-True-False-True-True]	50.9430μs	33.3598μs	29.9762 KOps/s	28.5143 KOps/s	$\textbf{\color{#35bf28}+5.13\%}$
test_step_mdp_speed[True-True-False-True-False]	46.1120μs	20.7486μs	48.1960 KOps/s	45.4424 KOps/s	$\textbf{\color{#35bf28}+6.06\%}$
test_step_mdp_speed[True-True-False-False-True]	36.3520μs	19.5146μs	51.2436 KOps/s	48.5872 KOps/s	$\textbf{\color{#35bf28}+5.47\%}$
test_step_mdp_speed[True-True-False-False-False]	39.5220μs	12.6536μs	79.0290 KOps/s	75.2492 KOps/s	$\textbf{\color{#35bf28}+5.02\%}$
test_step_mdp_speed[True-False-True-True-True]	57.4230μs	34.9507μs	28.6118 KOps/s	26.7372 KOps/s	$\textbf{\color{#35bf28}+7.01\%}$
test_step_mdp_speed[True-False-True-True-False]	36.4220μs	22.5737μs	44.2994 KOps/s	41.9121 KOps/s	$\textbf{\color{#35bf28}+5.70\%}$
test_step_mdp_speed[True-False-True-False-True]	84.6340μs	19.6198μs	50.9690 KOps/s	48.2735 KOps/s	$\textbf{\color{#35bf28}+5.58\%}$
test_step_mdp_speed[True-False-True-False-False]	30.6120μs	12.7914μs	78.1775 KOps/s	74.8987 KOps/s	$\color{#35bf28}+4.38\%$
test_step_mdp_speed[True-False-False-True-True]	65.8240μs	37.0181μs	27.0138 KOps/s	25.7843 KOps/s	$\color{#35bf28}+4.77\%$
test_step_mdp_speed[True-False-False-True-False]	43.4630μs	24.5666μs	40.7057 KOps/s	38.8263 KOps/s	$\color{#35bf28}+4.84\%$
test_step_mdp_speed[True-False-False-False-True]	42.6730μs	21.5685μs	46.3639 KOps/s	44.0215 KOps/s	$\textbf{\color{#35bf28}+5.32\%}$
test_step_mdp_speed[True-False-False-False-False]	28.9520μs	14.4096μs	69.3982 KOps/s	65.5239 KOps/s	$\textbf{\color{#35bf28}+5.91\%}$
test_step_mdp_speed[False-True-True-True-True]	60.2340μs	35.2696μs	28.3530 KOps/s	26.6231 KOps/s	$\textbf{\color{#35bf28}+6.50\%}$
test_step_mdp_speed[False-True-True-True-False]	46.6030μs	22.7633μs	43.9303 KOps/s	41.8337 KOps/s	$\textbf{\color{#35bf28}+5.01\%}$
test_step_mdp_speed[False-True-True-False-True]	43.5230μs	23.6127μs	42.3501 KOps/s	41.0574 KOps/s	$\color{#35bf28}+3.15\%$
test_step_mdp_speed[False-True-True-False-False]	36.6320μs	14.7401μs	67.8420 KOps/s	65.8034 KOps/s	$\color{#35bf28}+3.10\%$
test_step_mdp_speed[False-True-False-True-True]	65.3340μs	37.3096μs	26.8027 KOps/s	25.4113 KOps/s	$\textbf{\color{#35bf28}+5.48\%}$
test_step_mdp_speed[False-True-False-True-False]	85.3450μs	24.7889μs	40.3406 KOps/s	38.4910 KOps/s	$\color{#35bf28}+4.81\%$
test_step_mdp_speed[False-True-False-False-True]	88.7450μs	25.3880μs	39.3887 KOps/s	37.6568 KOps/s	$\color{#35bf28}+4.60\%$
test_step_mdp_speed[False-True-False-False-False]	30.5420μs	16.2487μs	61.5432 KOps/s	57.4131 KOps/s	$\textbf{\color{#35bf28}+7.19\%}$
test_step_mdp_speed[False-False-True-True-True]	69.8730μs	38.7369μs	25.8152 KOps/s	24.2345 KOps/s	$\textbf{\color{#35bf28}+6.52\%}$
test_step_mdp_speed[False-False-True-True-False]	53.3730μs	26.6270μs	37.5559 KOps/s	36.1367 KOps/s	$\color{#35bf28}+3.93\%$
test_step_mdp_speed[False-False-True-False-True]	50.1130μs	25.7718μs	38.8021 KOps/s	37.6354 KOps/s	$\color{#35bf28}+3.10\%$
test_step_mdp_speed[False-False-True-False-False]	74.8940μs	16.2301μs	61.6139 KOps/s	58.9139 KOps/s	$\color{#35bf28}+4.58\%$
test_step_mdp_speed[False-False-False-True-True]	63.9130μs	40.2229μs	24.8615 KOps/s	23.7029 KOps/s	$\color{#35bf28}+4.89\%$
test_step_mdp_speed[False-False-False-True-False]	43.8220μs	28.1410μs	35.5354 KOps/s	33.9058 KOps/s	$\color{#35bf28}+4.81\%$
test_step_mdp_speed[False-False-False-False-True]	69.6340μs	26.6309μs	37.5504 KOps/s	35.6792 KOps/s	$\textbf{\color{#35bf28}+5.24\%}$
test_step_mdp_speed[False-False-False-False-False]	34.2020μs	18.0552μs	55.3858 KOps/s	54.0149 KOps/s	$\color{#35bf28}+2.54\%$
test_values[generalized_advantage_estimate-True-True]	24.7482ms	24.1224ms	41.4552 Ops/s	40.9044 Ops/s	$\color{#35bf28}+1.35\%$
test_values[vec_generalized_advantage_estimate-True-True]	83.7000ms	3.2266ms	309.9238 Ops/s	310.4441 Ops/s	$\color{#d91a1a}-0.17\%$
test_values[td0_return_estimate-False-False]	0.1051ms	61.1319μs	16.3581 KOps/s	16.1971 KOps/s	$\color{#35bf28}+0.99\%$
test_values[td1_return_estimate-False-False]	52.4858ms	52.1262ms	19.1842 Ops/s	17.4311 Ops/s	$\textbf{\color{#35bf28}+10.06\%}$
test_values[vec_td1_return_estimate-False-False]	1.9891ms	1.7477ms	572.1870 Ops/s	568.4854 Ops/s	$\color{#35bf28}+0.65\%$
test_values[td_lambda_return_estimate-True-False]	85.4101ms	83.3349ms	11.9998 Ops/s	11.0035 Ops/s	$\textbf{\color{#35bf28}+9.05\%}$
test_values[vec_td_lambda_return_estimate-True-False]	2.0463ms	1.7467ms	572.5041 Ops/s	557.1664 Ops/s	$\color{#35bf28}+2.75\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	23.1906ms	23.0080ms	43.4632 Ops/s	40.6158 Ops/s	$\textbf{\color{#35bf28}+7.01\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.8659ms	0.6951ms	1.4386 KOps/s	1.4291 KOps/s	$\color{#35bf28}+0.67\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.7268ms	0.6417ms	1.5584 KOps/s	1.5523 KOps/s	$\color{#35bf28}+0.39\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.4986ms	1.4483ms	690.4867 Ops/s	685.5976 Ops/s	$\color{#35bf28}+0.71\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.9391ms	0.6622ms	1.5101 KOps/s	1.5049 KOps/s	$\color{#35bf28}+0.35\%$
test_dqn_speed	13.9465ms	7.0190ms	142.4709 Ops/s	136.2071 Ops/s	$\color{#35bf28}+4.60\%$
test_ddpg_speed	14.6781ms	13.6749ms	73.1269 Ops/s	69.9794 Ops/s	$\color{#35bf28}+4.50\%$
test_sac_speed	28.4581ms	27.7126ms	36.0847 Ops/s	34.4479 Ops/s	$\color{#35bf28}+4.75\%$
test_redq_speed	0.1369s	49.9380ms	20.0248 Ops/s	21.1661 Ops/s	$\textbf{\color{#d91a1a}-5.39\%}$
test_redq_deprec_speed	23.9296ms	23.0621ms	43.3611 Ops/s	38.4190 Ops/s	$\textbf{\color{#35bf28}+12.86\%}$
test_td3_speed	19.0898ms	18.7906ms	53.2181 Ops/s	51.1830 Ops/s	$\color{#35bf28}+3.98\%$
test_cql_speed	80.3911ms	79.4297ms	12.5898 Ops/s	12.1570 Ops/s	$\color{#35bf28}+3.56\%$
test_a2c_speed	25.9651ms	25.7101ms	38.8953 Ops/s	37.2783 Ops/s	$\color{#35bf28}+4.34\%$
test_ppo_speed	26.8301ms	25.7761ms	38.7957 Ops/s	37.2828 Ops/s	$\color{#35bf28}+4.06\%$
test_reinforce_speed	25.4880ms	24.7287ms	40.4389 Ops/s	38.9311 Ops/s	$\color{#35bf28}+3.87\%$
test_iql_speed	56.1140ms	55.2534ms	18.0984 Ops/s	17.5199 Ops/s	$\color{#35bf28}+3.30\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	0.1014s	2.0058ms	498.5605 Ops/s	544.5262 Ops/s	$\textbf{\color{#d91a1a}-8.44\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.9629ms	0.8429ms	1.1863 KOps/s	1.1847 KOps/s	$\color{#35bf28}+0.13\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.9311ms	0.8179ms	1.2227 KOps/s	1.2181 KOps/s	$\color{#35bf28}+0.38\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.4074ms	1.7740ms	563.6820 Ops/s	552.8787 Ops/s	$\color{#35bf28}+1.95\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.9549ms	0.8314ms	1.2028 KOps/s	1.2038 KOps/s	$\color{#d91a1a}-0.08\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.9307ms	0.8074ms	1.2386 KOps/s	1.2355 KOps/s	$\color{#35bf28}+0.25\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.9012ms	2.0458ms	488.8004 Ops/s	481.6302 Ops/s	$\color{#35bf28}+1.49\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.0691ms	0.9550ms	1.0471 KOps/s	1.0421 KOps/s	$\color{#35bf28}+0.48\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	1.0480ms	0.9311ms	1.0740 KOps/s	1.0658 KOps/s	$\color{#35bf28}+0.77\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.5862ms	1.8112ms	552.1208 Ops/s	540.4962 Ops/s	$\color{#35bf28}+2.15\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.0244ms	0.8412ms	1.1887 KOps/s	1.0234 KOps/s	$\textbf{\color{#35bf28}+16.16\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.9233ms	0.8193ms	1.2206 KOps/s	1.2160 KOps/s	$\color{#35bf28}+0.38\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	1.8989ms	1.7819ms	561.2083 Ops/s	553.8231 Ops/s	$\color{#35bf28}+1.33\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.9591ms	0.8315ms	1.2026 KOps/s	1.2007 KOps/s	$\color{#35bf28}+0.16\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.1096s	0.9547ms	1.0475 KOps/s	1.2340 KOps/s	$\textbf{\color{#d91a1a}-15.12\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.1148ms	2.0457ms	488.8388 Ops/s	479.2414 Ops/s	$\color{#35bf28}+2.00\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.1658ms	0.9557ms	1.0463 KOps/s	1.0398 KOps/s	$\color{#35bf28}+0.63\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.1167s	1.0712ms	933.5249 Ops/s	1.0609 KOps/s	$\textbf{\color{#d91a1a}-12.01\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1217s	10.0482ms	99.5205 Ops/s	101.1662 Ops/s	$\color{#d91a1a}-1.63\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	16.4047ms	14.0252ms	71.3004 Ops/s	71.1539 Ops/s	$\color{#35bf28}+0.21\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	6.0070ms	3.4083ms	293.4024 Ops/s	294.0767 Ops/s	$\color{#d91a1a}-0.23\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1182s	9.9020ms	100.9901 Ops/s	100.3291 Ops/s	$\color{#35bf28}+0.66\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	16.4546ms	13.9761ms	71.5507 Ops/s	71.2788 Ops/s	$\color{#35bf28}+0.38\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	6.3704ms	3.4206ms	292.3426 Ops/s	293.0784 Ops/s	$\color{#d91a1a}-0.25\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1185s	10.1570ms	98.4547 Ops/s	80.0562 Ops/s	$\textbf{\color{#35bf28}+22.98\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	0.1222s	16.3757ms	61.0660 Ops/s	70.3200 Ops/s	$\textbf{\color{#d91a1a}-13.16\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	6.8557ms	3.6273ms	275.6862 Ops/s	278.1606 Ops/s	$\color{#d91a1a}-0.89\%$

[matteobettini]

Some doc suggestions