Performance check (Continuous Actions) #48
Comments
|
@Miffyli All algorithms match performances except A2C (I ran it with the pytorch rmsprop, just wanted to check), which is a good sign after #110 (table created using the zoo) :
current a2c graphs (trained and plotted using the zoo, those are the deterministic evaluations): I will run A2C with the tensorflow RmsProp and also do some run of all algorithm with gSDE enabled (to check that it matches the paper). |
|
As expected changing the optimizer to tf rmsprop closes the gap. |
|
Results for gSDE are replicated (in fact, after solving PPO issue in #110 , it is even better, I will need to update the paper). |
|
@araffin Thanks for paying so much attention to performance checking. It makes us feel more confident about using SB3. I assume that for DDPG, TD3 and SAC, you are using the default parameters given in the documentation / paper. Just to give an example, in SB3 doc, for DDPG and TD3, the learning rates are 1e-3 and the batch sizes are 100; for SAC, the learning rate is 3e-4 and the batch size is 256. Personally, I find these hyper-parameter differences unjustified from an algorithmic standpoint. Although I'm aware that this is an effort to match the respective original publications, these are pretty similar algorithms. Is it possible to test them with shared hyper-parameters? Also, just to double check, where can I find the hyper-parameters you used for this exact replication (gSDE paper or the current SB3 doc)? Thanks again! |
actually, slightly different ones as I'm training on PyBullet envs (different from MuJoCo ones, used in the paper). You have instructions in the doc ;) I'm using the RL zoo: https://github.com/DLR-RM/rl-baselines3-zoo. Instructions: https://stable-baselines3.readthedocs.io/en/master/modules/sac.html#how-to-replicate-the-results
You are completely right. In fact, the original code of TD3 now shares SAC hyperparams (https://github.com/sfujim/TD3)
yes you can (but you need to deactivate gSDE for SAC, as gSDE for TD3 is no longer supported).
In the RL Zoo. You can even check the learning curves from the saved logs: https://github.com/DLR-RM/rl-trained-agents |
|
I feel amazed how well documented SB3 is. Thank you so much for the link to instructions, I should have noticed them myself while reading the doc. |
|
@araffin Just to double-check, I see that you used a linear decay learning rate scheduler for |
yes (also mentioned in the gSDE paper), there are important to stabilize the training. |
|
For DDPG, you wrote that you
Does this mean that, for running DDPG for this performance check, you didn't used the hyper-params specified here: https://github.com/DLR-RM/rl-trained-agents/tree/master/ddpg but instead you used the hyper-params specified here: https://github.com/DLR-RM/rl-trained-agents/tree/master/td3 ? |
They are the same ... Also in https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams/ddpg.yml |
oh true, I probably did some minor adjustments (reduce lr) to improve stability. I will update my comment. (I did not spend much time on DDPG as we treat it as a special case of TD3, cf code) |
|
Hi there, I'm using a custom environment for robotic grasping and for some reason my critic loss increases after a while. My reward starts to decrease as well and I believe this is connected to the critic loss going up.
I tried various hyper-params already ( Maybe a LR scheduler helps? |
|
@axkoenig I believe these are (still) the correct instructions and settings to replicate things: https://stable-baselines3.readthedocs.io/en/master/modules/sac.html#how-to-replicate-the-results (@araffin please correct me if I am wrong). |
|
Ok thanks, I was just wondering whether they were posted somewhere s.t. I don't need to train the model myself until the end |
yes, the RL Zoo is the place to go to replicate results. I saved the training/evaluation reward and the trained agent but not the rest of the metrics (although you can easily reproduce the run normally). |
|
Dead @araffin, |
Check that the algorithms reach expected performance.
This was already done prior to v0.5 for the gSDE paper but as we made big changes, it is good to check that again.
SB2 vs SB3 (Tensorflow Stable-Baselines vs Pytorch Stable-Baselines3)
a2c.pdf
a2c_ant.pdf
a2c_half.pdf
a2c_hopper.pdf
a2c_walker.pdf
ppo.pdf
ant_ppo.pdf
half_ppo.pdf
hopper_ppo.pdf
ppo_walker.pdf
sac.pdf
sac_ant.pdf
sac_half.pdf
sac_hopper.pdf
sac_walker.pdf
td3.pdf
td3_ant.pdf
td3_half.pdf
td3_hopper.pdf
td3_walker.pdf
See https://paperswithcode.com/paper/generalized-state-dependent-exploration-for for the score that should be reached in 1M (off-policy) or 2M steps (on-policy).
Test envs; PyBullet Envs
Tested with version 0.8.0 (
feat/perf-checkbranch in the two zoos)SB3 commit hash: cceffd5
rl-zoo commit hash: 99f7dd0321c5beea1a0d775ad6bc043d41f3e2db
Generalized State-Dependent Exploration (gSDE)
See https://paperswithcode.com/paper/generalized-state-dependent-exploration-for for the score that should be reached in 1M (off-policy) or 2M steps (on-policy).
gsde_onpolicy.pdf
gsde_onpolicy_ant.pdf
gsde_onpolicy_half.pdf
gsde_onpolicy_hopper.pdf
gsde_onpolicy_walker.pdf
gsde_off_policy.pdf
gsde_offpolicy_ant.pdf
gsde_offpolicy_half.pdf
gsde_offpolicy_hopper.pdf
gsde_offpolicy_walker.pdf
SB3 commit hash: b948b7f
rl zoo commit hash: b56c1470c9a958c196f60e814de893050e2469f0
DDPG
Using TD3 hyperparameters as base with some minor adjustements (lr,
batch_size) for stability.6 seeds, 1M steps.
The text was updated successfully, but these errors were encountered: