Reproduce MOPO results

[jdchang1]

Hi @takuseno,

I have been trying to reproduce the MOPO results using your library and I have been having trouble. I have been following your MOPO script in the reproduce directory and have been messing around with training the dynamics ensemble for much longer. However even after training for more than 500 epochs I fail to get an evaluation score higher than 7 for Hopper-medium (d4rl). Could you provide any pointers?

Some metrics that seem suspicious are the SAC losses. There seems to be a blowup in the critic loss. Thanks!

[takuseno]

@jdchang1 Hello, thank you for the issue. Currently, I did not spend much time on checking MOPO's performance for now. But, very recently, the d4rl dataset conversion was fixed.
8e141c0
It might be worth trying the same for now with the latest master branch.

[TakuyaHiraoka]

Hi @takuseno @jdchang1,

MOPO implemented in d3rlpy does not terminate its model rollout at terminal states.
The original MOPO evaluates whether the generated state is terminal or not by accessing the termination function of the true environment, and terminates the rollout at terminal states (lines 408 -- 446 in [1]).

I found that this model rollout termination improves d3rl MOPO's performance and gets it closer to the original one.
(d3rl MOPO with a quick patch for the model rollout termination (and its evaluation result) are available at [2])

[1] https://github.com/tianheyu927/mopo/blob/master/mopo/algorithms/mopo.py
[2] https://drive.google.com/file/d/1GvHWJj3sU1wl7NGxMibee-ZbIOt65Smb/view?usp=sharing

[takuseno]

@TakuyaHiraoka Thanks for the info! I never realized they did a very tricky hack. It seems not very practical in general. I would rather add a classifier trained to estimate terminal flags. Probably this explains the COMBO issue too.

[IantheChan]

Actually, in the official implementation of Morel, the authors use the same trick. Maybe using a known termination function is common in model-based offline RL?

[ZishunYu]

I agree with @IantheChan. To my knowledge, some model-based RL works uses this trick, which seems to be very critical for model-based RL. I find this makes sense for robot learning as it is somehow not too difficult to check whether a robot sensor status is feasible.