Awesome Diffusion Model in RL

This is a collection of research papers for Diffusion Model in RL. And the repository will be continuously updated to track the frontier of Diffusion RL.

Welcome to follow and star!

Table of Contents

• Awesome Diffusion Model in RL
  • Table of Contents
  • Overview of Diffusion Model in RL
    • Advantage
  • Papers
    • Arxiv
    • ICML 2023
    • ICLR 2023
    • ICRA 2023
    • NeurIPS 2022
    • ICML 2022
  • Contributing
  • License

Overview of Diffusion Model in RL

The Diffusion Model in RL was introduced by “Planning with Diffusion for Flexible Behavior Synthesis” by Janner, Michael, et al. It casts trajectory optimization as a diffusion probabilistic model that plans by iteratively refining trajectories.

There is another way: "Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning" by Wang, Z. proposed Diffusion Model as policy-optimization in offline RL, et al. Specifically, Diffusion-QL forms policy as a conditional diffusion model with states as the condition from the offline policy-optimization perspective.

Advantage

1. Bypass the need for bootstrapping for long term credit assignment.
2. Avoid undesirable short-sighted behaviors due to the discounting future rewards.
3. Enjoy the diffusion models widely used in language and vision, which are easy to scale and adapt to multi-modal data.

Papers

format:
- [title](paper link) [links]
  - author1, author2, and author3...
  - publisher
  - key 
  - code 
  - experiment environment

Arxiv

• MADiff: Offline Multi-agent Learning with Diffusion Models

  • Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, Weinan Zhang
  • Key: Multi-agent, Offline RL, Classifier-free
  • ExpEnv: MPE, SMAC, Multi-Agent Trajectory Prediction (MATP)

• Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning

  • Suzan Ece Ada, Erhan Oztop, Emre Ugur
  • Key: Offline RL, OOD Generalization
  • ExpEnv: 2D-Multimodal Contextual Bandit, D4RL

• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

  • Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Shuran Song
  • Key: Robot Manipulation
  • ExpEnv: Robomimic, Push-T, Multimodal Block Pushing, Franka Kitchen

• Diffusion-based Generation, Optimization, and Planning in 3D Scenes

  • Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, Song-Chun Zhu
  • Key: 3D Scene Understanding, Optimization, Planning
  • Code: official
  • ExpEnv: ScanNet, MultiDex, PROX

• AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners

  • Moritz Reuss, Maximilian Li, Xiaogang Jia, Rudolf Lioutikov
  • Key: Planning, Generalizability, Classifier-guided
  • ExpEnv: Maze2D, MuJoCo, KUKA Robot

• Goal-Conditioned Imitation Learning using Score-based Diffusion Policies

  • Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, Ping Luo
  • Key: Goal-Conditioned Imitation Learning, Robotics, Classifier-free
  • ExpEnv: CALVIN, Block-Push, Relay Kitchen

ICML 2023

• Optimizing DDPM Sampling with Shortcut Fine-Tuning

  • Ying Fan, Kangwook Lee
  • Publisher: ICML 2023
  • Key: Training Diffusion with RL, Online RL, Sampling Optimization
  • Code: official
  • ExpEnv: CIFAR10, CelebA

• MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL

  • Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, Zhixuan Liang
  • Publisher: ICML 2023
  • Key: Offline meta-RL, Conditional Trajectory Generation, Generalization, Classifier-guided
  • ExpEnv: MuJoCo

• Hierarchical diffusion for offline decision making

  • Wenhao Li, Xiangfeng Wang, Bo Jin, Hongyuan Zha
  • Publisher: ICML 2023
  • Key: Hierarchical Offline RL, Long Horizon Task, Classifier-free
  • Code: official
  • ExpEnv: MuJoCo, D4RL, NeoRL

• Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning

  • Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, Jun Zhu
  • Publisher: ICML 2023
  • Key: Offline RL, Constrained Policy Optimization, Classifier-guided
  • Code: official
  • ExpEnv: MuJoCo, D4RL

ICLR 2023

• Is Conditional Generative Modeling all you need for Decision-Making?

  • Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, Pulkit Agrawal
  • Publisher: ICLR 2023
  • Key: Offline RL, Generative Model, Policy Optimization, Classifier-free
  • Code: official
  • ExpEnv: D4RL

• Imitating Human Behaviour with Diffusion Models

  • Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, Sam Devlin
  • Publisher: ICLR 2023
  • Key: Offline RL, Policy Optimization, Imitation Learning, Classifier-free
  • ExpEnv: Claw, Kitchen, CSGO

• Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

  • Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, Jun Zhu
  • Publisher: ICLR 2023
  • Key: Offline RL, Generative models
  • Code: official
  • ExpEnv: D4RL

ICRA 2023

• Guided Conditional Diffusion for Controllable Traffic Simulation
  • Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, Marco Pavone
  • Publisher: ICRA 2023
  • Key: Traffic Simulation, Multi-Agent, Classifier-free
  • ExpEnv: nuScenes

NeurIPS 2022

• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
  • Zhendong Wang, Jonathan J Hunt, Mingyuan Zhou
  • Publisher: NeurIPS Deep RL Workshop 2022
  • Key: Offline RL, Policy Optimization
  • Code: official, unofficial
  • ExpEnv: MuJoco, D4RL

ICML 2022

• Planning with Diffusion for Flexible Behavior Synthesis
  • Michael Janner, Yilun Du, Joshua B. Tenenbaum, Sergey Levine
  • Publisher: ICML 2022 (long talk)
  • Key: Offline RL, Model-based RL, Trajectory Optimization, Classifier-guided
  • Code: official
  • ExpEnv: MuJoco, D4RL

Contributing

Our purpose is to make this repo even better. If you are interested in contributing, please refer to HERE for instructions in contribution.

License

Awesome Diffusion Model in RL is released under the Apache 2.0 license.