AnimateDiff

This a fork of the official repo. Made specifically to run in Colab with gradio UI. Local requires 12GB VRAM

(Original colab by @camenduru)

Features

• 31/07/2023 Additional GIF output. Default displays GIF since autolooping on gradio mp4 is not directly supported. Frames are also saved into the output folder. MP4 is still accessible via the mp4 tab in the output area of the UI.
• 30/07/2023 Longer videos via moving context window. Experimental. (Credit to https://github.com/dajes/AnimateDiff/tree/longer_videos)
• 29/07/2023 Init image. Euler sampler is disabled when using init images. (Credit to https://github.com/talesofai/AnimateDiff)
• 28/07/2023 Download models via UI
• 28/07/2023 100% inference speed due to fp16 (Credit to https://github.com/dajes/AnimateDiff/tree/longer_videos)
• 28/07/2023 Loading of multiple LoRAs (Without degrading the network with each generation)

Testing and replication

Example	Caption	Inference time	Config
	Euler with FP32 instead of FP16	~200s	sampler: "Euler" FP16: False
	Euler	~80s	sampler: "Euler"
	DDIM	~80s	sampler: "DDIM"
	PNDM	~110s	sampler: "PNDM"
	Euler with init image	~80s	sampler: "Euler" init_image: "configs/prompts/yoimiya-init.jpg"
	Double length video DDIM with init image using sliding context	~700s	sampler: "DDIM" init_image: "configs/prompts/yoimiya-init.jpg" temporal_context: 20 overlap: 5 strides: 1

Init image:

Common config between tests:

{
    "stable_diffusion": "/content/AnimateDiff/models/StableDiffusion/stable-diffusion-v1-5/",
    "motion_model": "mm_sd_v14.ckpt",
    "base_checkpoint": "AnythingV5Ink_v5PrtRE.safetensors",
    "prompt": "1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece \uff0cBestQuality \uff0cUltraDetailed",
    "n_prompt": "NSFW, lr, nsfw,(sketch, duplicate, ugly, huge eyes, text, logo, monochrome, worst face, (bad and mutated hands:1.3), (worst quality:2.0), (low quality:2.0), (blurry:2.0), horror, geometry, bad_prompt_v2, (bad hands), (missing fingers), multiple limbs, bad anatomy, (interlocked fingers:1.2), Ugly Fingers, (extra digit and hands and fingers and legs and arms:1.4), crown braid, ((2girl)), (deformed fingers:1.2), (long fingers:1.2),succubus wings,horn,succubus horn,succubus hairstyle, (bad-artist-anime), bad-artist, bad hand, grayscale, skin spots, acnes, skin blemishes",
    "num_inference_steps": 25,
    "guidance_scale": 7.5,
    "width": 512,
    "height": 512,
    "video_length": 40,
    "seed": 83026725601855,
    "temporal_context": 20,
    "strides": 1,
    "overlap": 5,
    "fp16": true,
    "lora_list": [
        {
            "path": "/content/AnimateDiff/models/loras/LineLine2D.safetensors",
            "alpha": 0.8
        },
        {
            "path": "/content/AnimateDiff/models/loras/yomiya.safetensors",
            "alpha": 0.8
        }
    ]
}

Why not build this as an A1111 extension?

• There's already an A1111 Extension at https://github.com/continue-revolution/sd-webui-animatediff
• Some features are easier to implement by building this seperately. It will allow us to explore different techniques to improve AnimateDiff.

Definitions

context_length: the length of the sliding window (limited by motion modules capacity), default to L.
context_overlap: how much neighbouring contexts overlap. By default context_length / 2
context_stride: (2^context_stride) is a max stride between 2 neighbour frames. By default 0

By dajes @ https://github.com/guoyww/AnimateDiff/pull/25

From original readme

AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
Yuwei Guo, Ceyuan Yang*, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, Bo Dai

*Corresponding Author

Features

• GPU Memory Optimization, ~12GB VRAM to inference
• User Interface: Gradio, A1111 WebUI Extension sd-webui-animatediff (by @continue-revolution)
• Google Colab: Colab (by @camenduru)

Common Issues

Installation

Please ensure the installation of xformer that is applied to reduce the inference memory.

Various resolution or number of frames

Currently, we recommend users to generate animation with 16 frames and 512 resolution that are aligned with our training settings. Notably, various resolution/frames may affect the quality more or less.

How to use it without any coding

1. Get lora models: train lora model with A1111 based on a collection of your own favorite images (e.g., tutorials English, Japanese, Chinese) or download Lora models from Civitai.

2. Animate lora models: using gradio interface or A1111 (e.g., tutorials English, Japanese, Chinese)

3. Be creative togther with other techniques, such as, super resolution, frame interpolation, music generation, etc.

Animating a given image

We totally agree that animating a given image is an appealing feature, which we would try to support officially in future. For now, you may enjoy other efforts from the talesofai.

Contributions from community

Contributions are always welcome!! The dev branch is for community contributions. As for the main branch, we would like to align it with the original technical report :)

Setup for Inference

Prepare Environment

We updated our inference code with xformers and a sequential decoding trick. Now AnimateDiff takes only ~12GB VRAM to inference, and run on a single RTX3090 !!

git clone https://github.com/guoyww/AnimateDiff.git
cd AnimateDiff

conda env create -f environment.yaml
conda activate animatediff

if using --format mp4

pip install "imageio[ffmpeg]"

Download Base T2I & Motion Module Checkpoints

We provide two versions of our Motion Module, which are trained on stable-diffusion-v1-4 and finetuned on v1-5 seperately. It's recommanded to try both of them for best results.

git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/

bash download_bashscripts/0-MotionModule.sh

You may also directly download the motion module checkpoints from Google Drive / HuggingFace / CivitAI, then put them in models/Motion_Module/ folder.

Prepare Personalize T2I

Here we provide inference configs for 6 demo T2I on CivitAI. You may run the following bash scripts to download these checkpoints.

bash download_bashscripts/1-ToonYou.sh
bash download_bashscripts/2-Lyriel.sh
bash download_bashscripts/3-RcnzCartoon.sh
bash download_bashscripts/4-MajicMix.sh
bash download_bashscripts/5-RealisticVision.sh
bash download_bashscripts/6-Tusun.sh
bash download_bashscripts/7-FilmVelvia.sh
bash download_bashscripts/8-GhibliBackground.sh
bash download_bashscripts/9-AdditionalNetworks.sh

Inference

After downloading the above peronalized T2I checkpoints, run the following commands to generate animations. The results will automatically be saved to samples/ folder.

python -m scripts.animate --config configs/prompts/1-ToonYou.yaml
python -m scripts.animate --config configs/prompts/2-Lyriel.yaml
python -m scripts.animate --config configs/prompts/3-RcnzCartoon.yaml
python -m scripts.animate --config configs/prompts/4-MajicMix.yaml
python -m scripts.animate --config configs/prompts/5-RealisticVision.yaml
python -m scripts.animate --config configs/prompts/6-Tusun.yaml
python -m scripts.animate --config configs/prompts/7-FilmVelvia.yaml
python -m scripts.animate --config configs/prompts/8-GhibliBackground.yaml
python -m scripts.animate --config configs/prompts/9-AdditionalNetworks.yml

To generate animations with a new DreamBooth/LoRA model, you may create a new config .yaml file in the following format:

NewModel:
  path: "[path to your DreamBooth/LoRA model .safetensors file]"
  base: "[path to LoRA base model .safetensors file, leave it empty string if not needed]"

  motion_module:
    - "models/Motion_Module/mm_sd_v14.ckpt"
    - "models/Motion_Module/mm_sd_v15.ckpt"
    
  steps:          25
  guidance_scale: 7.5

  prompt:
    - "[positive prompt]"

  n_prompt:
    - "[negative prompt]"

Then run the following commands:

python -m scripts.animate --config [path to the config file]

Gradio Demo

We have created a Gradio demo to make AnimateDiff easier to use. To launch the demo, please run the following commands:

conda activate animatediff
python app.py

By default, the demo will run at localhost:7860.

Gallery

Here we demonstrate several best results we found in our experiments.

Model：ToonYou

Model：Counterfeit V3.0

Model：Realistic Vision V2.0

Model： majicMIX Realistic

Model：RCNZ Cartoon

Model：FilmVelvia

Longer generations

You can also generate longer animations by using overlapping sliding windows.

python -m scripts.animate --config configs/prompts/{your_config}.yaml --L 64 --context_length 16 

Sliding window related parameters:

L - the length of the generated animation.

context_length - the length of the sliding window (limited by motion modules capacity), default to L.

context_overlap - how much neighbouring contexts overlap. By default context_length / 2

context_stride - (2^context_stride) is a max stride between 2 neighbour frames. By default 0

Extended this way gallery examples

Model：ToonYou

Model：Realistic Vision V2.0

Community Cases

Here are some samples contributed by the community artists. Create a Pull Request if you would like to show your results here😚.

Character Model：Yoimiya (with an initial reference image, see WIP fork for the extended implementation.)

Character Model：Paimon; Pose Model：Hold Sign

BibTeX

@article{guo2023animatediff,
  title={AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning},
  author={Guo, Yuwei and Yang, Ceyuan and Rao, Anyi and Wang, Yaohui and Qiao, Yu and Lin, Dahua and Dai, Bo},
  journal={arXiv preprint arXiv:2307.04725},
  year={2023}
}

Contact Us

Yuwei Guo: [email protected]
Ceyuan Yang: [email protected]
Bo Dai: [email protected]

Acknowledgements

Codebase built upon Tune-a-Video.