First* generation with IF

[tildebyte]

*OK, white lie, it's actually my third, but it's the first one where I think I've got control over the whole process 😁

Prompt: "a black and white medium format 85mm portrait of a kitten wearing a tuxedo on his way to a funeral, the image is high quality and highly detailed with the kitten's features clearly visible, photographer Edward Weston used Agfa Isopan ISO 25 film to create this image, which resembles Edward Weston's photograph Pepper No. 35"

(this is my canonical test prompt for all image generating models :D)

This was generated using a RunPod RTX A6000 w/48G VRAM, 12 vCPU, 62G RAM. It takes around 8-10 minutes to "warm up", then generates the image in roughly 60 seconds (at between 4-6 it/s depending on the stage. Weird.
I used a VSCode Remote Explorer SSH connection to run VSCode remotely, installing the Jupyter extension (into the pod) so I could open the "pipes_DeepFloyd_IF_v1_0" notebook (which I heavily modified - code follows)

# This isn't a notebook, so non-Python text is commented out
# !python3.10 -m pip install --upgrade pip
# %pip install \
#   --extra-index-url https://download.pytorch.org/whl/cu117 \
#   accelerate~=0.15.0 \
#   beautifulsoup4~=4.11.1 \
#   deepfloyd-if==1.0.1 \
#   diffusers~=0.16.0 \
#   ftfy~=6.1.1 \
#   huggingface_hub>=0.13.2 \
#   matplotlib \
#   numpy \
#   omegaconf \
#   Pillow>=9.2.0 \
#   protobuf==3.20.1 \
#   safetensors \
#   sentencepiece~=0.1.97 \
#   tokenizers~=0.13.2 \
#   torch==1.13.1+cu117 \
#   torchaudio==0.13.1 \
#   torchvision==0.14.1+cu117 \
#   tqdm \
#   transformers~=4.25.1 \
#   xformers==0.0.16 \
#   git+https://github.com/openai/CLIP.git

# *must* be from terminal
from huggingface_hub import login
login()

import os

os.environ['FORCE_MEM_EFFICIENT_ATTN'] = "1"

from datetime import datetime
from deepfloyd_if.pipelines import dream
from deepfloyd_if.modules import IFStageI, IFStageII, StableStageIII
from deepfloyd_if.modules.t5 import T5Embedder

device = 'cuda:0'
if_I = IFStageI('IF-I-XL-v1.0', device=device)
if_II = IFStageII('IF-II-L-v1.0', device=device)
if_III = StableStageIII('stable-diffusion-x4-upscaler', device=device)
t5 = T5Embedder(device="cpu")

prompt = ("a black and white medium format 85mm portrait of a kitten wearing a tuxedo on his"
          "way to a funeral, the image is high quality and highly detailed with the kitten's"
          "features clearly visible, photographer Edward Weston used Agfa Isopan ISO 25 film"
          "to create this image, which resembles Edward Weston's photograph Pepper No. 35")

result = dream(
    t5=t5, if_I=if_I, if_II=if_II, if_III=if_III,
    prompt=[prompt],
    seed = int((datetime.utcnow().timestamp() * 10 ** 6) % (2 ** 32 - 1)),
    negative_prompt=['picture frame'],
    aspect_ratio='3:2',
    if_I_kwargs={
        "guidance_scale": 7.0,
        "sample_timestep_respacing": "smart100",
    },
    if_II_kwargs={
        "guidance_scale": 4.0,
        "sample_timestep_respacing": "smart50",
    },
)

if_I.show(result['III'], size=14)

and that's all there is to it. If you use VSCode as I did, there's a tiny floppy-disk image which appears in the upper right when you hover over a generated image; click it to save locally.

This doesn't have to be expensive, by the way. As much stumbling around as I did, it cost slightly less than 2USD to create this image (of course, I'm already pretty confident in the prompt - I certainly wouldn't use RunPod for prompt exploration...)

[cswiger]

Nice prompt - I copied that to play with and notice that runs some of the words together: hisway kitten'sfeatures filmto
Needs a space at the end of each quoted string.

Was looking for a way to speed up the T5Embedder(device="cpu") as it takes hours here running on a single core.
I bet if you changed device="cpu" to device=device on that RTX A6k the "warm up" would become negligible :)

[tildebyte]

notice that runs some of the words together: hisway kitten'sfeatures filmto
Needs a space at the end of each quoted string.

Yeah, derp, I noticed that after posting 🙄

I bet if you changed device="cpu" to device=device on that RTX A6k the "warm up" would become negligible

I'll try it!