Have any got any successful example fine-tuned models?

[kabachuha]

Hi!

Wondering about any experiments with training that are going on 🙂

[ExponentialML]

Hi!

Wondering about any experiments with training that are going on slightly_smiling_face

I could run some experiments based on what the community wants. I've tried single video training (it's also implemented) and seems to work really well.

[Cubey42]

well I'm kinda dumb so I can't actually run the model and get an output using the code, but I do get outputs while it trains, I've done only some basic training (I took like 10 gifs of an anime girl and absolutely baked a model.
50000 steps was overkill for my small dataset, seemed to be finished learning at 5k steps.
at 256x256 I could run a batch size of 3 on a 4090, but I could do 384x384 at batch size 1, but the results haven't been nearly as good from the outputs I've seen, but I'll probably let it go to 10k before stopping it.

EDIT: okay so the video seems broken but it does work:
https://files.catbox.moe/jaw3ab.mp4

so it works, I just can't actually run the model outside of training yet

[ExponentialML]

so it works, I just can't actually run the model outside of training yet

It should work after training. After training is done, a checkpoint should be saved in './outputs'.

There should also be checkpoints based on this line.

[Cubey42]

I fired some neurons and got it the parameters I was looking for 👌

[sergiobr]

Great.
@Cubey42 by the way the video you posted above gives just a green screen to me.

I'm now working on some tiktok videos. The first bottleneck I found is outside this repo that is the BLIP process. I split the videos in pieces of 4 frames each. So the captioning process is taking so long (there about 800 'vids').
I'll check if it can already runs in parallel and I can work on this if it doesn't.

As soon I have my first results I'll post here.

[ExponentialML]

@sergiobr Yes I'm aware and agree that it's constraining (this is why the majority of my testing is single video, which works super well due to the large pretrained model domain).

I haven't really found a good automated way to do this due to context changes in video. It's much easier to do it for images and the like.

One way to speed things up is this parameter here for BLIP processing.

--prompt_amount, help="The amount of prompts per video that is processed.", default=25

You could probably do --prompt_amount 1 so that it only comes up with one prompt per video.

If you have 800 videos and are creating 25 prompts per 800, that's 20K prompts 🙂 .

[sergiobr]

@ExponentialML Oh, sounds great, I'll try this!
Yesterday I had success ˜˜over˜˜training some videos.
At the end it was really nice but I trained too much and with a high LR.
I'm thinking in insert class videos to do not change the model too much. But it's nice to know that it works.

[B34STW4RS]

I was wondering what the optimal video parameters would be, like resolution/length per clip?

[kabachuha]

Adding Animov-0.1 model here because it's a huge success

Original Diffusers weights:
https://huggingface.co/datasets/strangeman3107/animov-0.1

WebUI weights:
https://huggingface.co/kabachuha/animov-0.1-modelscope-original-format

Uploading w.mp4…

[1-sf]

can we finetune it with multiple video-1.mp4, video-2.mp4, ...... so on and their captions in video-1.txt, video-2.txt, .... so on

[1-sf]

edit: I ended up finetuning it with multiple videos. Encountered one issue, fixed it(for my usecase) and raised #63 for the benefit of the community