Train CLIP model with own data

[yafeng-c]

Hi,
Thanks for your amazing work.
It's more like a discussion. Anyway, I am trying to train the CLIP model with my own data.

train_samples = []
a loop:
    inp_example = InputExample(texts=[imgdata, textdata], label=scoredata)
    train_samples.append(inp_example)
train_dataloader = DataLoader(train_samples, batch_size=BATCH_SIZE, shuffle=True)

model =  SentenceTransformer('clip-ViT-B-32')

model.fit(train_objectives=[(train_dataloader, train_loss)],
          evaluator=evaluator,
          epochs=num_epochs,
          evaluation_steps=10,
          warmup_steps=warmup_steps,
          output_path=model_save_path)

It can run. But is this correct? By doing so, are all layers retrained? Is it possible to freeze some layers? (I used Tensorflow before, new in Pytorch).

[nreimers]

Looks good. Only the definition of the loss is missing

[milmin]

Hi how do we put image data in InputExample? Should we process image data? Because DataLoader does not accept PIL images...

[breadfan]

Have you got any workable model with this? I can't figure out how to make fine-tuning using images and promts

[tomaarsen]

Hello @breadfan & others,

I've yet to review this pull request, but examples/unsupervised_learning/CLIP/sample_clip.ipynb from #2390 seems to show how to finetune a CLIP model. It might be helpful for you all. When I have some more time, then I'll review that PR and add the finetuning example to the repository itself.

• Tom Aarsen