[OOM] Fine tuning CLIP

[AndrMoura]

Hello, Im trying to fine-tune a CLIP model with my own data (image-description pairs) with a GPU but mid-training Im getting OOM RAM. The RAM memory during training slowly goes up and up until oom.

This is a sample from my code:

model = SentenceTransformer("sentence-transformers/clip-ViT-B-32")

train_examples = [InputExample(texts=[Image.open(os.path.join(img_path, row[1]['img_name'])), row[1]['description']])  for row in train_captions.iterrows()]

train_dataloader = DataLoader(train_examples, 
                              shuffle=True, 
                              batch_size=16)

train_loss = losses.MultipleNegativesRankingLoss(model=model)
# setup evaluator
...

model.fit([(train_dataloader, train_loss)], 
            show_progress_bar=True,
            epochs=10,
            output_path=output_path)

I believe the problem lies on the 2nd line. When I change the train_examples to load text only:
train_examples = [InputExample(texts=[row[1]['description'], row[1]['description']]) for row in train_captions.iterrows()]

The model trains without any memory issues!! I must be doing something wrong with the image loading. What is the proper way to load images in train_examples variable?

Thank you.

[jpzhangvincent]

I'm also looking for examples for finetuning the CLIP model with sentence-transformer. Thanks!

[rhkenne]

Hey @nreimers, congrats on your move/promotion to cohere.ai. I would like to open a PR and address this issue. Any pointers on how to approach it ?

[yash-120304]

@AndrMoura did u solve ur problem?
can u tell me why did u not put label in ur InputExample

[AndrMoura]

@AndrMoura did u solve ur problem? can u tell me why did u not put label in ur InputExample

I didn't. I used the HF library to train my own CLIP.

As for the label, check the MultipleNegativesRankingLoss .

[yash-120304]

image= mapping.keys()
captions=mapping.values()
train_samples=[]
for img,caps in tqdm(zip(image,captions)):
img=img+'.jpg'
img=Image.open(os.path.join(img_dir,img))
image_emb = clip.encode([img], convert_to_tensor=True, show_progress_bar=False)
for cap in caps:
cap_emb = clip.encode([cap], convert_to_tensor=True, show_progress_bar=False)
score = util.semantic_search(image_emb,cap_emb)
input_example=InputExample(texts=[img,cap],label=score)
train_samples.append(input_example)

This is how i am computing my score is it correct?

[yash-120304]

HF library to train my own CLIP

Ohh how can I do that?

bcoz for now i am simply using sentenceTransformers library to import my clip model and am getting good results but i cant evaluate it and this is where i am stuck.

[httplups]

HI, I am having the same error with memory