[v3] How do you envision this working with local datasets?

[smerrill]

Hi, and thanks very much for this awesome library. I'm attempting to update my training pipeline to v3, but I've hit a snag; when I try to load a local Parquet file as a dataset, the model card generator attempts to hit the HF hub to look for my dataset, which it can't find. For now I could fake the name to be the same as an existing dataset, but what would you like the workflow to be for this in the future?

My test code:

dataset = load_dataset("./training_dataset/", data_files={'data.parquet'})['train'].train_test_split(test_size=0.2)

train_dataset = dataset["train"]
eval_dataset = dataset["test"]

model = SentenceTransformer(MODEL)
guide_model = SentenceTransformer(GUIDE_MODEL)

# 3. Define a loss function
loss = losses.CachedGISTEmbedLoss(model, mini_batch_size=16, guide=guide_model)

training_args = SentenceTransformerTrainingArguments(
    output_dir="checkpoints",
    num_train_epochs=10,
    per_device_train_batch_size=2048,
    per_device_eval_batch_size=2048,
    warmup_ratio=0.2,
    bf16=True
)

# 4. Create a trainer & train
trainer = SentenceTransformerTrainer(
    model=model,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    loss=loss,
    args=training_args
)
trainer.train()

And the end of my stack trace looks like the following:

RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-663d4582-26fdd83d076b6389554b8f7b;3ae1d179-d45b-41f0-aee4-2e2302089c7b)

Repository Not Found for url: https://huggingface.co/api/datasets/training_dataset.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

[smerrill]

For now I added a try: except: pass stanza around https://github.com/UKPLab/sentence-transformers/blob/v3.0-pre-release/sentence_transformers/model_card.py#L69 and https://github.com/UKPLab/sentence-transformers/blob/v3.0-pre-release/sentence_transformers/model_card.py#L74.

What do you think about a fetch_dataset_metadata property on SentenceTransformerTrainingArguments? If that sounds agreeable I can work up a PR.

[tomaarsen]

Hello!

This sounds like an edge case that I've missed. Could you give a slightly more detailed traceback? I'd love to see which call to Hugging Face failed. I have a suspicion that it might be in a location where potential failures should be caught. After all, the "extract metadata" method just exists to try and fetch some metadata, but that's not always possible - if it's not possible, it should just quietly fail and continue.

Also, thanks for experimenting with v3! These kinds of issues is how I can try and prevent big issues in the full release 🤗

• Tom Aarsen

[tomaarsen]

I'll close this as I merged the fix into v3.0-pre-release via #2636. Thanks for reporting and feel free to share if you have more feedback (positive or negative).

• Tom Aarsen