Skip to content

Commit

Permalink
wrap prepared_ds_path in str() to avoid TypeError in fsspec package
Browse files Browse the repository at this point in the history
`fsspec` calls `if "::" in path` on `prepared_ds_path`, which will throw an error if it is a `PosixPath` object.
  • Loading branch information
FrankRuis authored Apr 19, 2024
1 parent c10563c commit c9719f1
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/axolotl/utils/data/sft.py
Original file line number Diff line number Diff line change
Expand Up @@ -421,7 +421,7 @@ def for_d_in_datasets(dataset_configs):

if cfg.local_rank == 0:
LOG.info(f"Saving merged prepared dataset to disk... {prepared_ds_path}")
dataset.save_to_disk(prepared_ds_path)
dataset.save_to_disk(str(prepared_ds_path))
if cfg.push_dataset_to_hub:
LOG.info(
f"Saving merged prepared dataset with push_to_hub... {cfg.push_dataset_to_hub}/{ds_hash}"
Expand Down

0 comments on commit c9719f1

Please sign in to comment.