[BUG] A bug while fine-tuning the model by iteratively training and evaluating using a sliding time window

[hk63560892]

Bug description

I find out that there is no label in valid.parquet.

Steps/Code to reproduce bug

While I m running this code:
start_time_window_index = 1
final_time_window_index = 4
for time_index in range(start_time_window_index, final_time_window_index):
# Set data
time_index_train = time_index
time_index_eval = time_index + 1
train_paths = glob.glob(os.path.join(OUTPUT_DIR, f"{time_index_train}/train.parquet"))
eval_paths = glob.glob(os.path.join(OUTPUT_DIR, f"{time_index_eval}/valid.parquet"))
# Train on day related to time_index
print('*'20)
print("Launch training for day %s are:" %time_index)
print(''20 + '\n')
trainer.train_dataset_or_path = train_paths
trainer.reset_lr_scheduler()
trainer.train()
trainer.state.global_step +=1
# Evaluate on the following day
trainer.eval_dataset_or_path = eval_paths
train_metrics = trainer.evaluate(metric_key_prefix='eval')
print(''20)
print("Eval results for day %s are:\t" %time_index_eval)
print('\n' + ''*20 + '\n')
for key in sorted(train_metrics.keys()):
print(" %s = %s" % (key, str(train_metrics[key])))
wipe_memory()

the error appear:

---

Launch training for day 1 are:

---

/usr/local/lib/python3.10/dist-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set no_deprecation_warning=True to disable this warning
warnings.warn(
{'train_runtime': 4.0234, 'train_samples_per_second': 3817.691, 'train_steps_per_second': 14.913, 'train_loss': 10.525657145182292, 'epoch': 60.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:04<00:00, 14.92it/s]
TrainOutput(global_step=60, training_loss=10.525657145182292, metrics={'train_runtime': 4.0234, 'train_samples_per_second': 3817.691, 'train_steps_per_second': 14.913, 'total_flos': 0.0, 'train_loss': 10.525657145182292})
Traceback (most recent call last):
File "", line 17, in
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2932, in evaluate
output = eval_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/trainer.py", line 515, in evaluation_loop
metrics_results_detailed = model.calculate_metrics(preds, labels)
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/base.py", line 616, in calculate_metrics
head.calculate_metrics(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/base.py", line 453, in calculate_metrics
task.calculate_metrics(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/prediction_task.py", line 489, in calculate_metrics
result = metric(predictions, targets)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 301, in forward
self._forward_cache = self._forward_full_state_update(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 316, in _forward_full_state_update
self.update(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 465, in wrapped_func
update(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/ranking_metric.py", line 56, in update
metric = self._metric(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/ranking_metric.py", line 137, in _metric
if rel_indices.shape[0] > 0:
IndexError: tuple index out of range

Expected behavior

I expected there have label for evaluation

Environment details

• Transformers4Rec version: 23.12
• Platform:Docker
• Python version:3.10
• Huggingface Transformers version:4.27.1
• PyTorch version (GPU?):2.1.0a0+4136153
• Tensorflow version (GPU?):

Additional context

[rnyak]

@hk63560892 please share the link to the example notebook you are running? and what docker image you are using?

[hk63560892]

link:
https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/main/examples/tutorial/03-Session-based-recsys.ipynb
docker:
docker run -it --gpus device=0 -p 8000:8000 -p 8001:8001 -p 8002:8002 -p 8888:8888 -v <path_to_data>:/workspace/data/ nvcr.io/nvidia/merlin/merlin-pytorch:23.XX

thankyou!!

[rnyak]

@hk63560892 what docker image tag you are using? which 23.XX you are using? we have several ones start with 23. please be specific.

also note that the tutorials have not been maintained for a while so you can refer to other example notebooks in the examples directory.