How to save and change custom data into a exist dataset?

[Neutron3529]

for example, I have a dataset with data and label(e.g.,imagenet-2012)
I want to calculate current loss yhat=net(x);loss=loss_fn(yhat,y) and read such loss in the next epoch:
e.g.,yhat=net(x);loss=loss_fn(yhat,y)*loss_prev_epoch
I cannot save such loss since training dataset always turn on the shuffle option and I cannot shuffle the recorded loss and the original data in the same time

how to make such change?

[szha]

What I would recommend in this case is to:

• turn off shuffle and sampling in dataloader
• calculate loss of the training dataset in the original order
• save loss as a numpy array
• use the gluon.dataset.ArrayDataset to construct a new dataset with the additional field for loss

[szha]

@zhreshold can answer on the performance comparison for datasets.

[zhreshold]

Using ImageRecord over data.ImageNet offers better IO, otherwise these two are mostly interchanable in terms of transform perf.

[Neutron3529]

@zhreshold can answer on the performance comparison for datasets.

Thank you.
I think I could try combining a "index" term into the full data, then

custom_weight=mx.nd.zeros((data_count,))
for data,label,index in data_iter:
  x=data.as_in_context(mx.gpu())
  y=label.as_in_context(mx.gpu())
  with ag.record():
    yhat=net(x)
    loss=loss_fn(yhat,y)
  loss.backward()
  custom_weight[index]=weight_fun(yhat,y,x)#implicit copy data from gpu to cpu

Is it worth a PR?
I am going to try add a "index=False" parameter to dataloader, if such parameter worth a PR, I'll submit a PR.

[zhreshold]

Instead of modifying dataloader to return the indices, I would rather overwrite the RandomSampler to save the indices to a list and checkpointing them after each epoch, you can retrieve the indices in the next epoch then. DataLoader allows you to set the sampler to sampler=CustomRandomSampler rather than shuffle.

[Neutron3529]

It IS the solution.
Thank you!

[Neutron3529]

as @zhreshold mentioned above

Instead of modifying dataloader to return the indices, I would rather overwrite the RandomSampler to save the indices to a list and checkpointing them after each epoch, you can retrieve the indices in the next epoch then. DataLoader allows you to set the sampler to sampler=CustomRandomSampler rather than shuffle.

That is the solution with no modifying the mxnet API.
Only 3 steps is needed:

1. write a CustomRandomSampler using a global variable index, update index and return after each iter.
2. using sampler=CustomRandomSampler istead of the default setting.
3. create a custom_weight array, to save the right weights.

I'll try imagenet dataset after bought a rtx3090.

[zhreshold]

wow, good luck with the rtx3090!