Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Checkpointing issues in 0.4.0 #1549

Open
TuKo opened this issue Dec 15, 2023 · 2 comments
Open

Checkpointing issues in 0.4.0 #1549

TuKo opened this issue Dec 15, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@TuKo
Copy link

TuKo commented Dec 15, 2023

When trying to save a checkpoint the, torch.save(checkpoint_data, fname, pickle_module=dill) call in save_checkpoint() function fails. It complains about a recursive definition when calling torch.save() inside the respective function.

Additional context:
I've followed the checkpoint example with the few changes below.
The model is a CNN model (inheriting from DynamicModule) with an IncrementalClassifier as the output layer. This is in order to do class-incremental learning. Starts with 2 output classes and adapts in the train-eval loop.
The dataset is MNIST loaded from torchvision (not avalanche) and the benchmark is created with nc_benchmark with 10 experiences and task_labels=False. Only ToTensor() transform is used for the data.
The strategy is Naive. I'm using a ReplayBuffer, which I try removing it but still fails.
The evaluation plugin has the accuracy_metrics, the forgetting_metrics, and an interactive logger.
The main loop calls adaptation(), train(), eval() in that order.

System: Mac 2023 with M2

**Versions: **

avalanche-lib 0.4.0
torch 2.1.1 
torchvision 0.16.1
python 3.10.13

/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py:412: PicklingWarning: Cannot locate reference to <class 'typing.ParamSpec'>.
  StockPickler.save(self, obj, save_persistent_id)
/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py:412: PicklingWarning: Cannot pickle <class 'typing.ParamSpec'>: typing.ParamSpec has recursive self-references that trigger a RecursionError.
  StockPickler.save(self, obj, save_persistent_id)
Traceback (most recent call last):
  File "/Users/me/projects/cl/train.py", line 172, in <module>
    main(args)
  File "/Users/me/projects/cl/train.py", line 144, in main
    save_checkpoint(strategy, filename)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/avalanche/training/checkpoint.py", line 88, in save_checkpoint
    torch.save(checkpoint_data, fname, pickle_module=dill)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/torch/serialization.py", line 619, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol, _disable_byteorder_record)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/torch/serialization.py", line 831, in _save
    pickler.dump(obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 418, in dump
    StockPickler.dump(self, obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 487, in dump
    self.save(obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1212, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1212, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 687, in save_reduce
    save(cls)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1812, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1093, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 902, in save_tuple
    save(element)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 887, in save_tuple
    save(element)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1812, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1107, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1812, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1093, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 902, in save_tuple
    save(element)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1212, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1990, in save_function
    StockPickler.save_global(pickler, obj, name=name)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 1100, in save_global
    self.save_reduce(getattr, (parent, lastname))
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 887, in save_tuple
    save(element)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1849, in save_type
    StockPickler.save_global(pickler, obj, name=obj_name)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 1076, in save_global
    raise PicklingError(
_pickle.PicklingError: Can't pickle <class 'typing.ParamSpec'>: it's not the same object as typing.ParamSpec
@TuKo TuKo added the bug Something isn't working label Dec 15, 2023
@ghost

This comment was marked as spam.

Sign in to view
@AntonioCarta
Copy link
Collaborator

@lrzpellegrini do you know if there are any workaround for this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@TuKo @AntonioCarta