exp run: --temp crashes

[jorgeorpinel]

Bug Report

Description

exp run --temp throws a Python error

Reproduce

$ git clone [email protected]:iterative/example-get-started.git
$ cd example-get-started
$ dvc pull
$ dvc exp run --temp
'data/data.xml.dvc' didn't change, skipping
Stage 'prepare' didn't change, skipping
Stage 'featurize' didn't change, skipping
Stage 'train' didn't change, skipping
Stage 'evaluate' didn't change, skipping
Exception ignored in: <finalize object at 0x7ff0a023d6b0; dead>
Traceback (most recent call last):
  File "/usr/lib/python3.6/weakref.py", line 548, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/usr/lib/python3.6/tempfile.py", line 938, in _cleanup
    _rmtree(name)
  File "/usr/lib/python3.6/shutil.py", line 477, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 475, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/jop/DVC-repos/example-get-started/.dvc/tmp/exps/tmpnu_88y_x'

Expected

Per 1. in #5593, it should generate an experiment (identical to HEAD in this case).

Environment information

$ dvc doctor
DVC version: 2.0.5
---------------------------------
Platform: Python 3.6.9 on Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-Ubuntu-18.04-bionic
Supports: All remotes
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sdb
Repo: dvc, git

[pmrowla]

I'm able to reproduce this in Python 3.6 on linux but not in Python 3.8

(.venv-linux-38) vagrant@ubuntu-bionic:~/example-get-started$ dvc exp run --temp
'data/data.xml.dvc' didn't change, skipping
Stage 'prepare' didn't change, skipping
Stage 'featurize' didn't change, skipping
Stage 'train' didn't change, skipping
Stage 'evaluate' didn't change, skipping
(.venv-linux-38) vagrant@ubuntu-bionic:~/example-get-started$ dvc doctor
DVC version: 2.0.4+e4c37f
---------------------------------
Platform: Python 3.8.8 on Linux-4.15.0-136-generic-x86_64-with-glibc2.27
Supports: All remotes
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda1
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sda1
Repo: dvc, git

(.venv-linux-36) vagrant@ubuntu-bionic:~/example-get-started$ dvc exp run --temp
'data/data.xml.dvc' didn't change, skipping
Stage 'prepare' didn't change, skipping
Stage 'featurize' didn't change, skipping
Stage 'train' didn't change, skipping
Stage 'evaluate' didn't change, skipping
Exception ignored in: <finalize object at 0x7f129ef00690; dead>
Traceback (most recent call last):
  File "/usr/lib/python3.6/weakref.py", line 548, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/usr/lib/python3.6/tempfile.py", line 938, in _cleanup
    _rmtree(name)
  File "/usr/lib/python3.6/shutil.py", line 477, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 475, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/vagrant/example-get-started/.dvc/tmp/exps/tmpjauqau6l'
(.venv-linux-36) vagrant@ubuntu-bionic:~/example-get-started$ dvc doctor
DVC version: 2.0.4+e4c37f
---------------------------------
Platform: Python 3.6.9 on Linux-4.15.0-136-generic-x86_64-with-Ubuntu-18.04-bionic
Supports: All remotes
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda1
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sda1
Repo: dvc, git

[pmrowla]

Also probably related to #5478

[pmrowla]

So after investigation, the message is not actually an error or crash, it's just a logged warning from python internals. There will be a PR to cleanup how we handle this so the message gets suppressed.

It also looks like for temp dir runs we are actually end up checking whether or not the experiment is identical to HEAD (via git when pulling the experiment ref back from the executor), and just don't create the duplicate experiment in this case.

The action point for #5593 should be to make workspace runs act the same way (skip creation of the duplicate empty experiment).

[jorgeorpinel]

Sounds good!