dvc <command> fails with known numpy/win10 error; Windows 10 (20H2) for dvc version >=1.10.0;

[mzietlow]

Bug Report

I have two systems running Win10, one on 2004, the other on 20H2 and I have installed DVC via https://github.com/iterative/dvc/releases.

When executing dvc , I get a Win10 specific error message (see my output of $ dvc version). Roughly, this is caused by details in ucrtbase.dll which, in turn, lead to errors in numpy's fmod function.

I have rolled my system-wide numpy installations back to numpy==1.19.3 which alleviates the issue as described (in conjunction with other workarounds) in https://developercommunity.visualstudio.com/comments/1244066/view.html.

However, as dvc does not seem use my system-wide python interpreters, solving it for them doesn't get me off the ground.

Suggestion

Although I dislike pinning, I suggest to set numpy==1.19.3 until ucrtbase.dll is fixed.
Alternatively, users with Win10 {20H2, 2004} must either:

1. use dvc with version < 1.10.0, or
2. install via pip and pin numpy == 1.19.3.

Maybe someone else is able to come up with a better solution.

Please provide information about your setup

Output of dvc version:

$ dvc version
ERROR: unexpected error - The current Numpy installation ('C:\\Program Files (x86)\\DVC (Data Version Control)\\numpy\\__init__.pyc') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86
Traceback (most recent call last):
  File "dvc\__main__.py", line 7, in <module>
  File "dvc\main.py", line 119, in main
  File "dvc\info.py", line 33, in get_dvc_info
  File "dvc\info.py", line 126, in _get_supported_remotes
  File "dvc\tree\base.py", line 104, in get_missing_deps
  File "importlib\__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\pyarrow\__init__.py", line 63, in <module>
  File "pyarrow\lib.pyx", line 24, in init pyarrow.lib
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\numpy\__init__.py", line 305, in <module>
  File "site-packages\numpy\__init__.py", line 302, in _win_os_check
RuntimeError: The current Numpy installation ('C:\\Program Files (x86)\\DVC (Data Version Control)\\numpy\\__init__.pyc') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86
[11548] Failed to execute script __main__

$ dvc -V
1.10.2

While with version < 1.10.0

$ dvc version
DVC version: 1.9.1 (exe)
---------------------------------
Platform: Python 3.7.9 on Windows-10-10.0.20262-SP0
Supports: All remotes
Cache types: hardlink
Cache directory: NTFS on M:\
Workspace directory: NTFS on M:\
Repo: dvc, git

Additional Information (if any):

$ dvc version --verbose
2020-12-02 11:56:19,891 ERROR: unexpected error - The current Numpy installation ('C:\\Program Files (x86)\\DVC (Data Version Control)\\numpy\\__init__.pyc') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc\main.py", line 90, in main
  File "dvc\command\version.py", line 12, in run
  File "dvc\info.py", line 33, in get_dvc_info
  File "dvc\info.py", line 126, in _get_supported_remotes
  File "dvc\tree\base.py", line 104, in get_missing_deps
  File "importlib\__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\pyarrow\__init__.py", line 63, in <module>
  File "pyarrow\lib.pyx", line 24, in init pyarrow.lib
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\numpy\__init__.py", line 305, in <module>
  File "site-packages\numpy\__init__.py", line 302, in _win_os_check
RuntimeError: The current Numpy installation ('C:\\Program Files (x86)\\DVC (Data Version Control)\\numpy\\__init__.pyc') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc\__main__.py", line 7, in <module>
  File "dvc\main.py", line 119, in main
  File "dvc\info.py", line 33, in get_dvc_info
  File "dvc\info.py", line 126, in _get_supported_remotes
  File "dvc\tree\base.py", line 104, in get_missing_deps
  File "importlib\__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\pyarrow\__init__.py", line 63, in <module>
  File "pyarrow\lib.pyx", line 24, in init pyarrow.lib
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\numpy\__init__.py", line 305, in <module>
  File "site-packages\numpy\__init__.py", line 302, in _win_os_check
RuntimeError: The current Numpy installation ('C:\\Program Files (x86)\\DVC (Data Version Control)\\numpy\\__init__.pyc') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86
[9648] Failed to execute script __main__

[skshetry]

@mzietlow, it seems that there's not much we can do as it's a Windows issue. I am not sure if pyarrow would be happy with 1.19.3. Let me try :)

[skshetry]

@mzietlow, can you use pip packages? We only need numpy for hdfs support, only because it comes with the pyarrow. Also, it seems conda packages will work fine.

[mzietlow]

@skshetry I am able to use DVC via my local python3.9 installation as a workaround.

[mzietlow]

Thank you!

[skshetry]

@mzietlow, was it fixed? If not, let's keep this open. Another user also experienced the same issue: https://discord.com/channels/485586884165107732/485596304961962003/785851699856277524

[mzietlow]

I assumed it to be fixed with the next release, but sure let's keep it open 😇

[efiop]

Another user (from mailing list) ran into it. Makes me wonder about #2760 again.

[efiop]

Thanks to @skshetry for implementing a workaround! 🙏 We are releasing 1.11.3 with a fix right now.

[mzietlow]

I have kept an eye on the issue and it seems to be fixed in both the newest Numpy (1.19.5) and Windows (21H1) releases. (Have not had time to verify though.)