[test] When I run run_tests.py on my Windows, I always get errors.

[0xzhang]

When I run run_tests.py on my WSL's Ubuntu, it works fine in most cases. But on Windows, I always get errors. Only works in rare cases and I already set thread_num to 1 to avoid strange problem. I'm very confused right now

D:\Lernen\taichi\source\taichi>python .\tests\run_tests.py -v -k test_f16 -a cpu -t 1
[Taichi] version 1.0.2, llvm 10.0.0, commit 20fa3206, win, python 3.8.10
[I 04/30/22 18:13:29.256 14988] [shell.py:_shell_pop_print@33] Graphical python shell detected, using wrapped sys.stdout
Namespace(arch='cpu', cov_append=None, coverage=None, cpp=True, exclusive=False, fail_fast=None, failed_first=None, files=[], keys='test_f16', marks=None, rerun=None, show_output=False, threads='1', verbose=True)
Running on Arch=cpu
Running C++ tests...
.\tests\run_tests.py:26: UserWarning: C++ tests are skipped due to missing taichi_cpp_tests in D:\Lernen\taichi\source\taichi\tests\../build.Try building taichi with `TAICHI_CMAKE_ARGS='-DTI_BUILD_TESTS:BOOL=ON' python setup.py develop`if you want to enable it.
  warnings.warn(

Running Python tests...

Starting 1 testing thread(s)...
============================================================= test session starts ==============================================================
platform win32 -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- D:\APPs\Python\Python38\python.exe
cachedir: .pytest_cache
rootdir: D:\Lernen\taichi\source\taichi, configfile: pyproject.toml
collected 2842 items / 2802 deselected / 40 selected

tests/python/test_f16.py::test_snode_read_write[arch=x64-0] Windows fatal exception: access violation

Current thread 0x00003a8c (most recent call first):
  File "D:\APPs\Python\Python38\lib\site-packages\taichi-1.0.2-py3.8-win-amd64.egg\taichi\_snode\fields_builder.py", line 147 in _finalize
  File "D:\APPs\Python\Python38\lib\site-packages\taichi-1.0.2-py3.8-win-amd64.egg\taichi\_snode\fields_builder.py", line 135 in finalize
  File "D:\APPs\Python\Python38\lib\site-packages\taichi-1.0.2-py3.8-win-amd64.egg\taichi\lang\impl.py", line 277 in materialize_root_fb
  File "D:\APPs\Python\Python38\lib\site-packages\taichi-1.0.2-py3.8-win-amd64.egg\taichi\lang\impl.py", line 323 in materialize
  File "D:\APPs\Python\Python38\lib\site-packages\taichi-1.0.2-py3.8-win-amd64.egg\taichi\lang\field.py", line 241 in _initialize_host_accessors
  File "D:\APPs\Python\Python38\lib\site-packages\taichi-1.0.2-py3.8-win-amd64.egg\taichi\lang\field.py", line 326 in __setitem__
  File "D:\APPs\Python\Python38\lib\site-packages\taichi-1.0.2-py3.8-win-amd64.egg\taichi\lang\util.py", line 297 in wrapped
  File "D:\Lernen\taichi\source\taichi\tests\python\test_f16.py", line 17 in test_snode_read_write
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\python.py", line 183 in pytest_pyfunc_call
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\python.py", line 1641 in runtest
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\runner.py", line 162 in pytest_runtest_call
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\runner.py", line 255 in <lambda>
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\runner.py", line 311 in from_call
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\runner.py", line 254 in call_runtest_hook
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\runner.py", line 215 in call_and_report
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\runner.py", line 126 in runtestprotocol
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\runner.py", line 109 in pytest_runtest_protocol
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\main.py", line 348 in pytest_runtestloop
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\main.py", line 323 in _main
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\main.py", line 269 in wrap_session
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "D:\APPs\Python\Python38\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "D:\APPs\Python\Python38\lib\site-packages\_pytest\config\__init__.py", line 162 in main
  File ".\tests\run_tests.py", line 100 in _test_python
  File ".\tests\run_tests.py", line 213 in test
  File ".\tests\run_tests.py", line 218 in <module>

[qiao-bo]

Hi, I can reproduce this problem Windows fatal exception: access violation. It might be flaky, but i don't know why #4886 passed the CI.

It happens on Windows tests, concrete error in log is:

2022-05-05T10:25:54.4742478Z         # create a new program:
2022-05-05T10:25:54.4742622Z         impl.get_runtime().create_program()
2022-05-05T10:25:54.4742808Z     
2022-05-05T10:25:54.4743084Z         _logging.trace('Materializing runtime...')
2022-05-05T10:25:54.4743344Z >       impl.get_runtime().prog.materialize_runtime()
2022-05-05T10:25:54.4743704Z E       RuntimeError: [taichi/system/virtual_memory.h:VirtualMemoryAllocator@37] Virtual memory allocation (1073741824 B) failed.
2022-05-05T10:25:54.4743987Z 
2022-05-05T10:25:54.4744206Z venv\lib\site-packages\taichi-1.0.2-py3.7-win-amd64.egg\taichi\lang\misc.py:457: RuntimeError

This seems relate to Out of memory during CPU virtual memory allocation. I am looking into it.
cc @ailzhang @0xzhang

[qiao-bo]

Failed and error test summary

2022-05-05T11:18:22.2540158Z =========================== short test summary info ===========================
2022-05-05T11:18:22.2540342Z FAILED tests/python/test_ndarray.py::test_default_fp_ndarray[none-dtype0] - R...
2022-05-05T11:18:22.2540532Z FAILED tests/python/test_runtime.py::test_init_arg[none-fast_math-values6] - ...
2022-05-05T11:18:22.2540700Z FAILED tests/python/test_runtime.py::test_init_arg[none-async_mode-values7]
2022-05-05T11:18:22.2540861Z FAILED tests/python/test_runtime.py::test_init_arg[none-flatten_if-values8]
2022-05-05T11:18:22.2541046Z FAILED tests/python/test_runtime.py::test_init_arg[none-simplify_before_lower_access-values9]
2022-05-05T11:18:22.2541220Z FAILED tests/python/test_runtime.py::test_init_arg[none-kernel_profiler-values11]
2022-05-05T11:18:22.2541383Z ERROR tests/python/test_ad_basics.py::test_atan2[arch=x64-1-<lambda>-<lambda>1]
2022-05-05T11:18:22.2541544Z ERROR tests/python/test_ad_for.py::test_ad_sum[arch=x64-0] - RuntimeError: [t...
2022-05-05T11:18:22.2541724Z ERROR tests/python/test_ndarray.py::test_matrix_ndarray[arch=x64-0-shape2-dtype3-1-2]
2022-05-05T11:18:22.2541899Z ERROR tests/python/test_ndarray.py::test_matrix_ndarray[arch=x64-1-shape2-dtype0-1-2]
2022-05-05T11:18:22.2542064Z ERROR tests/python/test_ndrange.py::test_tensor_based_3d[arch=x64-0] - Runtim...
2022-05-05T11:18:22.2542228Z ERROR tests/python/test_oop.py::test_classfunc[arch=x64-1] - RuntimeError: [t...
2022-05-05T11:18:22.2542389Z ERROR tests/python/test_ndrange.py::test_tensor_based_3d[arch=x64-1] - Runtim...
2022-05-05T11:18:22.2542550Z ERROR tests/python/test_oop.py::test_oop[arch=x64-0] - RuntimeError: [taichi/...
2022-05-05T11:18:22.2542711Z ERROR tests/python/test_ndrange.py::test_static_grouped[arch=x64-0] - Runtime...
2022-05-05T11:18:22.2542871Z ERROR tests/python/test_oop.py::test_oop[arch=x64-1] - RuntimeError: [taichi/...
2022-05-05T11:18:22.2543032Z ERROR tests/python/test_ndrange.py::test_static_grouped[arch=x64-1] - Runtime...
2022-05-05T11:18:22.2543209Z ERROR tests/python/test_oop.py::test_oop_two_items[arch=x64-0] - RuntimeError...
2022-05-05T11:18:22.2543370Z ERROR tests/python/test_ndrange.py::test_static_grouped_static[arch=x64-0] - ...
2022-05-05T11:18:22.2543532Z ERROR tests/python/test_oop.py::test_oop_two_items[arch=x64-1] - RuntimeError...
2022-05-05T11:18:22.2543693Z ERROR tests/python/test_ndrange.py::test_static_grouped_static[arch=x64-1] - ...
2022-05-05T11:18:22.2543852Z ERROR tests/python/test_oop.py::test_oop_inherit_ok[arch=x64-0] - RuntimeErro...
2022-05-05T11:18:22.2544012Z ERROR tests/python/test_ndrange.py::test_field_init_eye[arch=x64-0] - Runtime...
2022-05-05T11:18:22.2544175Z ERROR tests/python/test_oop.py::test_oop_inherit_ok[arch=x64-1] - RuntimeErro...
2022-05-05T11:18:22.2544335Z ERROR tests/python/test_oop.py::test_hook[arch=x64-1] - RuntimeError: [taichi...
2022-05-05T11:18:22.2544493Z ERROR tests/python/test_ndrange.py::test_nested_ndrange[arch=x64-0] - Runtime...
2022-05-05T11:18:22.2544653Z ERROR tests/python/test_oop.py::test_oop_with_portery_decorator[arch=x64-0]
2022-05-05T11:18:22.2544813Z ERROR tests/python/test_ndrange.py::test_nested_ndrange[arch=x64-1] - Runtime...
2022-05-05T11:18:22.2545015Z ERROR tests/python/test_mesh.py::test_mesh_reordered_opt[arch=x64-1] - Runtim...
2022-05-05T11:18:22.2545183Z ERROR tests/python/test_simple_matrix_slice.py::test_slice[arch=x64-0] - Runt...
2022-05-05T11:18:22.2545345Z ERROR tests/python/test_oop.py::test_oop_with_portery_decorator[arch=x64-1]
2022-05-05T11:18:22.2545506Z ERROR tests/python/test_ndrange.py::test_ndrange_ast_transform[arch=x64-0] - ...
2022-05-05T11:18:22.2545728Z ERROR tests/python/test_simple_matrix_slice.py::test_slice[arch=x64-1] - Runt...
2022-05-05T11:18:22.2545893Z ERROR tests/python/test_simple_matrix_slice.py::test_dyn[arch=x64] - RuntimeE...
2022-05-05T11:18:22.2546050Z ERROR tests/python/test_oop.py::test_oop_with_static_decorator[arch=x64-0] - ...
2022-05-05T11:18:22.2546208Z ERROR tests/python/test_simple_matrix_slice.py::test_no_dyn[arch=x64] - Runti...
2022-05-05T11:18:22.2546369Z ERROR tests/python/test_ndrange.py::test_ndrange_ast_transform[arch=x64-1] - ...
2022-05-05T11:18:22.2546538Z ERROR tests/python/test_snode_layout_inspection.py::test_primitives[arch=x64-0]
2022-05-05T11:18:22.2546705Z ERROR tests/python/test_snode_layout_inspection.py::test_primitives[arch=x64-1]
2022-05-05T11:18:22.2546865Z ERROR tests/python/test_oop.py::test_oop_with_static_decorator[arch=x64-1] - ...
2022-05-05T11:18:22.2547024Z ERROR tests/python/test_ndrange.py::test_grouped_ndrange_star[arch=x64-0] - R...
2022-05-05T11:18:22.2547188Z ERROR tests/python/test_snode_layout_inspection.py::test_bit_struct[arch=x64-0]
2022-05-05T11:18:22.2547353Z ERROR tests/python/test_snode_layout_inspection.py::test_bit_struct[arch=x64-1]
2022-05-05T11:18:22.2547544Z ERROR tests/python/test_optimization.py::test_advanced_store_forwarding_nested_loops[arch=x64-0]
2022-05-05T11:18:22.2547709Z ERROR tests/python/test_ndrange.py::test_grouped_ndrange_star[arch=x64-1] - R...
2022-05-05T11:18:22.2547868Z ERROR tests/python/test_sort.py::test_sort[arch=x64-0] - RuntimeError: [taich...
2022-05-05T11:18:22.2548028Z ERROR tests/python/test_sort.py::test_sort[arch=x64-1] - RuntimeError: [taich...
2022-05-05T11:18:22.2548216Z ERROR tests/python/test_optimization.py::test_advanced_store_forwarding_nested_loops[arch=x64-1]
2022-05-05T11:18:22.2548394Z ERROR tests/python/test_sparse_activate.py::test_pointer[arch=x64-0] - Runtim...
2022-05-05T11:18:22.2548560Z ERROR tests/python/test_ndrange.py::test_ndrange_three_arguments[arch=x64-0]
2022-05-05T11:18:22.2548724Z ERROR tests/python/test_sparse_activate.py::test_pointer[arch=x64-1] - Runtim...
2022-05-05T11:18:22.2548886Z ERROR tests/python/test_sparse_activate.py::test_non_dfs_snode_order[arch=x64-0]
2022-05-05T11:18:22.2549072Z ERROR tests/python/test_optimization.py::test_advanced_unused_store_elimination_if[arch=x64-0]
2022-05-05T11:18:22.2549234Z ERROR tests/python/test_ndrange.py::test_ndrange_three_arguments[arch=x64-1]
2022-05-05T11:18:22.2549395Z ERROR tests/python/test_sparse_activate.py::test_non_dfs_snode_order[arch=x64-1]
2022-05-05T11:18:22.2549580Z ERROR tests/python/test_optimization.py::test_advanced_unused_store_elimination_if[arch=x64-1]
2022-05-05T11:18:22.2549745Z ERROR tests/python/test_sparse_basics.py::test_pointer[arch=x64-0] - RuntimeE...
2022-05-05T11:18:22.2549916Z ERROR tests/python/test_ndrange.py::test_ndrange_start_greater_than_end[arch=x64-0]
2022-05-05T11:18:22.2550077Z ERROR tests/python/test_sparse_basics.py::test_pointer[arch=x64-1] - RuntimeE...
2022-05-05T11:18:22.2550241Z ERROR tests/python/test_sparse_basics.py::test_pointer_is_active[arch=x64-0]
2022-05-05T11:18:22.2550420Z ERROR tests/python/test_optimization.py::test_local_store_in_nested_for_and_if[arch=x64-0]
2022-05-05T11:18:22.2550590Z ERROR tests/python/test_ndrange.py::test_ndrange_start_greater_than_end[arch=x64-1]
2022-05-05T11:18:22.2550752Z ERROR tests/python/test_sparse_basics.py::test_pointer_is_active[arch=x64-1]
2022-05-05T11:18:22.2550951Z ERROR tests/python/test_optimization.py::test_advanced_store_elimination_in_loop[arch=x64-0]
2022-05-05T11:18:22.2551157Z ERROR tests/python/test_new_allocator.py::test_3d[arch=x64-0] - RuntimeError:...
2022-05-05T11:18:22.2551348Z ERROR tests/python/test_optimization.py::test_advanced_store_elimination_in_loop[arch=x64-1]
2022-05-05T11:18:22.2551513Z ERROR tests/python/test_sparse_basics.py::test_pointer2[arch=x64-0] - Runtime...
2022-05-05T11:18:22.2551675Z ERROR tests/python/test_new_allocator.py::test_3d[arch=x64-1] - RuntimeError:...
2022-05-05T11:18:22.2551903Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_subtraction[arch=x64-0-dtype1]
2022-05-05T11:18:22.2552083Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_subtraction[arch=x64-1-dtype0]
2022-05-05T11:18:22.2552249Z ERROR tests/python/test_optimization.py::test_parallel_assignment[arch=x64-0]
2022-05-05T11:18:22.2552411Z ERROR tests/python/test_new_allocator.py::test_matrix[arch=x64-0] - RuntimeEr...
2022-05-05T11:18:22.2552589Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_subtraction[arch=x64-1-dtype1]
2022-05-05T11:18:22.2552787Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_scalar_multiplication[arch=x64-0-dtype0]
2022-05-05T11:18:22.2552952Z ERROR tests/python/test_optimization.py::test_parallel_assignment[arch=x64-1]
2022-05-05T11:18:22.2553146Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_scalar_multiplication[arch=x64-0-dtype1]
2022-05-05T11:18:22.2553308Z ERROR tests/python/test_new_allocator.py::test_matrix[arch=x64-1] - RuntimeEr...
2022-05-05T11:18:22.2553522Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_scalar_multiplication[arch=x64-1-dtype0]
2022-05-05T11:18:22.2553716Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_scalar_multiplication[arch=x64-1-dtype1]
2022-05-05T11:18:22.2553879Z ERROR tests/python/test_packed_size.py::test_packed_size[arch=x64-0] - Runtim...
2022-05-05T11:18:22.2554039Z ERROR tests/python/test_new_allocator.py::test_alloc_in_kernel[arch=x64-0] - ...
2022-05-05T11:18:22.2554218Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_transpose[arch=x64-0-dtype0]
2022-05-05T11:18:22.2554397Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_transpose[arch=x64-0-dtype1]
2022-05-05T11:18:22.2554560Z ERROR tests/python/test_packed_size.py::test_packed_size[arch=x64-1] - Runtim...
2022-05-05T11:18:22.2554734Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_transpose[arch=x64-1-dtype0]
2022-05-05T11:18:22.2554897Z ERROR tests/python/test_new_allocator.py::test_alloc_in_kernel[arch=x64-1] - ...
2022-05-05T11:18:22.2555075Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_transpose[arch=x64-1-dtype1]
2022-05-05T11:18:22.2555277Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_elementwise_multiplication[arch=x64-0-dtype0]
2022-05-05T11:18:22.2555448Z ERROR tests/python/test_parallel_range_for.py::test_parallel_range_for[arch=x64-0]
2022-05-05T11:18:22.2555651Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_elementwise_multiplication[arch=x64-0-dtype1]
2022-05-05T11:18:22.2555854Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_elementwise_multiplication[arch=x64-1-dtype0]
2022-05-05T11:18:22.2556023Z ERROR tests/python/test_parallel_range_for.py::test_parallel_range_for[arch=x64-1]
2022-05-05T11:18:22.2556239Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_elementwise_multiplication[arch=x64-1-dtype1]
2022-05-05T11:18:22.2556424Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_multiplication[arch=x64-0-dtype0]
2022-05-05T11:18:22.2556608Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_multiplication[arch=x64-0-dtype1]
2022-05-05T11:18:22.2556772Z ERROR tests/python/test_parallel_range_for.py::test_serial_for[arch=x64-0] - ...
2022-05-05T11:18:22.2556954Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_multiplication[arch=x64-1-dtype0]
2022-05-05T11:18:22.2557136Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_multiplication[arch=x64-1-dtype1]
2022-05-05T11:18:22.2557339Z ERROR tests/python/test_parallel_range_for.py::test_serial_for[arch=x64-1] - ...
2022-05-05T11:18:22.2557554Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_nonsymmetric_multiplication[arch=x64-0-dtype0]
2022-05-05T11:18:22.2557756Z ERROR tests/python/test_sparse_matrix.py::test_sparse_matrix_nonsymmetric_multiplication[arch=x64-0-dtype1]
2022-05-05T11:18:22.2557856Z = 6 failed, 2618 passed, 79 skipped, 66 warnings, 95 errors, 282 rerun in 257.19s (0:04:17) =
2022-05-05T11:18:22.2565223Z ##[error]Process completed with exit code 1.

[0xzhang]

@qiao-bo I guess one possibility is that the Paddle's develop package on GPU is too large. Therefore it is likely to affect other tests after import paddle. I also encounter some strange problems when I test locally.

[0xzhang]

In addition to this problem, another problem is that it may cause CI to get stuck at CleanUp process.

[qiao-bo]

@0xzhang Thanks for the input. I am playing with some test settings at #4910. Lets see if relaxing Taichi's memory limitation helps.

In addition to this problem, another problem is that it causes CI to get stuck at CleanUp process.

This is also mysteries to me at the moment. I do notice that the CI clean-up is stuck is because some python packages cannot be deleted on windows. (probably installed by Paddle ;/).

[qiao-bo]

This should be fixed by #4910. Closed.