Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement dask client. #120

Merged
merged 9 commits into from
Jul 25, 2021
Merged

Implement dask client. #120

merged 9 commits into from
Jul 25, 2021

Conversation

josuemtzmo
Copy link
Contributor

Improvements:

  • Support for dask client.
  • Add useful links in user preferences.

Within the node tree, the dask dashboard is displayed.

Screen Shot 2021-07-24 at 6 56 38 pm

The user preference allows the user to initiate blenderNC using dask.
Screen Shot 2021-07-24 at 6 56 59 pm

@josuemtzmo josuemtzmo linked an issue Jul 24, 2021 that may be closed by this pull request
2 tasks
@josuemtzmo josuemtzmo added this to the 0.4.1 milestone Jul 24, 2021
@josuemtzmo josuemtzmo requested a review from whatnick July 24, 2021 09:19
@josuemtzmo josuemtzmo self-assigned this Jul 24, 2021
@josuemtzmo josuemtzmo added enhancement New feature or request ESOWC - 2021 ESOWC Challenge Testing Improve test labels Jul 24, 2021
@josuemtzmo
Copy link
Contributor Author

Coverage will drop, as many of the implementations require changes in the user interface. However, changing the preference blendernc_use_dask is now tested.

Copy link
Collaborator

@whatnick whatnick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dask worker temp files are checked in..please add these to .gitignore and git rm here.

@josuemtzmo
Copy link
Contributor Author

Thanks, I completely forgot to remove them!

@josuemtzmo
Copy link
Contributor Author

I removed the testing of dask, as currently running dask in background mode will generate the following issue, as a consequence of multiprocessing modifying the sys.path.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/bin/2.92/python/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/bin/2.92/python/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
    prepare(preparation_data)
  File "/bin/2.92/python/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/bin/2.92/python/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "/bin/2.92/python/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/bin/2.92/python/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/bin/2.92/python/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/blendernc/tests/blendernc_nodetree_settings.test1.py", line 6, in <module>
    import bpy
  File "/bin/2.92/scripts/modules/bpy/__init__.py", line 38, in <module>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from _bpy import (
ModuleNotFoundError: No module named '_bpy'

I will look more into this, but I believe there will be no way to test this soon. A really interesting discussion on this issue can be found at:
TylerGubala/blenderpy#23

… multiple processes crashes with: '+[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.'.
@josuemtzmo
Copy link
Contributor Author

I figured that if the dask.Distributed.Client is initiated with the processes=False flag, then the issues disappear. This also solves BlenderNC crashing within the UI when loading a dataset and creating forks of the child processes.

@codecov
Copy link

codecov bot commented Jul 24, 2021

Codecov Report

Merging #120 (6188bf7) into master (fb11443) will decrease coverage by 0.39%.
The diff coverage is 53.96%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #120      +/-   ##
==========================================
- Coverage   72.33%   71.94%   -0.40%     
==========================================
  Files          39       39              
  Lines        2606     2666      +60     
  Branches      312      317       +5     
==========================================
+ Hits         1885     1918      +33     
- Misses        634      660      +26     
- Partials       87       88       +1     
Flag Coverage Δ
unittests 71.94% <53.96%> (-0.40%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
blendernc/blendernc.py 100.00% <ø> (ø)
blendernc/panels.py 50.75% <37.50%> (-2.95%) ⬇️
blendernc/preferences.py 74.50% <61.11%> (-5.21%) ⬇️
blendernc/messages.py 73.58% <100.00%> (+1.58%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb11443...6188bf7. Read the comment docs.

@josuemtzmo josuemtzmo merged commit c7ccfe6 into master Jul 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request ESOWC - 2021 ESOWC Challenge Testing Improve test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement dask
2 participants