Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelism example does not work #100

Open
yuchiuchao opened this issue Feb 23, 2020 · 12 comments
Open

Parallelism example does not work #100

yuchiuchao opened this issue Feb 23, 2020 · 12 comments

Comments

@yuchiuchao
Copy link

Hello, I am trying to reproduce the results in the example notebook
qutip-notebooks/examples/temporal-photon-scattering.ipynb at this location
https://github.com/qutip/qutip-notebooks/blob/master/examples/temporal-photon-scattering.ipynb

I could reproduce all results with no problem, except when I got into the parallelized part. The notebook says BUSY, but there is basically no CPU consumed by the notebook and it stays there forever. Could you suggest some troubleshooting ideas for m to further debug this problem. There is no error message produced. Everything looks quiet, except the BUSY indicator and 0 CPU consumption. I checked the worker_count value, which is 7, and I have 8 CPU's. Query for pool returned
<multiprocessing.pool.Pool at 0x11cd8f10>

This is the version table:

Software Version
QuTiP 4.4.1
Numpy 1.17.1
SciPy 1.3.1
matplotlib 3.1.1
Cython 0.29.13
Number of CPUs 8
BLAS Info OPENBLAS
IPython 7.7.0
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)]
OS nt [win32]

Thanks!

Yu Chao
Fermi National Accelerator Laboratory
@nathanshammah
Copy link
Member

@fminga can you help with this?

@yuchiuchao
Copy link
Author

I ran it in the Jupyter browser mode and captured the following error traceback lines when the parallelized part was executed. Hope this provides a clue:

[I 16:03:14.918 NotebookApp] Kernel started: 2ee5d5e0-49db-4738-9f39-01ece96ba77e
[I 16:03:16.781 NotebookApp] Adapting to protocol v5.1 for kernel 2ee5d5e0-49db-4738-9f39-01ece96ba77e
Process SpawnPoolWorker-1:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-2:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-6:
Process SpawnPoolWorker-4:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-5:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-3:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-7:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-8:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-9:
Process SpawnPoolWorker-10:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
Process SpawnPoolWorker-11:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-12:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-13:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-14:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-15:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-16:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-17:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-20:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
Process SpawnPoolWorker-19:
Process SpawnPoolWorker-18:
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-21:
Traceback (most recent call last):
Process SpawnPoolWorker-22:
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
Traceback (most recent call last):
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-23:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-24:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>
Process SpawnPoolWorker-25:
Traceback (most recent call last):
File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>

@nathanshammah
Copy link
Member

nathanshammah commented Feb 27, 2020 •
edited
Loading

Dear Yu,

Regarding this and especially #101: the comment in that issue has a tone that I do not like. Here we all try to help each other. I am glad that issue is closed, but let me address both issues here, as anyone subscribed to the repository got a notification and I do not want it to lead to a toxic environment for the community, as it clearly puts off maintainers and contributors.

tone

First, to the way issues are raised and answered in this community:
QuTiP is maintained by a team of volunteers. QuTiP is developed and maintained among all the things the contributors do, as this is generally on top of research or work duties of early-career researchers, mostly students and post docs. 

It is a community project and a patch of many people’s contributions to the code base. Some eventually may not work on all users configurations. However, all notebooks are thoroughly tested, as well as the main qutip codebase, with excellent best practices such as independent code review, PEP8, continuous integration tools, automatic syntax readability checks, and with unit tests that cover more than 70% of the lines of code.

When the open-source ecosystem updates or the rest of the code base is changed, it may happen that some things do not work out for all users configurations. However, I think it is unfair to say that anyone who contributes here would "waste" someone's time, as per #101. There are over 60 tutorials and lectures on the website, and most run fine, the others generally with minor bugs. Any help in fixing them is welcome.

Let me also mention that parallel instances are pretty advanced features, so they do not involve beginners expectations.

parallelism

Now, to the code problems:
I am sorry that you could not run the notebooks and I hope someone can look into helping solve the problems. If you ask politely, it is more likely that someone will look into this.

I tried to run the temporal-photon-scattering.ipynb and the parallel-JC-steadystate.ipynb notebooks (I expect that is the second notebook you cite) and both ran fine. The first one took a long time in the last cell, but this can be amended by decreasing the lists length. It is using all of the CPU cores on my machine. I suspect something is off with your environment configuration, as many users make extensive use of qutip's parfor, although some may be sending jobs in the form of python scripts. Printing qutip.about() can help in understanding more, as it provides some additional information to the table. Mine is:

QuTiP Version:      4.5.0
Numpy Version:      1.17.1
Scipy Version:      1.2.1
Cython Version:     0.29.8
Matplotlib Version: 3.1.2
Python Version:     3.7.3
Number of CPUs:     2
BLAS Info:          OPENBLAS
OPENMP Installed:   False
INTEL MKL Ext:      False
Platform Info:      Darwin (x86_64)
Installation path:  /miniconda3/lib/python3.7/site-packages/qutip

@nwlambert
Copy link
Member

The error in Jupyter notebook is a problem with multiprocessing in notebooks on windows. I had this problem before, and I think the workaround is to put the function scattering_probability_multiprocess() in an external .py file, and import it in the jupyter notebook.

I don't know if this will fix the original problem.

@yuchiuchao
Copy link
Author

yuchiuchao commented Feb 27, 2020 via email

@yuchiuchao
Copy link
Author

yuchiuchao commented Feb 27, 2020 via email

@nwlambert
Copy link
Member

I do not know the origin, but googling suggests that it is a lack of compatibility between multiprocessing and notebooks in windows.

Your options are to not use notebooks, refactor the code using the above workaround, or try using python under Windows Subsystems for Linux (though this requires a bit of work of installing WSL, ubuntu, python etc).

for the workaround, simply pasting this into an external file:

from qutip import scattering_probability, destroy

def Htls_rft(gamma, pulseLength, pulseArea):
    RabiFreq = pulseArea / (2*pulseLength)
    sm = destroy(2)
    return [[sm.dag() + sm, lambda t, args: RabiFreq * (t < pulseLength)]]

def scattering_probability_multiprocess(pulse_area, n, psi0, sm, tlist, gamma, pulse_length):
    # Helper function to allow pool.map parallelism
    return scattering_probability(Htls_rft(gamma, pulse_length, pulse_area), psi0, n, [sm], tlist)

Importing scattering_probability_multiprocess from the external file, and then calling this works


for n in emission_nums:
    args = [(pulse_area, n, psi0, sm, tlist, gamma, pulse_length) for pulse_area in pulse_areas]
    
    scatter_probs = pool.starmap(scattering_probability_multiprocess, args)
    plt.plot(pulse_areas / np.pi, scatter_probs, label = "$P_{}$".format(n))

@yuchiuchao
Copy link
Author

yuchiuchao commented Feb 27, 2020 via email

@nwlambert nwlambert reopened this Feb 27, 2020
@yuchiuchao
Copy link
Author

yuchiuchao commented Feb 27, 2020 via email

@goerz
Copy link
Member

goerz commented Feb 27, 2020

I've run into this problem as well. The root cause is a rather subtle difference in how subprocesses are spawned in Windows vs Linux. As of Python 3.8, the problem also affects macOS. I started to explore a potential solution in qutip/qutip#1092 but didn't have time to finish it. See the tweets linked in that PR for more details on the underlying technical issue. Ultimately, it's that top level functions in notebooks can't be pickled (while functions in scripts/modules can)

My solution was to use a third-party library, loky as a replacement for the standard-library pickle. I did the PR mostly just for testing whether this works (before getting stuck on difficulties running the tests), but I'm pretty sure it would solve the problem. It would be up for debate through whether we want to add loky as an additional dependency for qutip. If so, we could try to actually finish that PR.

@goerz
Copy link
Member

goerz commented Feb 27, 2020

Also see https://qucontrol.github.io/krotov/v1.0.0/notebooks/08_example_ensemble.html (cell 2), which is where I ran into this.

@yuchiuchao
Copy link
Author

yuchiuchao commented Feb 27, 2020 via email

This was referenced Mar 9, 2020
Closed
Closed
@goerz goerz mentioned this issue Mar 22, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@goerz @nwlambert @nathanshammah @yuchiuchao