Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support multiprocessing without importing gmt in each process #217

Open
norfordb opened this issue Jul 25, 2018 · 4 comments · May be fixed by #3392
Open

Support multiprocessing without importing gmt in each process #217

norfordb opened this issue Jul 25, 2018 · 4 comments · May be fixed by #3392
Labels
feature request New feature wanted
Milestone
1.0.0

Comments

@norfordb
Copy link

norfordb commented Jul 25, 2018 •
edited by leouieda
Loading

Feedback or description of feature requested

When attempting to call a script using gmt-python by way of the python multiprocessing library (multiprocessing.pool), an error is returned, as shown below.

Are you willing to help implement and maintain this feature? No

Full code that generated the error

import gmt
import multiprocessing as mp

def gmt_fun(n): #a very simple function that uses gmt-python
	fig = gmt.Figure()
	fig.coast(R="-20/20/-20/20", J="M10i", G='gray', S="lightblue", B=True) 
	fig.savefig("{}.pdf".format(n), show=False)

#gmt_fun(5) #calls the function without multiprocessing - uncomment as desired to see that the function works on its own

with mp.Pool(2) as p: #calls the process with multiprocessing, using 2 cores
	a = p.map(gmt_fun,[x for x in range(0,2)])

The code does work with the following change:

import multiprocessing as mp

def gmt_fun(n): #a very simple function that uses gmt-python
	import gmt
	fig = gmt.Figure()
	fig.coast(R="-20/20/-20/20", J="M10i", G='gray', S="lightblue", B=True) 
	fig.savefig("{}.pdf".format(n), show=False)

#gmt_fun(5) #calls the function without multiprocessing - uncomment as desired to see that the function works on its own

with mp.Pool(2) as p: #calls the process with multiprocessing, using 2 cores
	a = p.map(gmt_fun,[x for x in range(0,2)])

Full error message

gmt-python-session [ERROR]: Not available in classic mode
gmt-python-session [ERROR]: Not available in classic mode
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/norfordb/anaconda2/envs/gmt-python/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/Users/norfordb/anaconda2/envs/gmt-python/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "tweaker.py", line 6, in gmt_fun
    fig = gmt.Figure()
  File "/Users/norfordb/anaconda2/envs/gmt-python/lib/python3.6/site-packages/gmt/figure.py", line 64, in __init__
    self._activate_figure()
  File "/Users/norfordb/anaconda2/envs/gmt-python/lib/python3.6/site-packages/gmt/figure.py", line 85, in _activate_figure
    lib.call_module("figure", "{} {}".format(self._name, fmt))
  File "/Users/norfordb/anaconda2/envs/gmt-python/lib/python3.6/site-packages/gmt/clib/session.py", line 481, in call_module
    module, status, self._error_message
gmt.exceptions.GMTCLibError: Module 'figure' failed with status code 30:
gmt-python-session [ERROR]: Not available in classic mode
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "tweaker.py", line 13, in <module>
    a = p.map(gmt_fun,[x for x in range(0,2)])
  File "/Users/norfordb/anaconda2/envs/gmt-python/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/norfordb/anaconda2/envs/gmt-python/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
gmt.exceptions.GMTCLibError: Module 'figure' failed with status code 30:
gmt-python-session [ERROR]: Not available in classic mode

System information

  • Operating system: OSX
  • Python installation (Anaconda, system, ETS): Anaconda
  • Version of GMT: 6.0.0a16
  • Version of Python: 3.6.5
  • Version of this package: 0.1a3+112.gefabb47
  • If using conda, paste the output of conda list below:
# Output of conda list
appnope                   0.1.0                    py36_0    conda-forge
atomicwrites              1.1.5                    py36_0    conda-forge
attrs                     17.4.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
blas                      1.1                    openblas    conda-forge
bokeh                     0.13.0                   py36_0    conda-forge
boost                     1.66.0                   py36_1    conda-forge
boost-cpp                 1.66.0                        1    conda-forge
bottleneck                1.2.1            py36h7eb728f_1    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.1.18                     0    conda-forge
cairo                     1.14.10                       0    conda-forge
certifi                   2018.1.18                py36_0    conda-forge
cftime                    1.0.0                    py36_0    conda-forge
click                     6.7                        py_1    conda-forge
cloudpickle               0.5.3                      py_0    conda-forge
curl                      7.59.0                        0    conda-forge
cycler                    0.10.0                   py36_0    conda-forge
Cython                    0.28.4                    <pip>
cytoolz                   0.9.0.1          py36h470a237_0    conda-forge
dask                      0.18.2                     py_0    conda-forge
dask-core                 0.18.2                     py_0    conda-forge
dcw-gmt                   1.1.2                         0    conda-forge
decorator                 4.2.1                    py36_0    conda-forge
distributed               1.22.0                   py36_0    conda-forge
expat                     2.2.5                         0    conda-forge
fastkml                   0.11                      <pip>
fftw                      3.3.7                         0    conda-forge
fontconfig                2.12.6                        0    conda-forge
freetype                  2.8.1                         0    conda-forge
freexl                    1.0.5                         0    conda-forge
gdal                      2.2.4                    py36_0    conda-forge
geos                      3.6.2                         1    conda-forge
geotiff                   1.4.2                         1    conda-forge
gettext                   0.19.8.1                      0    conda-forge
ghostscript               9.22                          0    conda-forge
giflib                    5.1.4                         0    conda-forge
glib                      2.55.0                        0    conda-forge
gmt                       6.0.0a16        blas_openblas_200  [blas_openblas]  conda-forge/label/dev
gmt-python                0.1a3+112.gefabb47           <pip>
gshhg-gmt                 2.3.7                         0    conda-forge
h5netcdf                  0.6.1                      py_0    conda-forge
h5py                      2.8.0            py36h470a237_0    conda-forge
hdf4                      4.2.13                        0    conda-forge
hdf5                      1.10.1                        2    conda-forge
heapdict                  1.0.0                    py36_0    conda-forge
icu                       58.2                          0    conda-forge
ipython                   6.4.0                    py36_0    conda-forge
ipython_genutils          0.2.0                    py36_0    conda-forge
jedi                      0.11.1                   py36_0    conda-forge
jinja2                    2.10                       py_1    conda-forge
jpeg                      9b                            2    conda-forge
json-c                    0.12.1                        0    conda-forge
kealib                    1.4.7                         4    conda-forge
kiwisolver                1.0.1                    py36_1    conda-forge
krb5                      1.14.6                        0    conda-forge
libdap4                   3.18.3                        2    conda-forge
libffi                    3.2.1                         3    conda-forge
libgdal                   2.2.4                         5    conda-forge
libgfortran               3.0.0                         0    conda-forge
libiconv                  1.15                          0    conda-forge
libkml                    1.3.0                         6    conda-forge
libnetcdf                 4.6.1                         2    conda-forge
libpng                    1.6.34                        0    conda-forge
libpq                     9.6.3                         0    conda-forge
libspatialite             4.3.0a                       19    conda-forge
libssh2                   1.8.0                         2    conda-forge
libtiff                   4.0.9                         0    conda-forge
libxml2                   2.9.8                         0    conda-forge
locket                    0.2.0                      py_2    conda-forge
markupsafe                1.0                      py36_0    conda-forge
matplotlib                2.2.2                    py36_1    conda-forge
more-itertools            4.1.0                      py_0    conda-forge
msgpack-python            0.5.6            py36h2d50403_2    conda-forge
ncurses                   5.9                          10    conda-forge
netcdf4                   1.4.0                    py36_0    conda-forge
nose                      1.3.7                    py36_2    conda-forge
numpy                     1.14.5                    <pip>
numpy                     1.14.5          py36_blas_openblashd3ea46f_201  [blas_openblas]  conda-forge
olefile                   0.45.1                   py36_0    conda-forge
openblas                  0.2.20                        7    conda-forge
openjpeg                  2.3.0                         2    conda-forge
openssl                   1.0.2n                        0    conda-forge
packaging                 17.1                       py_0    conda-forge
pandas                    0.23.2                    <pip>
pandas                    0.23.3                   py36_0    conda-forge
parso                     0.1.1                      py_0    conda-forge
partd                     0.3.8                      py_1    conda-forge
pcre                      8.41                          1    conda-forge
pexpect                   4.4.0                    py36_0    conda-forge
pickleshare               0.7.4                    py36_0    conda-forge
pillow                    5.1.0                    py36_0    conda-forge
pip                       10.0.1                    <pip>
pip                       9.0.3                    py36_0    conda-forge
pixman                    0.34.0                        1    conda-forge
pluggy                    0.6.0                      py_0    conda-forge
poppler                   0.61.1                        3    conda-forge
poppler-data              0.4.8                         0    conda-forge
proj4                     4.9.3                         5    conda-forge
prompt_toolkit            1.0.15                   py36_0    conda-forge
psutil                    5.4.6                    py36_0    conda-forge
ptyprocess                0.5.2                    py36_0    conda-forge
py                        1.5.3                      py_0    conda-forge
pygeoif                   0.7                       <pip>
pygments                  2.2.0                    py36_0    conda-forge
pyparsing                 2.2.0                    py36_0    conda-forge
pytest                    3.6.2                    py36_0    conda-forge
pytest-mpl                0.9                        py_0    conda-forge
python                    3.6.5                         0    conda-forge
python-dateutil           2.7.2                      py_0    conda-forge
python-dateutil           2.7.3                     <pip>
pytz                      2018.3                     py_0    conda-forge
pytz                      2018.5                    <pip>
pyyaml                    3.12                     py36_1    conda-forge
readline                  7.0                           0    conda-forge
scipy                     1.0.1           py36_blas_openblas_200  [blas_openblas]  conda-forge
setuptools                39.0.1                   py36_0    conda-forge
simplegeneric             0.8.1                    py36_0    conda-forge
six                       1.11.0                   py36_1    conda-forge
sortedcontainers          2.0.4                      py_1    conda-forge
sqlite                    3.20.1                        2    conda-forge
tblib                     1.3.2                      py_1    conda-forge
tk                        8.6.7                         0    conda-forge
toolz                     0.9.0                      py_0    conda-forge
tornado                   5.0.1                    py36_1    conda-forge
traitlets                 4.3.2                    py36_0    conda-forge
wcwidth                   0.1.7                    py36_0    conda-forge
wheel                     0.31.0                   py36_0    conda-forge
xarray                    0.10.8                   py36_0    conda-forge
xarray                    0.10.7                    <pip>
xerces-c                  3.2.0                         0    conda-forge
xz                        5.2.3                         0    conda-forge
yaml                      0.1.7                         0    conda-forge
zict                      0.1.3                      py_0    conda-forge
zlib                      1.2.11                        0    conda-forge
@welcome
Copy link

welcome bot commented Jul 25, 2018

👋 Thanks for opening your first issue here! Please make sure you filled out the template with as much detail as possible. You might also want to take a look at our contributing guidelines and code of conduct.

@leouieda
Copy link
Member

@norfordb thanks! OK, I think I know what the problem is. The figure module only works inside a GMT modern mode session, which is initiated by a gmt begin command. We're running this at import time. When you pass the function to a new process, the import gmt didn't happen in that process so gmt begin was never called there.

I'll try to think of a way around this. For now, we'll your code as a test case so we know how it works.

Again, thank you for reporting this! 🥇

@leouieda leouieda changed the title Multiprocessing Support multiprocessing without importing gmt in each process Jan 16, 2019
@leouieda leouieda added the feature request New feature wanted label Jan 16, 2019
@flixha
Copy link

flixha commented Jan 5, 2021

Thanks a lot for reporting this problem - I was really scratching my head why plotting in parallel didn't work!

For the record, here is how I work around this error for now: One can manually reload pygmt at the top of the function that is run in parallel (gmt_fun in the example above), like:

from importlib import reload

def gmt_fun(n):
    import pygmt
    reload(pygmt)
    ...

@raychani
Copy link

Thanks a lot for reporting this problem - I was really scratching my head why plotting in parallel didn't work!

For the record, here is how I work around this error for now: One can manually reload pygmt at the top of the function that is run in parallel (gmt_fun in the example above), like:

from importlib import reload

def gmt_fun(n):
    import pygmt
    reload(pygmt)
    ...

Thanks for your solution.

It works on my linux(centos) machine.
But, does not works on my windows machine.
In linux, the session path use PID of child process.
But in windows, the session path use PID of parent process.

@weiji14 weiji14 mentioned this issue Apr 28, 2021
Open
@weiji14 weiji14 added this to the 1.0.0 milestone Mar 14, 2022
@weiji14 weiji14 mentioned this issue Sep 6, 2022
Open
This was referenced Dec 28, 2023
Merged
Merged
@seisman seisman mentioned this issue Apr 25, 2024
Merged
This was referenced Aug 11, 2024
Draft
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request New feature wanted
Projects
None yet
Milestone
1.0.0
Development

Successfully merging a pull request may close this issue.

POC + WIP: Support multiprocessing Version 1 GenericMappingTools/pygmt
5 participants
@leouieda @raychani @flixha @weiji14 @norfordb