Unable to run 'pipx list' on Termux (Android) #229

z0rc · 2019-09-28T15:13:55Z

Running pipx list on Termux results in following stack trace:

venvs are in /data/data/com.termux/files/home/.local/pip
x/venvs
apps are exposed on your $PATH at /data/data/com.termux/
files/home/.local/bin
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multipr
ocessing' (/data/data/com.termux/files/usr/lib/python3.7
/lib-dynload/_multiprocessing.cpython-37m.so)

During handling of the above exception, another exceptio
n occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/home/.local/bin/pipx
", line 10, in <module>
    sys.exit(cli())
  File "/data/data/com.termux/files/home/.local/pipx/ven
vs/pipx/lib/python3.7/site-packages/pipx/main.py", line 
547, in cli
    exit(run_pipx_command(parsed_pipx_args))
  File "/data/data/com.termux/files/home/.local/pipx/ven
vs/pipx/lib/python3.7/site-packages/pipx/main.py", line 
198, in run_pipx_command
    commands.list_packages(venv_container)
  File "/data/data/com.termux/files/home/.local/pipx/ven
vs/pipx/lib/python3.7/site-packages/pipx/commands.py", l
ine 638, in list_packages
    with multiprocessing.Pool() as p:
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/pool.py", line 158, in __init__
    self._setup_queues()
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/pool.py", line 251, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/context.py", line 112, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/queues.py", line 332, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/context.py", line 66, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/synchronize.py", line 32, in <module>
    " synchronization primitives needed will not" +
ImportError: This platform lacks a functioning sem_open 
implementation, therefore, the required synchronization 
primitives needed will not function, see issue 3770.

This is on OnePlus 5T with Android 9.0.

Technically it's a Python and/or Android issue, where multiprocessing module isn't working because Android kernel is lacking in semaphores syscalls. Also see https://bugs.python.org/issue3770

AFAIK there is still a way to workaround this, but it would require switching multiprocessing to threading module. For example, this is how black dealt with similar issue: psf/black#533

The text was updated successfully, but these errors were encountered:

cs01 · 2019-10-21T01:21:50Z

Thanks for the suggestion on the workaround. I am guessing that is the way to go. Pull requests welcome if you want to give it a shot.

Platforms like Android Termux do not have support for semaphores and thus cannot use `multiprocessing.Pool` Use the multiprocessing.dummy package as a fallback, which provides the Pool API backed by a ThreadPool. Fixes #229

pschmitt · 2020-05-17T12:39:28Z

Sorry for necrobumping but I still encounter the error on Termux with pipx 0.15.1.3.
Did I miss something obvious?

cs01 · 2020-05-17T22:32:59Z

This is the PR that closed it is #277. The diff is small, maybe you could take a look and see if it looks like it should work on your platform. Are you still getting the same error?

pschmitt · 2020-05-18T05:16:43Z

On a fully updated version of Termux you get the following:

Click to expand! (Spoiler: it's the same error as reported by OA)

pipx --version; pipx list                                                                                                                                                   
0.15.1.3
venvs are in /data/data/com.termux/files/home/.local/pipx/venvs
apps are exposed on your $PATH at /data/data/com.termux/files/home/.local/bin
<class 'function'>
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.8/lib-dynload/_multiprocessing.cpython-38.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/home/.local/bin/pipx", line 8, in <module>
    sys.exit(cli())
  File "/data/data/com.termux/files/home/.local/lib/python3.8/site-packages/pipx/main.py", line 590, in cli
    return run_pipx_command(parsed_pipx_args)
  File "/data/data/com.termux/files/home/.local/lib/python3.8/site-packages/pipx/main.py", line 195, in run_pipx_command
    return commands.list_packages(venv_container)
  File "/data/data/com.termux/files/home/.local/lib/python3.8/site-packages/pipx/commands/list_packages.py", line 35, in list_packages
    with Pool() as p:
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/dummy/__init__.py", line 124, in Pool
    return ThreadPool(processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

I dug a little further and it turns out that multiprocessing.dummy.Pool is not usable on Termux;

$ python3 -c "from multiprocessing.dummy import Pool; Pool()"                                                                                                                  
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.8/lib-dynload/_multiprocessing.cpython-38.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/dummy/__init__.py", line 124, in Pool
    return ThreadPool(processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

cs01 · 2020-05-18T05:30:59Z

Thanks. Let me know if this branch fixes it
https://github.com/pipxproject/pipx/tree/cs01/multiprocessing

pschmitt · 2020-05-18T05:31:26Z

~~That won't work since both multiprocessing.Pool and multiprocessing.dummy.Pool are unavailable.~~

pschmitt · 2020-05-18T05:33:01Z

I commented too early. Hang on.

pschmitt · 2020-05-18T05:38:47Z

Jup. That works @cs01 ! Thanks.

pschmitt · 2020-05-18T06:02:36Z

$ pip install -U pipx
$ pipx list
venvs are in /data/data/com.termux/files/home/.local/pipx/venvs
apps are exposed on your $PATH at /data/data/com.termux/files/home/.local/bin
   package flake8 3.8.1, Python 3.8.3
    - flake8

Thanks, @cs01 !

cs01 · 2020-05-18T07:03:42Z

Glad it’s working. Thanks for following back up so we could get this fixed.

cs01 added help wanted Extra attention is needed needs review from maintainer labels Oct 3, 2019

cs01 mentioned this issue Oct 21, 2019

Implementation of stored pipx metadata #222

Merged

justsh mentioned this issue Nov 15, 2019

fallback to threads if no semaphore support #277

Merged

gaborbernat closed this as completed in #277 Nov 29, 2019

cs01 reopened this May 17, 2020

cs01 mentioned this issue May 18, 2020

do not use multiprocessing.Pool if unavailable #410

Merged

cs01 closed this as completed in #410 May 18, 2020

purarue mentioned this issue Oct 12, 2023

pipx list prints psutil error on android #1084

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to run 'pipx list' on Termux (Android) #229

Unable to run 'pipx list' on Termux (Android) #229

z0rc commented Sep 28, 2019

cs01 commented Oct 21, 2019

pschmitt commented May 17, 2020

cs01 commented May 17, 2020 •

edited

Loading

pschmitt commented May 18, 2020 •

edited

Loading

cs01 commented May 18, 2020

pschmitt commented May 18, 2020 •

edited

Loading

pschmitt commented May 18, 2020

pschmitt commented May 18, 2020

pschmitt commented May 18, 2020

cs01 commented May 18, 2020

Unable to run 'pipx list' on Termux (Android) #229

Unable to run 'pipx list' on Termux (Android) #229

Comments

z0rc commented Sep 28, 2019

cs01 commented Oct 21, 2019

pschmitt commented May 17, 2020

cs01 commented May 17, 2020 • edited Loading

pschmitt commented May 18, 2020 • edited Loading

cs01 commented May 18, 2020

pschmitt commented May 18, 2020 • edited Loading

pschmitt commented May 18, 2020

pschmitt commented May 18, 2020

pschmitt commented May 18, 2020

cs01 commented May 18, 2020

cs01 commented May 17, 2020 •

edited

Loading

pschmitt commented May 18, 2020 •

edited

Loading

pschmitt commented May 18, 2020 •

edited

Loading