Single import-dependence on pynng causes M1 Mac install error

[janosh]

maggma depends on pynng for a single import Pair1:

maggma/src/maggma/cli/distributed.py

Lines 8 to 16 in 9e037d7

	from typing import List
	from monty.json import jsanitize
	from monty.serialization import MontyDecoder
	from pynng import Pair1
	from maggma.cli.multiprocessing import multi
	from maggma.core import Builder
	from maggma.utils import tqdm

pynng causes pip install errors on M1 Macs (tracked in codypiersall/pynng#89). Also, see related issue hackingmaterials/atomate#728.

What's your take on maintaining the functionality of Pair1 as part of maggma itself? Else I would try help fix the error upstream.

[janosh]

pynng appears to be unmaintained which would be a reason to incorporate Pair1 into maggma.

[munrojm]

Hi @janosh, we appreciate you bringing this up. Maintaining Pair1 functionality within the code could be one option. Before deciding, I would like to take some time to see if there are any other workable solutions. I will keep this thread open until we have a resolution.

[janosh]

Happy to submit the PR that incorporates Pair1 into maggma if no other solutions arise.

[munrojm]

We appreciate the offer. Ideally, I think it would be good if there was another well-maintained library we could use. My plan is to dedicate some time later this week to working through some maggma issues. I will look more into the details then. Do you have a sense of how much would need to be added? There looks to be a lot of stuff in the Socket class.

[shyamd]

Another option is ZeroMQ

[munrojm]

Looks like a good option. I'll give it a shot.

[munrojm]

@janosh, I have replaced the use of Pair1 with ZeroMQ's REQ/REP client-server setup through pyzmq. Would you have any time to test out installation? I unfortunately don't have access to an M1 machine. It is in the zero_mq branch (PR #543).

[janosh]

@munrojm Fix confirmed! Installation is error-free on the zero_mq branch. Thanks a lot.

pip install logs

$ git checkout zero_mq
$ pip install .
Processing /Users/janosh/Repos/maggma
  Preparing metadata (setup.py) ... done
Requirement already satisfied: setuptools in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (59.5.0)
Requirement already satisfied: pymongo>=4.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (4.0.1)
Requirement already satisfied: monty>=1.0.2 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (2022.1.12.1)
Requirement already satisfied: mongomock>=3.10.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (3.23.0)
Requirement already satisfied: pydash>=4.1.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (5.1.0)
Requirement already satisfied: jsonschema>=3.1.1 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (4.4.0)
Requirement already satisfied: tqdm>=4.19.6 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (4.62.3)
Requirement already satisfied: mongogrant>=0.3.1 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (0.3.3)
Requirement already satisfied: aioitertools>=0.5.1 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (0.8.0)
Requirement already satisfied: numpy>=1.17.3 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (1.22.1)
Requirement already satisfied: pydantic>=0.32.2 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (1.8.2)
Requirement already satisfied: fastapi>=0.42.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (0.72.0)
Requirement already satisfied: pyzmq==22.3.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (22.3.0)
Requirement already satisfied: dnspython>=1.16.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (2.1.0)
Requirement already satisfied: sshtunnel>=0.1.5 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (0.4.0)
Requirement already satisfied: msgpack>=0.5.6 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (1.0.3)
Requirement already satisfied: orjson>=3.6.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from maggma==0.38.2.dev24+gdd58756) (3.6.5)
Requirement already satisfied: starlette==0.17.1 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from fastapi>=0.42.0->maggma==0.38.2.dev24+gdd58756) (0.17.1)
Requirement already satisfied: anyio<4,>=3.0.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from starlette==0.17.1->fastapi>=0.42.0->maggma==0.38.2.dev24+gdd58756) (3.4.0)
Requirement already satisfied: attrs>=17.4.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from jsonschema>=3.1.1->maggma==0.38.2.dev24+gdd58756) (21.3.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from jsonschema>=3.1.1->maggma==0.38.2.dev24+gdd58756) (0.18.0)
Requirement already satisfied: Click in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (8.0.3)
Requirement already satisfied: Flask>=1.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (2.0.2)
Requirement already satisfied: requests in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (2.27.1)
Requirement already satisfied: sentinels in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from mongomock>=3.10.0->maggma==0.38.2.dev24+gdd58756) (1.0.0)
Requirement already satisfied: six in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from mongomock>=3.10.0->maggma==0.38.2.dev24+gdd58756) (1.16.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from pydantic>=0.32.2->maggma==0.38.2.dev24+gdd58756) (4.0.1)
Requirement already satisfied: paramiko>=2.7.2 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from sshtunnel>=0.1.5->maggma==0.38.2.dev24+gdd58756) (2.9.1)
Requirement already satisfied: Jinja2>=3.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from Flask>=1.0->mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (3.0.3)
Requirement already satisfied: itsdangerous>=2.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from Flask>=1.0->mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (2.0.1)
Requirement already satisfied: Werkzeug>=2.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from Flask>=1.0->mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (2.0.2)
Requirement already satisfied: cryptography>=2.5 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from paramiko>=2.7.2->sshtunnel>=0.1.5->maggma==0.38.2.dev24+gdd58756) (3.4.8)
Requirement already satisfied: pynacl>=1.0.1 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from paramiko>=2.7.2->sshtunnel>=0.1.5->maggma==0.38.2.dev24+gdd58756) (1.4.0)
Requirement already satisfied: bcrypt>=3.1.3 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from paramiko>=2.7.2->sshtunnel>=0.1.5->maggma==0.38.2.dev24+gdd58756) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from requests->mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from requests->mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (2.0.9)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from requests->mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (1.26.7)
Requirement already satisfied: certifi>=2017.4.17 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from requests->mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (2021.10.8)
Requirement already satisfied: sniffio>=1.1 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from anyio<4,>=3.0.0->starlette==0.17.1->fastapi>=0.42.0->maggma==0.38.2.dev24+gdd58756) (1.2.0)
Requirement already satisfied: cffi>=1.1 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from bcrypt>=3.1.3->paramiko>=2.7.2->sshtunnel>=0.1.5->maggma==0.38.2.dev24+gdd58756) (1.15.0)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from Jinja2>=3.0->Flask>=1.0->mongogrant>=0.3.1->maggma==0.38.2.dev24+gdd58756) (2.0.1)
Requirement already satisfied: pycparser in /Users/janosh/.venv/py310/lib/python3.10/site-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko>=2.7.2->sshtunnel>=0.1.5->maggma==0.38.2.dev24+gdd58756) (2.21)
Building wheels for collected packages: maggma
  Building wheel for maggma (setup.py) ... done
  Created wheel for maggma: filename=maggma-0.38.2.dev24+gdd58756-py3-none-any.whl size=82120 sha256=c9c9f9432e573b1540c5a266cb6c68a1244ede3485d5d477f0745e97e43134cf
  Stored in directory: /private/var/folders/5m/sb0wx83j0m36xk_zswkfnffh0000gn/T/pip-ephem-wheel-cache-cjwcqz5v/wheels/88/14/03/17c3c16e3f5dc370a0e5e126414cdd74713b7d767f0831870b
Successfully built maggma
Installing collected packages: maggma
  Attempting uninstall: maggma
    Found existing installation: maggma 0.38.2.dev25+gd7d93f0.d20220125
    Uninstalling maggma-0.38.2.dev25+gd7d93f0.d20220125:
      Successfully uninstalled maggma-0.38.2.dev25+gd7d93f0.d20220125
Successfully installed maggma-0.38.2.dev24+gdd58756

[munrojm]

Great! Thank you again for bringing this up. Will close the issue when I finalize and merge.