Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove gRPC services and grpcio requirement #485

Merged
merged 9 commits into from
Jun 18, 2022
Merged

Conversation

mryab
Copy link
Member

@mryab mryab commented Jun 16, 2022

This PR removes the gRPC service definitions from hivemind/proto. Also, since we only need grpcio when compiling the proto files, I've moved them to the development dependencies (installed automatically if you use pip install -e .[dev], can be installed manually if you are not interested in tests)

@mryab
Copy link
Member Author

mryab commented Jun 16, 2022

Checked that keeping grpcio-tools does not require its installation for a user by publishing to test.pypi.org and installing hivemind from scratch:

~$ pip install -U --extra-index-url https://pypi.org/simple -i https://test.pypi.org/simple/ hivemind==1.1.0.dev0
Looking in indexes: https://test.pypi.org/simple/, https://pypi.org/simple
Collecting hivemind==1.1.0.dev0
  Downloading https://test-files.pythonhosted.org/packages/c7/ee/7bc955388e099709a6ef6180604d68c7b7b2e4c50146c2fd608b1e159132/hivemind-1.1.0.dev0-py3-none-any.whl (9.2 MB)
     |████████████████████████████████| 9.2 MB 1.8 MB/s
Collecting configargparse>=1.2.3
  Using cached ConfigArgParse-1.5.3-py3-none-any.whl (20 kB)
Collecting prefetch-generator>=1.0.1
  Using cached prefetch_generator-1.0.1.tar.gz (3.4 kB)
Collecting multiaddr>=0.0.9
  Using cached multiaddr-0.0.9-py2.py3-none-any.whl (16 kB)
Collecting msgpack>=0.5.6
  Downloading msgpack-1.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (316 kB)
     |████████████████████████████████| 316 kB 1.2 MB/s
Collecting sortedcontainers
  Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Collecting pymultihash>=0.8.2
  Using cached pymultihash-0.8.2-py3-none-any.whl (13 kB)
Collecting scipy>=1.2.1
  Downloading scipy-1.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.2 MB)
     |████████████████████████████████| 42.2 MB 158 kB/s
Collecting PyYAML
  Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
     |████████████████████████████████| 682 kB 73.9 MB/s
Collecting protobuf>=3.12.2
  Using cached protobuf-4.21.1-cp37-abi3-manylinux2014_x86_64.whl (407 kB)
Requirement already satisfied: numpy>=1.17 in ./miniconda3/envs/hivemind_env1/lib/python3.10/site-packages (from hivemind==1.1.0.dev0) (1.22.3)
Collecting pydantic>=1.8.1
  Downloading pydantic-1.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.0 MB)
     |████████████████████████████████| 12.0 MB 56.5 MB/s
Collecting uvloop>=0.14.0
  Downloading uvloop-0.16.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.3 MB)
     |████████████████████████████████| 4.3 MB 55.8 MB/s
Requirement already satisfied: torch>=1.6.0 in ./miniconda3/envs/hivemind_env1/lib/python3.10/site-packages (from hivemind==1.1.0.dev0) (1.11.0)
Requirement already satisfied: cryptography>=3.4.6 in ./miniconda3/envs/hivemind_env1/lib/python3.10/site-packages (from hivemind==1.1.0.dev0) (37.0.1)
Requirement already satisfied: cffi>=1.12 in ./miniconda3/envs/hivemind_env1/lib/python3.10/site-packages (from cryptography>=3.4.6->hivemind==1.1.0.dev0) (1.15.0)
Requirement already satisfied: pycparser in ./miniconda3/envs/hivemind_env1/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=3.4.6->hivemind==1.1.0.dev0) (2.21)
Collecting netaddr
  Using cached netaddr-0.8.0-py2.py3-none-any.whl (1.9 MB)
Collecting base58
  Using cached base58-2.1.1-py3-none-any.whl (5.6 kB)
Requirement already satisfied: six in ./miniconda3/envs/hivemind_env1/lib/python3.10/site-packages (from multiaddr>=0.0.9->hivemind==1.1.0.dev0) (1.16.0)
Collecting varint
  Using cached varint-1.0.2.tar.gz (1.9 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./miniconda3/envs/hivemind_env1/lib/python3.10/site-packages (from pydantic>=1.8.1->hivemind==1.1.0.dev0) (4.1.1)
Building wheels for collected packages: prefetch-generator, varint
  Building wheel for prefetch-generator (setup.py) ... done
  Created wheel for prefetch-generator: filename=prefetch_generator-1.0.1-py3-none-any.whl size=3961 sha256=c9bf5389bb8210e888e7175189e66f6f1a8117d6fb4db8667ebcd7b8a95019c0
  Stored in directory: /home/mryab/.cache/pip/wheels/3d/20/e7/7a45ee9e97711b465664c3108f7045afaa0be105d783c553bc
  Building wheel for varint (setup.py) ... done
  Created wheel for varint: filename=varint-1.0.2-py3-none-any.whl size=1979 sha256=3f7328e8e168b7861d9b86d86fd6f57fde73ef585e587977591a6913e6ccf270
  Stored in directory: /home/mryab/.cache/pip/wheels/39/48/5e/33919c52a2a695a512ca394a5308dd12626a40bbcd288de814
Successfully built prefetch-generator varint
Installing collected packages: varint, netaddr, base58, uvloop, sortedcontainers, scipy, PyYAML, pymultihash, pydantic, protobuf, prefetch-generator, multiaddr, msgpack, configargparse, hivemind
Successfully installed PyYAML-6.0 base58-2.1.1 configargparse-1.5.3 hivemind-1.1.0.dev0 msgpack-1.0.4 multiaddr-0.0.9 netaddr-0.8.0 prefetch-generator-1.0.1 protobuf-4.21.1 pydantic-1.9.1 pymultihash-0.8.2 scipy-1.8.1 sortedcontainers-2.4.0 uvloop-0.16.0 varint-1.0.2
~$ hivemind-dht
Jun 16 14:50:03.640 [INFO] Running a DHT instance. To connect other peers to this one, use --initial_peers /ip4/172.27.77.70/tcp/46463/p2p/QmS4nq6xPgs1aG56zPoEPVdUdTqKV5qP6rQCjgfHDvWxAc
Jun 16 14:50:03.640 [INFO] Full list of visible multiaddresses: /ip4/172.27.77.70/tcp/46463/p2p/QmS4nq6xPgs1aG56zPoEPVdUdTqKV5qP6rQCjgfHDvWxAc /ip4/127.0.0.1/tcp/46463/p2p/QmS4nq6xPgs1aG56zPoEPVdUdTqKV5qP6rQCjgfHDvWxAc
Jun 16 14:50:03.641 [INFO] 1 DHT nodes (including this one) are in the local routing table
Jun 16 14:50:03.641 [INFO] Local storage contains 0 keys

@codecov
Copy link

codecov bot commented Jun 16, 2022

Codecov Report

Merging #485 (1828756) into master (f60e34a) will increase coverage by 0.24%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #485      +/-   ##
==========================================
+ Coverage   85.57%   85.82%   +0.24%     
==========================================
  Files          80       80              
  Lines        7794     7794              
==========================================
+ Hits         6670     6689      +19     
+ Misses       1124     1105      -19     
Impacted Files Coverage Δ
hivemind/optim/progress_tracker.py 97.80% <0.00%> (-1.10%) ⬇️
hivemind/dht/node.py 91.68% <0.00%> (+0.23%) ⬆️
hivemind/averaging/averager.py 89.44% <0.00%> (+1.19%) ⬆️
hivemind/averaging/matchmaking.py 88.69% <0.00%> (+4.46%) ⬆️

README.md Outdated
@@ -53,6 +53,7 @@ To install hivemind from source, simply run the following:
```
git clone https://github.com/learning-at-home/hivemind.git
cd hivemind
pip install "grpcio-tools>=1.33.2" # for protobuf compilation
Copy link
Member

@justheuristic justheuristic Jun 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we say pip install -r requirements-dev.txt instead?
Rationale: less chance that we forget to update the version later.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is, check the diff and PR description. The rationale is that you don't need to install all the development deps just to make hivemind work

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, have it your way :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And regarding the "forget to update later" part: we haven't bumped this requirement for the past 2 years and are unlikely to to do this soon, so I'm not sure if this is something that we should worry about

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is not okay for me, such things should be moved to requirements (that's exactly what requirements are for).

Copy link
Member Author

@mryab mryab Jun 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, it is present in the development requirements file. The issue is that we do not need this package at runtime when we are installing hivemind normally (e.g. from pip), so a regular user should not need to install it (because the pip-installed version contains already compiled protobuf files)

Copy link
Member

@justheuristic justheuristic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done!

README.md Outdated
@@ -53,6 +53,7 @@ To install hivemind from source, simply run the following:
```
git clone https://github.com/learning-at-home/hivemind.git
cd hivemind
pip install "grpcio-tools>=1.33.2" # for protobuf compilation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is not okay for me, such things should be moved to requirements (that's exactly what requirements are for).

@mryab mryab changed the title Remove gRPC services and Protobuf compiler requirement Remove gRPC services and grpcio requirement Jun 17, 2022
@mryab mryab merged commit 712e428 into master Jun 18, 2022
@mryab mryab deleted the remove_grpc_compilation branch June 18, 2022 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants