Add Api Configuration

README.md

@@ -11,25 +11,25 @@
 
 **[The Golem Network](https://golem.network)** fosters a global group of creators building ambitious software solutions that will shape the technological landscape of future generations by accessing computing resources across the platform. Golem Network is an accessible, reliable, open access and censorship-resistant protocol, democratizing access to digital resources and connecting users through a flexible, open-source platform.
 
-
 **Yapapi** is the Python high-level API that allows developers to connect to their Golem nodes and manage their distributed, computational loads through Golem Network.
 
 ## Golem application development
 
 For a detailed introduction to using Golem and yapapi to run your tasks on Golem and a guide to Golem Network application development in general, [please consult our handbook](https://handbook.golem.network/requestor-tutorials/flash-tutorial-of-requestor-development).
 
-
 ### Installation
 
 `yapapi` is available as a [PyPI package](https://pypi.org/project/yapapi/).
 
 You can install it through `pip`:
-```
+
+```bash
 pip install yapapi
 ```
 
 Or if your project uses [`poetry`](https://python-poetry.org/) you can add it to your dependencies like this:
-```
+
+```bash
 poetry add yapapi
 ```
 
@@ -40,19 +40,24 @@ For a comprehensive API reference, please refer to [our official readthedocs pag
 ## Local setup for yapapi developers
 
 ### Poetry
+
 `yapapi` uses [`poetry`](https://python-poetry.org/) to manage its dependencies and provide a runner for common tasks.
 
 If you don't have `poetry` available on your system then follow its [installation instructions](https://python-poetry.org/docs/#installation) before proceeding.
 Verify your installation by running:
-```
+
+```bash
 poetry --version
 ```
 
 ### Project dependencies
+
 To install the project's dependencies run:
-```
+
+```bash
 poetry install
 ```
+
 By default, `poetry` looks for the required Python version on your `PATH` and creates a virtual environment for the project if there's none active (or already configured by Poetry).
 
 All of the project's dependencies will be installed to that virtual environment.
@@ -62,7 +67,7 @@ All of the project's dependencies will be installed to that virtual environment.
 `yapapi` uses [Poe the Poet](https://github.com/nat-n/poethepoet) for running tasks.
 Declarations of project tasks can be found in `pyproject.toml`.
 
-```
+```bash
 poetry run poe test
 ```
 
@@ -78,69 +83,71 @@ Next, [configure goth's GitHub API token](https://github.com/golemfactory/goth#g
 
 Make sure you have OpenSSH installed and added to path
 
-```
+```bash
 ssh -V
 ```
 
 Now, you can install goth and its additional python requirements:
 
-```
+```bash
 poetry install -E integration-tests
 ```
 
 Finally, generate goth's default assets:
 
-```
+```bash
 poetry run poe goth-assets
 ```
 
 #### Running the tests
 
 Once you have the environment set up, to run all the integration tests, use:
 
-```
+```bash
 poetry run poe goth-tests
 ```
 
 ### Contributing
 
 It is recommended to run unit tests and static code analysis before committing changes.
 
-```
+```bash
 poetry run poe check
 ```
 
 You can clean up the artifacts created during the test runs with:
 
-```
+```bash
 poetry run poe clean
 ```
 
 ## See also
 
 * [Golem](https://golem.network), a global, open-source, decentralized supercomputer that anyone can access.
 * Learn what you need to know to set-up your Golem requestor node:
-    * [Requestor development: a quick primer](https://handbook.golem.network/requestor-tutorials/flash-tutorial-of-requestor-development)
-    * [Run first task on Golem](https://handbook.golem.network/requestor-tutorials/flash-tutorial-of-requestor-development/run-first-task-on-golem)
+  * [Requestor development: a quick primer](https://handbook.golem.network/requestor-tutorials/flash-tutorial-of-requestor-development)
+  * [Run first task on Golem](https://handbook.golem.network/requestor-tutorials/flash-tutorial-of-requestor-development/run-first-task-on-golem)
 * Have a look at the most important concepts behind any Golem application: [Golem application fundamentals](https://handbook.golem.network/requestor-tutorials/golem-application-fundamentals)
 * Learn about preparing your own Docker-like images for the [VM runtime](https://handbook.golem.network/requestor-tutorials/vm-runtime)
 * Write your own app with yapapi:
-    * [Task Model development](https://handbook.golem.network/requestor-tutorials/task-processing-development)
-    * [Service Model development](https://handbook.golem.network/requestor-tutorials/service-development)
+  * [Task Model development](https://handbook.golem.network/requestor-tutorials/task-processing-development)
+  * [Service Model development](https://handbook.golem.network/requestor-tutorials/service-development)
 
 ## Environment variables
 
 It's possible to set various elements of `yagna` configuration through environment variables.
 `yapapi` currently supports the following environment variables:
-- `YAGNA_ACTIVITY_URL`, URL to `yagna` activity API, e.g. `http://localhost:7500/activity-api/v1`
-- `YAGNA_API_URL`, base URL to `yagna` REST API, e.g. `http://localhost:7500`
-- `YAGNA_APPKEY`, `yagna` app key to be used, e.g. `a70facb9501d4528a77f25574ab0f12b`
-- `YAGNA_MARKET_URL`, URL to `yagna` market API, e.g. `http://localhost:7500/market-api/v1`
-- `YAGNA_PAYMENT_NETWORK`, Ethereum network name for `yagna` to use, e.g. `rinkeby`
-- `YAGNA_PAYMENT_DRIVER`, payment driver name for `yagna` to use, e.g. `erc20`
-- `YAGNA_PAYMENT_URL`, URL to `yagna` payment API, e.g. `http://localhost:7500/payment-api/v1`
-- `YAGNA_SUBNET`, name of the `yagna` sub network to be used, e.g. `public`
-- `YAPAPI_USE_GFTP_CLOSE`, if set to a _truthy_ value (e.g. "1", "Y", "True", "on") then `yapapi`
+
+* `YAGNA_ACTIVITY_URL`, URL to `yagna` activity API, e.g. `http://localhost:7500/activity-api/v1`
+* `YAGNA_API_URL`, base URL to `yagna` REST API, e.g. `http://localhost:7500`
+* `YAGNA_APPKEY`, `yagna` app key to be used, e.g. `a70facb9501d4528a77f25574ab0f12b`
+* `YAGNA_MARKET_URL`, URL to `yagna` market API, e.g. `http://localhost:7500/market-api/v1`
+* `YAGNA_PAYMENT_NETWORK`, Ethereum network name for `yagna` to use, e.g. `rinkeby`
+* `YAGNA_PAYMENT_DRIVER`, payment driver name for `yagna` to use, e.g. `erc20`
+* `YAGNA_PAYMENT_URL`, URL to `yagna` payment API, e.g. `http://localhost:7500/payment-api/v1`
+* `YAGNA_NET_URL`, URL to `yagna` net APU, e.g. `http://localhost:7500/net-api/v1`
+* `YAGNA_SUBNET`, name of the `yagna` sub network to be used, e.g. `public`
+* `YAPAPI_USE_GFTP_CLOSE`, if set to a _truthy_ value (e.g. "1", "Y", "True", "on") then `yapapi`
   will ask `gftp` to close files when there's no need to publish them any longer. This may greatly
   reduce the number of files kept open while `yapapi` is running but requires `yagna`
   0.7.3 or newer, with older versions it will cause errors.

examples/low-level-api/list-offers.py

@@ -7,6 +7,7 @@
 import sys
 
 from yapapi import props as yp
+from yapapi.config import ApiConfig
 from yapapi.log import enable_default_logger
 from yapapi.props.builder import DemandBuilder
 from yapapi.rest import Activity, Configuration, Market, Payment  # noqa
@@ -44,7 +45,7 @@ def main():
         asyncio.get_event_loop().run_until_complete(
             asyncio.wait_for(
                 list_offers(
-                    Configuration(),
+                    Configuration(api_config=ApiConfig()),  # YAGNA_APPKEY will be loaded from env
                     subnet_tag=subnet,
                 ),
                 timeout=4,

pyproject.toml

@@ -115,7 +115,7 @@ ya-market = "0.1.0"
 test = "pytest --cov=yapapi --ignore tests/goth_tests"
 goth-assets = "python -m goth create-assets tests/goth_tests/assets"
 goth-tests = "pytest -svx tests/goth_tests  --config-override docker-compose.build-environment.use-prerelease=true --config-path tests/goth_tests/assets/goth-config-testing.yml --ssh-verify-connection --reruns 3 --only-rerun AssertionError --only-rerun TimeoutError --only-rerun goth.runner.exceptions.TemporalAssertionError --only-rerun urllib.error.URLError --only-rerun goth.runner.exceptions.CommandError"
-typecheck = "mypy --check-untyped-defs --no-implicit-optional ."
+typecheck = "mypy --check-untyped-defs --no-implicit-optional --show-error-codes ."
 _codestyle_isort = "isort --check-only --diff ."
 _codestyle_black = "black --check --diff ."
 codestyle = ["_codestyle_isort", "_codestyle_black"]

tests/conftest.py

@@ -1,5 +1,9 @@
+import os
 import pytest
 
+from yapapi.config import ApiConfig
+from yapapi.golem import Golem
+
 
 def pytest_addoption(parser):
 
@@ -29,3 +33,19 @@ async def _gftp_aexit(self, *args):
     monkeypatch.setattr(_Engine, "_create_allocations", _engine_create_allocations)
     monkeypatch.setattr(GftpProvider, "__aenter__", _gftp_aenter)
     monkeypatch.setattr(GftpProvider, "__aexit__", _gftp_aexit)
+
+
+@pytest.fixture
+def purge_yagna_os_env() -> None:
+    for key in [
+        "YAGNA_APPKEY",
+        "YAGNA_API_URL",
+        "YAGNA_MARKET_URL",
+        "YAGNA_PAYMENT_URL",
+        "YAGNA_NET_URL",
+        "YAGNA_ACTIVITY_URL",
+        "YAGNA_SUBNET",
+        "YAGNA_PAYMENT_DRIVER",
+        "YAGNA_PAYMENT_NETWORK",
+    ]:
+        os.environ.pop(key, None)

tests/engine/test_debit_note_intervals.py

@@ -16,8 +16,12 @@
 def mock_engine(
     agreement: Agreement, debit_note: DebitNote, num_debit_notes=0, num_payable_debit_notes=0
 ) -> _Engine:
-    with mock.patch("yapapi.engine.rest.Configuration"):
-        engine = _Engine(budget=0.0, strategy=mock.Mock(), event_consumer=mock.Mock())
+    engine = _Engine(
+        budget=0.0,
+        strategy=mock.Mock(),
+        event_consumer=mock.Mock(),
+        api_config=mock.Mock(),
+    )
 
     engine._all_agreements[agreement.id] = agreement  # noqa
     engine._num_debit_notes[debit_note.activity_id] = num_debit_notes  # noqa

tests/engine/test_engine.py

@@ -2,18 +2,14 @@
 import pytest
 from unittest.mock import Mock
 
+from tests.factories.golem import GolemFactory
 from yapapi import Golem
+from yapapi.config import ApiConfig
 import yapapi.engine
 from yapapi.engine import Job
 import yapapi.rest
 
 
-@pytest.fixture(autouse=True)
-def mock_rest_configuration(monkeypatch):
-    """Mock `yapapi.rest.Configuration`."""
-    monkeypatch.setattr(yapapi.rest, "Configuration", Mock)
-
-
 @pytest.mark.parametrize(
     "default_subnet, subnet_arg, expected_subnet",
     [
@@ -29,9 +25,9 @@ def test_set_subnet_tag(default_subnet, subnet_arg, expected_subnet, monkeypatch
     monkeypatch.setattr(yapapi.engine, "DEFAULT_SUBNET", default_subnet)
 
     if subnet_arg is not None:
-        golem = Golem(budget=1.0, subnet_tag=subnet_arg)
+        golem = GolemFactory(subnet_tag=subnet_arg)
     else:
-        golem = Golem(budget=1.0)
+        golem = GolemFactory()
     assert golem.subnet_tag == expected_subnet
 
 

tests/factories/config.py

@@ -0,0 +1,10 @@
+import factory
+
+import yapapi.config
+
+
+class ApiConfigFactory(factory.Factory):
+    class Meta:
+        model = yapapi.config.ApiConfig
+
+    app_key = "yagna-app-key"

tests/factories/golem.py

@@ -0,0 +1,12 @@
+import factory
+
+from tests.factories.config import ApiConfigFactory
+import yapapi.golem
+
+
+class GolemFactory(factory.Factory):
+    class Meta:
+        model = yapapi.golem.Golem
+
+    budget = 10.0
+    api_config = factory.SubFactory(ApiConfigFactory)

tests/goth_tests/test_renegotiate_proposal/requestor.py

@@ -8,6 +8,7 @@
 
 from examples import utils
 from yapapi import props as yp
+from yapapi.config import ApiConfig
 from yapapi.log import enable_default_logger
 from yapapi.props.builder import DemandBuilder
 from yapapi.rest import Configuration, Market
@@ -93,7 +94,7 @@ def main():
         asyncio.get_event_loop().run_until_complete(
             asyncio.wait_for(
                 renegotiate_offers(
-                    Configuration(),
+                    Configuration(api_config=ApiConfig()),
                     subnet_tag=subnet,
                 ),
                 timeout=140,

tests/goth_tests/test_resubscription.py

@@ -85,7 +85,7 @@ def patch_collect_offers(monkeypatch):
 
 async def unsubscribe_demand(sub_id: str) -> None:
     """Auxiliary function that calls `unsubscribeDemand` operation for given `sub_id`."""
-    config = yapapi.rest.Configuration()
+    config = yapapi.rest.Configuration(api_config=yapapi.config.ApiConfig())
     market_client = config.market()
     requestor_api = yapapi.rest.market.RequestorApi(market_client)
     await requestor_api.unsubscribe_demand(sub_id)

tests/rest/test_allocation.py

@@ -1,12 +1,13 @@
 from decimal import Decimal
 import pytest
 
+from yapapi.config import ApiConfig
 from yapapi.rest import Configuration, Payment
 
 
 @pytest.fixture
 async def yapapi_payment(request):
-    conf = Configuration(app_key=request.config.getvalue("yaApiKey"))
+    conf = Configuration(api_config=ApiConfig(app_key=request.config.getvalue("yaApiKey")))
     async with conf.payment() as p:
         yield Payment(p)
 

tests/services/test_cluster.py

@@ -5,6 +5,7 @@
 from unittest import mock
 from unittest.mock import Mock, patch
 
+from tests.factories.golem import GolemFactory
 from yapapi import Golem
 from yapapi.services import Service, ServiceRunner
 
@@ -88,7 +89,7 @@ def _get_new_engine(self):
     monkeypatch.setattr(Golem, "_get_new_engine", _get_new_engine)
 
     with patch("yapapi.services.ServiceRunner.spawn_instance") as spawn_instance:
-        golem = Golem(budget=1)
+        golem = GolemFactory()
         try:
             await golem.run_service(
                 service_class=_TestService, payload=Mock(), network=Mock(), **kwargs

tests/strategy/test_default_strategies.py

@@ -3,6 +3,7 @@
 import pytest
 from unittest.mock import Mock
 
+from tests.factories.golem import GolemFactory
 from tests.factories.rest.market import OfferProposalFactory
 from yapapi import Golem
 from yapapi.props.com import Counter
@@ -116,10 +117,7 @@ async def _test_strategy(strategy, cpu_price_cap, time_price_cap, fixed_price_ca
 @pytest.mark.asyncio
 async def test_default_strategy_type(monkeypatch):
     """Test if the default strategy is composed of appropriate `MarketStrategy` subclasses."""
-
-    monkeypatch.setattr(yapapi.rest, "Configuration", Mock)
-
-    golem = Golem(budget=1.0)
+    golem = GolemFactory()
     default_strategy = golem.strategy
     assert isinstance(default_strategy, DecreaseScoreForUnconfirmedAgreement)
     assert isinstance(default_strategy.base_strategy, LeastExpensiveLinearPayuMS)
@@ -128,11 +126,8 @@ async def test_default_strategy_type(monkeypatch):
 @pytest.mark.asyncio
 async def test_user_strategy_not_modified(monkeypatch):
     """Test that a user strategy is not wrapped in `DecreaseScoreForUnconfirmedAgreement`."""
-
-    monkeypatch.setattr(yapapi.rest, "Configuration", Mock)
-
     user_strategy = Mock()
-    golem = Golem(budget=1.0, strategy=user_strategy)
+    golem = GolemFactory(strategy=user_strategy)
     assert golem.strategy == user_strategy