migrate dependencies to setup.py and update CI config #855

rkingsbury · 2023-09-01T13:04:57Z

Migrate all requirements into setup.py and fix problems with the CI using the auto-generated requirements files

Automated dependency upgrades

rkingsbury · 2023-09-01T16:37:21Z

@munrojm I'm cleaning up our dependency management, and I can't seem to figure out the source of this error in test_cluster_run. Do you have any ideas? I think my changes here should create an equivalent environment to what we had before, but I have a feeling we are now somehow getting a different version of requests.

_______________________________ test_cluster_run _______________________________

payload = '', base = '/?', debug = True

    def search_helper(payload, base: str = "/?", debug=True) -> Tuple[Response, Any]:
        """
        Helper function to directly query search endpoints
        Args:
            store: store f
            base: base of the query, default to /query?
            client: TestClient generated from FastAPI
            payload: query in dictionary format
            debug: True = print out the url, false don't print anything
    
        Returns:
            request.Response object that contains the response of the corresponding payload
        """
        owner_store = MemoryStore("owners", key="name")
        owner_store.connect()
        owner_store.update([d.dict() for d in owners])
    
        pets_store = MemoryStore("pets", key="name")
        pets_store.connect()
        pets_store.update([jsonable_encoder(d) for d in pets])
    
        resources = {
            "owners": [
                ReadOnlyResource(
                    owner_store,
                    Owner,
                    query_operators=[
                        StringQueryOperator(model=Owner),  # type: ignore
                        NumericQuery(model=Owner),  # type: ignore
                        SparseFieldsQuery(model=Owner),
                        PaginationQuery(),
                    ],
                )
            ],
            "pets": [
                ReadOnlyResource(
                    pets_store,
                    Owner,
                    query_operators=[
                        StringQueryOperator(model=Pet),
                        NumericQuery(model=Pet),
                        SparseFieldsQuery(model=Pet),
                        PaginationQuery(),
                    ],
                )
            ],
        }
        api = API(resources=resources)
    
        client = TestClient(api.app)
    
        url = base + urlencode(payload)
        if debug:
            print(url)
        res = client.get(url)
        try:
>           data = res.json().get("data", [])

tests/api/test_api.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/httpx/_models.py:756: in json
    return jsonlib.loads(self.text, **kwargs)
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/json/__init__.py:346: in loads
    return _default_decoder.decode(s)
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/json/decoder.py:337: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <json.decoder.JSONDecoder object at 0x7f760501eed0>
s = '\n    <!DOCTYPE html>\n    <html>\n    <head>\n    <link type="text/css" rel="stylesheet" href="[https://cdn.jsdelivr....\n](https://cdn.jsdelivr..../n)        SwaggerUIBundle.SwaggerUIStandalonePreset\n        ],\n    })\n    </script>\n    </body>\n    </html>\n    '
idx = 5

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.
    
        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.
    
        """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 2 column 5 (char 5)

/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/json/decoder.py:355: JSONDecodeError

During handling of the above exception, another exception occurred:

owner_store = <maggma.stores.mongolike.MemoryStore object at 0x7f75f6e4fdd0>
pet_store = <maggma.stores.mongolike.MemoryStore object at 0x7f75f6ea6a90>

    def test_cluster_run(owner_store, pet_store):
>       res, data = search_helper(payload="")

tests/api/test_api.py:139: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

payload = '', base = '/?', debug = True

    def search_helper(payload, base: str = "/?", debug=True) -> Tuple[Response, Any]:
        """
        Helper function to directly query search endpoints
        Args:
            store: store f
            base: base of the query, default to /query?
            client: TestClient generated from FastAPI
            payload: query in dictionary format
            debug: True = print out the url, false don't print anything
    
        Returns:
            request.Response object that contains the response of the corresponding payload
        """
        owner_store = MemoryStore("owners", key="name")
        owner_store.connect()
        owner_store.update([d.dict() for d in owners])
    
        pets_store = MemoryStore("pets", key="name")
        pets_store.connect()
        pets_store.update([jsonable_encoder(d) for d in pets])
    
        resources = {
            "owners": [
                ReadOnlyResource(
                    owner_store,
                    Owner,
                    query_operators=[
                        StringQueryOperator(model=Owner),  # type: ignore
                        NumericQuery(model=Owner),  # type: ignore
                        SparseFieldsQuery(model=Owner),
                        PaginationQuery(),
                    ],
                )
            ],
            "pets": [
                ReadOnlyResource(
                    pets_store,
                    Owner,
                    query_operators=[
                        StringQueryOperator(model=Pet),
                        NumericQuery(model=Pet),
                        SparseFieldsQuery(model=Pet),
                        PaginationQuery(),
                    ],
                )
            ],
        }
        api = API(resources=resources)
    
        client = TestClient(api.app)
    
        url = base + urlencode(payload)
        if debug:
            print(url)
        res = client.get(url)
        try:
            data = res.json().get("data", [])
        except Exception:
>           data = res.reason
E           AttributeError: 'Response' object has no attribute 'reason'

tests/api/test_api.py:133: AttributeError
----------------------------- Captured stdout call -----------------------------
/?

...

=========================== short test summary info ============================
FAILED api/api::cluster run - AttributeError: 'Response' object has no attribute 'reason'
== 1 failed, 213 passed, 8 skipped, 6 xfailed, 2 xpassed in 157.24s (0:02:37) ==

codecov · 2023-09-06T20:54:31Z

Codecov Report

Patch coverage has no change and project coverage change: -0.20% ⚠️

Comparison is base (508c173) 88.14% compared to head (0c2025a) 87.94%.
Report is 39 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #855      +/-   ##
==========================================
- Coverage   88.14%   87.94%   -0.20%     
==========================================
  Files          44       44              
  Lines        3593     3593              
==========================================
- Hits         3167     3160       -7     
- Misses        426      433       +7

see 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

rkingsbury added 2 commits September 1, 2023 09:04

CI: add missing test dependency

bc35000

CI: rearrange requirements in setup.py

9e1778e

rkingsbury changed the title ~~CI: add missing test dependency~~ migrate dependencies to setup.py and update CI config Sep 1, 2023

rkingsbury added the release:minor label Sep 1, 2023

rkingsbury added 7 commits September 1, 2023 11:35

CI: auto dep test integration updates

7d9ad19

Deps: add uvicorn to setup.py

6689970

GH actions updates with new reqs files

f9e7b13

mkdocs config update

91f175e

update numpy testing import

b28b63e

Deps: fix responses < 0.22 in setup.py

2b8653e

CI configuration cleanups per repo-review

8cf8cd4

rkingsbury force-pushed the main branch 2 times, most recently from 59c00d1 to 8cf8cd4 Compare September 1, 2023 16:23

github-actions and others added 9 commits September 1, 2023 16:26

update dependencies for . (macos-latest/py3.10)

aaf707a

update dependencies for . (macos-latest/py3.11)

f407032

update dependencies for . (macos-latest/py3.8)

dd5d815

update dependencies for . (macos-latest/py3.9)

0e3af06

update dependencies for . (ubuntu-latest/py3.10)

c07c265

update dependencies for . (ubuntu-latest/py3.11)

230d317

update dependencies for . (ubuntu-latest/py3.8)

b72f534

update dependencies for . (ubuntu-latest/py3.9)

bd1ef20

Merge pull request #327 from rkingsbury/auto-dependency-upgrades

ee8ff1b

Automated dependency upgrades

Jason added 3 commits September 6, 2023 13:39

Change test exception catch

ec82542

Merge remote-tracking branch 'origin/main' into rking_main

286c31b

Fix req merge

0c2025a

rkingsbury merged commit 6e27a98 into materialsproject:main Sep 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate dependencies to setup.py and update CI config #855

migrate dependencies to setup.py and update CI config #855

rkingsbury commented Sep 1, 2023 •

edited

Loading

rkingsbury commented Sep 1, 2023

codecov bot commented Sep 6, 2023

migrate dependencies to setup.py and update CI config #855

migrate dependencies to setup.py and update CI config #855

Conversation

rkingsbury commented Sep 1, 2023 • edited Loading

rkingsbury commented Sep 1, 2023

codecov bot commented Sep 6, 2023

Codecov Report

rkingsbury commented Sep 1, 2023 •

edited

Loading