Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release to master #55

Merged
merged 52 commits into from
Oct 16, 2023
Merged
Changes from 3 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
7cf101d
Token-based authorization.
mgcam Mar 3, 2022
ce17035
correct name for the test file
mgcam Mar 9, 2022
60aa467
Bug fix: need left join to link tokens to pipelines
mgcam Mar 9, 2022
ced9197
Provide clue when testing mode not enabled
nerdstrike Mar 7, 2022
5adeeba
Remove spurious test mark
nerdstrike Mar 7, 2022
1192809
Document some settings for production use
nerdstrike Mar 9, 2022
6230a85
Improve regexp for the token string.
mgcam Mar 9, 2022
93dea5e
Use custom exception for token validation
mgcam Mar 9, 2022
908c868
Exception declaration and propagation
mgcam Mar 9, 2022
24b40ad
Copyright notices added/fixed
mgcam Mar 9, 2022
d314852
Token auth for request to create a pipeline.
mgcam Mar 10, 2022
45fa130
Merge pull request #33 from mgcam/auth4pipelines
nerdstrike Mar 10, 2022
e2c00c7
Merge pull request #34 from nerdstrike/quality_of_life
mgcam Mar 10, 2022
36f7271
Copyright notices added/fixed
mgcam Mar 9, 2022
ad032b9
Simple token auth for task endpoints
mgcam Mar 10, 2022
1d75aed
Pipeline-aware token auth for task endpoints
mgcam Mar 11, 2022
18a69b7
Fix typo
mgcam Mar 14, 2022
4f36cb6
Simplified class definitions for custom exceptions.
mgcam Mar 14, 2022
cfca748
Merge pull request #35 from mgcam/token_auth4task_endpoints
nerdstrike Mar 14, 2022
03e0da1
Update the user guide to describe token use.
nerdstrike Mar 15, 2022
10e518f
Addendum about proxies
nerdstrike Mar 15, 2022
75e1b15
Mask some tool baggage
nerdstrike Mar 15, 2022
97f05d6
Big refactor to make package install and be testable in develop and r…
nerdstrike Mar 17, 2022
253de7d
Make scripts and server correctly use postgres schema
nerdstrike Mar 22, 2022
5282e51
Prevent tests from receiving postgres driver options
nerdstrike Mar 23, 2022
59453b6
Make user guide refer to https wherever possible
nerdstrike Mar 24, 2022
fb82a89
Merge pull request #36 from nerdstrike/token_admin
mgcam Mar 24, 2022
9f4b71b
Complete get_tasks coverage and allow filtering by pipeline name and …
nerdstrike Apr 8, 2022
0e6384e
Merge pull request #42 from nerdstrike/filtered_tasks
mgcam Apr 9, 2022
90dc184
Newer starlette releases require httpx, and redirect handling has cha…
nerdstrike Dec 13, 2022
8661051
Merge pull request #46 from wtsi-npg/fix_starlette_dependency
mgcam Dec 16, 2022
cfab67a
Move from checkout@v2 to @v3
kjsanger Dec 16, 2022
23cdb35
Merge pull request #47 from wtsi-npg/kjsanger-patch-2
jmtcsngr Dec 16, 2022
6b36e5c
Prevent sqlalchemy 2, pydantic 2 from installing
nerdstrike Aug 15, 2023
f93107d
Merge branch 'devel' into stabilize_deps
nerdstrike Aug 15, 2023
2016f5d
Merge pull request #49 from nerdstrike/stabilize_deps
nerdstrike Aug 15, 2023
dc38619
Switch to pyproject file for package setup
nerdstrike Jul 26, 2023
36a996a
Use setup.cfg to find modules, remove redundant setup.py
nerdstrike Aug 22, 2023
c4ee822
Annoying httpx dependency absence
nerdstrike Aug 22, 2023
497acdd
Push to python 3.10
nerdstrike Aug 22, 2023
34b39ae
Merge pull request #48 from nerdstrike/update_packaging
mgcam Aug 22, 2023
010b0c0
Compatibility upgrade to use sqlachemy 2
nerdstrike Aug 22, 2023
f870e48
Adjust pydantic use until tests pass.
nerdstrike Aug 24, 2023
9c3e56d
Satisfy pydantic deprecation warnings
nerdstrike Aug 24, 2023
fe491c5
Merge pull request #50 from nerdstrike/sqlalchemy_upgrade
mgcam Aug 25, 2023
aae9ee3
Merge pull request #51 from nerdstrike/upgrade_pydantic
mgcam Aug 25, 2023
7432a8e
Use best/better practices for type hints and pydantic constraints.
nerdstrike Oct 11, 2023
3bae5f3
Merge pull request #52 from nerdstrike/pydantic_second_pass
mgcam Oct 11, 2023
3a0b662
Try to prevent dormant DB connection from causing 500 errors for the …
nerdstrike Oct 12, 2023
f96876d
Merge pull request #53 from nerdstrike/db_reconnect
mgcam Oct 12, 2023
80ca8fc
Version update
nerdstrike Oct 16, 2023
bd9f0f0
Merge pull request #54 from nerdstrike/devel
nerdstrike Oct 16, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@ dependencies = [
"aiosqlite",
"asyncpg",
"fastapi",
"pydantic < 2.0.0",
"pydantic > 2.0.0",
"pysqlite3",
"psycopg2-binary",
"sqlalchemy >2",
4 changes: 2 additions & 2 deletions server/npg/porch/endpoints/tasks.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2021, 2022 Genome Research Ltd.
# Copyright (C) 2021, 2022, 2023 Genome Research Ltd.
#
# Author: Kieron Taylor kt19@sanger.ac.uk
# Author: Marina Gourtovaia mg8@sanger.ac.uk
@@ -173,7 +173,7 @@ async def update_task(
)
async def claim_task(
pipeline: Pipeline,
num_tasks: PositiveInt = 1,
num_tasks: PositiveInt | None = 1,
db_accessor=Depends(get_DbAccessor),
permission=Depends(validate)
) -> List[Task]:
11 changes: 6 additions & 5 deletions server/npg/porch/models/permission.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2022 Genome Research Ltd.
# Copyright (C) 2022. 2023 Genome Research Ltd.
#
# Author: Kieron Taylor kt19@sanger.ac.uk
# Author: Marina Gourtovaia mg8@sanger.ac.uk
@@ -19,7 +19,7 @@
# this program. If not, see <http://www.gnu.org/licenses/>.

from enum import Enum
from pydantic import BaseModel, Field, validator
from pydantic import BaseModel, Field, field_validator, FieldValidationInfo
from typing import Optional

from npg.porch.models.pipeline import Pipeline
@@ -49,11 +49,12 @@ class Permission(BaseModel):
title = 'A role associated with the presented credentials',
)

@validator('role')
def no_pipeline4special_users(cls, v, values):
@field_validator('role')
@classmethod
def no_pipeline4special_users(cls, v: str, info: FieldValidationInfo):

if (v == RolesEnum.POWER_USER
and ('pipeline' in values and values['pipeline'] is not None)):
and ('pipeline' in info.data and info.data['pipeline'] is not None)):
raise ValueError('Power user cannot be associated with a pipeline')

return v
10 changes: 5 additions & 5 deletions server/npg/porch/models/task.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2021, 2022 Genome Research Ltd.
# Copyright (C) 2021, 2022, 2023 Genome Research Ltd.
#
# Author: Kieron Taylor kt19@sanger.ac.uk
# Author: Marina Gourtovaia mg8@sanger.ac.uk
@@ -46,7 +46,7 @@ class Task(BaseModel):
title='Task Input',
description='A structured parameter set that uniquely identifies a piece of work, and enables an iteration of a pipeline' # noqa: E501
)
status: Optional[TaskStateEnum]
status: Optional[TaskStateEnum] = None

def generate_task_id(self):
return hashlib.sha256(ujson.dumps(self.task_input, sort_keys=True).encode()).hexdigest()
@@ -65,13 +65,13 @@ def __eq__(self, other):
'''
if not isinstance(other, Task):
if isinstance(other, dict):
other = Task.parse_obj(other)
other = Task.model_validate(other)
else:
return False

truths = []
for k, v in self.dict().items():
other_d = other.dict()
for k, v in self.model_dump().items():
other_d = other.model_dump()
if k == 'pipeline':
truths.append(v['name'] == other_d[k]['name'])
elif k == 'task_input_id':
6 changes: 3 additions & 3 deletions tests/model_permission_test.py
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@

from npg.porch.models.pipeline import Pipeline
from npg.porch.models.permission import Permission, PermissionValidationException
from pydantic.error_wrappers import ValidationError
from pydantic import ValidationError


def test_model_create():
@@ -37,12 +37,12 @@ def test_xvalidation_role_pipeline():

def test_error_with_insufficient_args():

with pytest.raises(ValidationError, match=r'requestor_id\s+field required'):
with pytest.raises(ValidationError, match=r'requestor_id\s+Field required'):
Permission(
role = 'regular_user',
pipeline = Pipeline(name='number one')
)
with pytest.raises(ValidationError, match=r'role\s+field required'):
with pytest.raises(ValidationError, match=r'role\s+Field required'):
Permission(
requestor_id = 1,
pipeline = Pipeline(name='number one')
20 changes: 10 additions & 10 deletions tests/pipeline_route_test.py
Original file line number Diff line number Diff line change
@@ -16,18 +16,18 @@
def http_create_pipeline(fastapi_testclient, pipeline):

response = fastapi_testclient.post(
'/pipelines', json=pipeline.dict(), follow_redirects=True
'/pipelines', json=pipeline.model_dump(), follow_redirects=True
)
assert response.status_code == status.HTTP_403_FORBIDDEN

response = fastapi_testclient.post(
'/pipelines', json=pipeline.dict(), follow_redirects=True,
'/pipelines', json=pipeline.model_dump(), follow_redirects=True,
headers=headers
)
assert response.status_code == status.HTTP_403_FORBIDDEN

response = fastapi_testclient.post(
'/pipelines', json=pipeline.dict(), follow_redirects=True,
'/pipelines', json=pipeline.model_dump(), follow_redirects=True,
headers=headers4power_user
)
assert response.status_code == status.HTTP_201_CREATED
@@ -41,7 +41,7 @@ def test_pipeline_get(async_minimum, fastapi_testclient):
assert response.status_code == status.HTTP_403_FORBIDDEN
response = fastapi_testclient.get('/pipelines', headers=headers)
assert response.status_code == status.HTTP_200_OK
pipeline = Pipeline.parse_obj(response.json()[0])
pipeline = Pipeline.model_validate(response.json()[0])
assert pipeline, 'Response fits into the over-the-wire model'
assert pipeline.name == 'ptest one'
assert pipeline.version == '0.3.14'
@@ -77,7 +77,7 @@ def test_pipeline_filtered_get(async_minimum, fastapi_testclient):
assert response.status_code == status.HTTP_200_OK
pipes = response.json()
assert len(pipes) == 1, 'Only one pipeline matches the uri'
assert pipes[0] == second_pipeline
assert pipes[0] == second_pipeline.model_dump()


def test_get_known_pipeline(async_minimum, fastapi_testclient):
@@ -87,7 +87,7 @@ def test_get_known_pipeline(async_minimum, fastapi_testclient):
)
assert response.status_code == status.HTTP_200_OK

pipeline = Pipeline.parse_obj(response.json())
pipeline = Pipeline.model_validate(response.json())
assert pipeline, 'Response fits into the over-the-wire model'
assert pipeline.name == 'ptest one'
assert pipeline.version == '0.3.14'
@@ -109,13 +109,13 @@ def test_create_pipeline(async_minimum, fastapi_testclient):
)

response = http_create_pipeline(fastapi_testclient, desired_pipeline)
pipeline = Pipeline.parse_obj(response)
pipeline = Pipeline.model_validate(response)
assert pipeline == desired_pipeline, 'Got back what we put in'

# Create the same pipeline
response = fastapi_testclient.post(
'/pipelines',
json=desired_pipeline.dict(),
json=desired_pipeline.model_dump(),
follow_redirects=True,
headers=headers4power_user
)
@@ -136,7 +136,7 @@ def test_create_pipeline(async_minimum, fastapi_testclient):
'/pipelines', headers=headers
)
assert response.status_code == status.HTTP_200_OK
assert response.json()[1:] == [desired_pipeline, second_desired_pipeline]
assert response.json()[1:] == [desired_pipeline.model_dump(), second_desired_pipeline.model_dump()]

# Create a very poorly provenanced pipeline
third_desired_pipeline = Pipeline(
@@ -147,7 +147,7 @@ def test_create_pipeline(async_minimum, fastapi_testclient):

response = fastapi_testclient.post(
'/pipelines',
json=third_desired_pipeline.dict(),
json=third_desired_pipeline.model_dump(),
follow_redirects=True,
headers=headers4power_user
)
18 changes: 9 additions & 9 deletions tests/task_route_test.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from starlette import status

from npg.porch.models import Task, TaskStateEnum
from npg.porch.models import Task, TaskStateEnum, Pipeline

# Not testing get-all-tasks as this method will ultimately go

@@ -27,7 +27,7 @@ def test_task_creation(async_minimum, fastapi_testclient):

response = fastapi_testclient.post(
'tasks',
json=task_one.dict(),
json=task_one.model_dump(),
follow_redirects=True,
headers=headers4ptest_one
)
@@ -37,7 +37,7 @@ def test_task_creation(async_minimum, fastapi_testclient):
# Try again and expect to fail
response = fastapi_testclient.post(
'tasks',
json=task_one.dict(),
json=task_one.model_dump(),
follow_redirects=True,
headers=headers4ptest_one
)
@@ -55,7 +55,7 @@ def test_task_creation(async_minimum, fastapi_testclient):
# to have a valid token for a pipeline that does not exist.
response = fastapi_testclient.post(
'tasks',
json=task_two.dict(),
json=task_two.model_dump(),
follow_redirects=True,
headers=headers4ptest_one
)
@@ -76,7 +76,7 @@ def test_task_update(async_minimum, fastapi_testclient):
)
assert response.status_code == status.HTTP_200_OK

modified_task = Task.parse_obj(response.json())
modified_task = Task.model_validate(response.json())
assert modified_task == task

# Now invalidate the task by changing the signature
@@ -85,7 +85,7 @@ def test_task_update(async_minimum, fastapi_testclient):
}
response = fastapi_testclient.put(
'/tasks',
json=modified_task.dict(),
json=modified_task.model_dump(),
follow_redirects=True,
headers=headers4ptest_one
)
@@ -95,12 +95,12 @@ def test_task_update(async_minimum, fastapi_testclient):
# And change the reference pipeline to something wrong.
# This token is valid, but for a different pipeline. It is impossible
# to have a valid token for a pipeline that does not exist.
modified_task.pipeline = {
modified_task.pipeline = Pipeline.model_validate({
'name': 'ptest one thousand'
}
})
response = fastapi_testclient.put(
'/tasks',
json=modified_task.dict(),
json=modified_task.model_dump(),
follow_redirects=True,
headers=headers4ptest_one
)