Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Adds custom project's metadata in the webserver API #4421

Merged
merged 53 commits into from
Jul 7, 2023
Merged
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
3d68a7a
projects metadata table
pcrespov Jun 26, 2023
0c99a77
projects to jobs table
pcrespov Jun 26, 2023
a44fed1
WIP
pcrespov Jun 26, 2023
d59b57a
adds jobs_metadata to pg tables and renames table to metadata_jobs
pcrespov Jun 27, 2023
f064d29
all tables together
pcrespov Jun 27, 2023
0dec772
doc
pcrespov Jun 27, 2023
56b0b13
doc
pcrespov Jun 27, 2023
0c66bff
drafts tests
pcrespov Jun 27, 2023
727a7ab
migration projects-metadata
pcrespov Jun 27, 2023
e06046a
migration projects-jobs-metadata
pcrespov Jun 27, 2023
1c468ee
updates read
pcrespov Jun 27, 2023
9a7a388
drafts job-metadata repo
pcrespov Jun 28, 2023
1bf09b3
drafts API
pcrespov Jun 28, 2023
6cd0a39
minor
pcrespov Jun 29, 2023
014f3bd
layout apis
pcrespov Jul 4, 2023
9ca523b
cleanup
pcrespov Jul 4, 2023
a960d7f
updates code
pcrespov Jul 4, 2023
8c50887
rm migration
pcrespov Jul 4, 2023
7dab446
fixes erd
pcrespov Jul 4, 2023
0a25f51
only one table
pcrespov Jul 4, 2023
98e2260
common models
pcrespov Jul 4, 2023
9a55e4e
buildling controler and service
pcrespov Jul 5, 2023
903bbf1
building service and model
pcrespov Jul 5, 2023
bc74f95
migrations
pcrespov Jul 5, 2023
f7266da
project metadata repo ready
pcrespov Jul 5, 2023
eb49d72
db
pcrespov Jul 6, 2023
7769d4b
access rights
pcrespov Jul 6, 2023
b9397c6
error handling
pcrespov Jul 6, 2023
6e8af4b
rename error
pcrespov Jul 6, 2023
3dd1f29
fixes test
pcrespov Jul 6, 2023
19dd826
oas
pcrespov Jul 6, 2023
00f1f38
services/webserver api version: 0.23.0 → 0.24.0
pcrespov Jul 6, 2023
d13cbc0
updates OAS
pcrespov Jul 6, 2023
5016853
fixes
pcrespov Jul 6, 2023
f2d8c2c
fixes deletion in test
pcrespov Jul 6, 2023
7cfdf8a
fix merge
pcrespov Jul 6, 2023
8688df1
minor
pcrespov Jul 6, 2023
a1c71f7
cleanup doc
pcrespov Jul 6, 2023
fb387e7
minor
pcrespov Jul 6, 2023
bb11dd0
cleanup
pcrespov Jul 6, 2023
94944ef
minor fix from last PR
pcrespov Jul 6, 2023
a3cafe7
minor
pcrespov Jul 6, 2023
ddb2221
Revert "minor fix from last PR"
pcrespov Jul 6, 2023
f96a7db
fixes route
pcrespov Jul 7, 2023
19c18d8
@sandereg review: todos in disguise
pcrespov Jul 7, 2023
ce0c5a3
@sanderegg review: put->patch
pcrespov Jul 7, 2023
c489d20
@sanderegg review: renamed all as metadata
pcrespov Jul 7, 2023
66ece81
@sandergg review: renamed functions
pcrespov Jul 7, 2023
79daf7d
OAS
pcrespov Jul 7, 2023
794e167
fix
pcrespov Jul 7, 2023
fe2ecea
cleanup
pcrespov Jul 7, 2023
efca20e
fix
pcrespov Jul 7, 2023
9d263d8
fixes
pcrespov Jul 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions api/specs/webserver/openapi-projects-metadata.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
paths:
/projects/{project_id}/metadata/custom:
get:
tags:
- project
summary: Get Project Custom Metadata
operationId: get_project_custom_metadata
parameters:
- required: true
schema:
type: string
format: uuid
title: Project Id
name: project_id
in: path
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/Envelope_ProjectCustomMetadataGet_'
put:
tags:
- project
summary: Replace Project Custom Metadata
operationId: replace_project_custom_metadata
parameters:
- required: true
schema:
type: string
format: uuid
title: Project Id
name: project_id
in: path
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/ProjectCustomMetadataReplace'
required: true
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/Envelope_ProjectCustomMetadataGet_'
components:
schemas:
Envelope_ProjectCustomMetadataGet_:
properties:
data:
$ref: '#/components/schemas/ProjectCustomMetadataGet'
error:
title: Error
type: object
title: Envelope[ProjectCustomMetadataGet]
ProjectCustomMetadataGet:
properties:
projectUuid:
type: string
format: uuid
title: Projectuuid
metadata:
additionalProperties:
anyOf:
- type: boolean
- type: integer
- type: number
- type: string
type: object
title: Metadata
description: Custom key-value map
type: object
required:
- projectUuid
title: ProjectCustomMetadataGet
ProjectCustomMetadataReplace:
properties:
metadata:
additionalProperties:
anyOf:
- type: boolean
- type: integer
- type: number
- type: string
type: object
title: Metadata
type: object
required:
- metadata
title: ProjectCustomMetadataReplace
5 changes: 4 additions & 1 deletion api/specs/webserver/openapi.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
openapi: 3.0.0
info:
title: "osparc-simcore web API"
version: 0.23.0
version: 0.24.0
description: "API designed for the front-end app"
contact:
name: IT'IS Foundation
Expand Down Expand Up @@ -256,6 +256,9 @@ paths:
/projects/{project_id}/outputs:
$ref: "./openapi-projects-ports.yaml#/paths/~1projects~1{project_id}~1outputs"

/projects/{project_id}/metadata/custom:
$ref: "./openapi-projects-metadata.yaml#/paths/~1projects~1{project_id}~1metadata~1custom"

/projects/{project_id}/metadata/ports:
$ref: "./openapi-projects-ports.yaml#/paths/~1projects~1{project_id}~1metadata~1ports"

Expand Down
63 changes: 63 additions & 0 deletions api/specs/webserver/scripts/openapi_projects_metadata.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
""" Helper script to automatically generate OAS

This OAS are the source of truth
"""

# pylint: disable=redefined-outer-name
# pylint: disable=unused-argument
# pylint: disable=unused-variable
# pylint: disable=too-many-arguments


from enum import Enum
from typing import Annotated

from _common import CURRENT_DIR, create_openapi_specs
from fastapi import Depends, FastAPI, status
from models_library.api_schemas_webserver.projects_metadata import (
ProjectMetadataGet,
ProjectMetadataUpdate,
)
from models_library.generics import Envelope
from simcore_service_webserver.projects._metadata_handlers import (
ProjectMetadataGet,
ProjectPathParams,
)

app = FastAPI(redoc_url=None)

TAGS: list[str | Enum] = ["project"]


#
# API entrypoints
#


@app.get(
"/projects/{project_id}/metadata",
response_model=Envelope[ProjectMetadataGet],
tags=TAGS,
operation_id="get_project_metadata",
status_code=status.HTTP_200_OK,
)
async def get_project_metadata(_params: Annotated[ProjectPathParams, Depends()]):
...


@app.patch(
"/projects/{project_id}/metadata",
response_model=Envelope[ProjectMetadataGet],
tags=TAGS,
operation_id="update_project_metadata",
status_code=status.HTTP_200_OK,
)
async def update_project_metadata(
_params: Annotated[ProjectPathParams, Depends()], _body: ProjectMetadataUpdate
):
...


if __name__ == "__main__":

create_openapi_specs(app, CURRENT_DIR.parent / "openapi-projects-metadata.yaml")
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
from typing import TypeAlias

from pydantic import Field, StrictBool, StrictFloat, StrictInt

from ..projects import ProjectID
from ._base import InputSchema, OutputSchema

# Limits metadata values
MetaValueType: TypeAlias = StrictBool | StrictInt | StrictFloat | str
MetadataDict: TypeAlias = dict[str, MetaValueType]


class ProjectMetadataGet(OutputSchema):
project_uuid: ProjectID
custom: MetadataDict = Field(
default_factory=dict, description="Custom key-value map"
)


class ProjectMetadataUpdate(InputSchema):
custom: MetadataDict
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
"""new projects_metadata table

Revision ID: f3285aff5e84
Revises: 58b24613c3f7
Create Date: 2023-07-05 15:06:56.003418+00:00

"""
from typing import Final

import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql

# revision identifiers, used by Alembic.
revision = "f3285aff5e84"
down_revision = "58b24613c3f7"
branch_labels = None
depends_on = None


# auto-update modified
# TRIGGERS ------------------------
_TABLE_NAME: Final[str] = "projects_metadata"
_TRIGGER_NAME: Final[str] = "trigger_auto_update" # NOTE: scoped on table
_PROCEDURE_NAME: Final[
str
] = f"{_TABLE_NAME}_auto_update_modified()" # NOTE: scoped on database
modified_timestamp_trigger = sa.DDL(
f"""
DROP TRIGGER IF EXISTS {_TRIGGER_NAME} on {_TABLE_NAME};
CREATE TRIGGER {_TRIGGER_NAME}
BEFORE INSERT OR UPDATE ON {_TABLE_NAME}
FOR EACH ROW EXECUTE PROCEDURE {_PROCEDURE_NAME};
"""
)

# PROCEDURES ------------------------
update_modified_timestamp_procedure = sa.DDL(
f"""
CREATE OR REPLACE FUNCTION {_PROCEDURE_NAME}
RETURNS TRIGGER AS $$
BEGIN
NEW.modified := current_timestamp;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
"""
)


def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table(
"projects_metadata",
sa.Column("project_uuid", sa.String(), nullable=False),
sa.Column(
"custom",
postgresql.JSONB(astext_type=sa.Text()),
server_default=sa.text("'{}'::jsonb"),
nullable=False,
),
sa.Column(
"created",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.Column(
"modified",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.ForeignKeyConstraint(
["project_uuid"],
["projects.uuid"],
name="fk_projects_metadata_project_uuid",
onupdate="CASCADE",
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("project_uuid"),
)
# ### end Alembic commands ###

# custom
op.execute(update_modified_timestamp_procedure)
op.execute(modified_timestamp_trigger)


def downgrade():
# custom
op.execute(f"DROP TRIGGER IF EXISTS {_TRIGGER_NAME} on {_TABLE_NAME};")
op.execute(f"DROP FUNCTION {_PROCEDURE_NAME};")

# ### commands auto generated by Alembic - please adjust! ###
op.drop_table("projects_metadata")
# ### end Alembic commands ###
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
"""
These tables were designed to be controled by projects-plugin in
the webserver's service
"""

import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import JSONB

from ._common import (
column_created_datetime,
column_modified_datetime,
register_modified_datetime_auto_update_trigger,
)
from .base import metadata
from .projects import projects

projects_metadata = sa.Table(
"projects_metadata",
#
# Keeps "third-party" metadata attached to a project
#
# These SHOULD NOT be actual properties of the project (e.g. uuid, name etc)
# but rather information attached by third-parties that "decorate" or qualify
# a project resource
#
# Things like 'stars', 'quality', 'classifiers', 'dev', etc (or any kind of stats)
# should be moved here
#
metadata,
sa.Column(
"project_uuid",
sa.String,
sa.ForeignKey(
projects.c.uuid,
onupdate="CASCADE",
ondelete="CASCADE",
name="fk_projects_metadata_project_uuid",
),
nullable=False,
primary_key=True,
doc="The project unique identifier is also used to identify the associated job",
),
sa.Column(
"custom",
JSONB,
nullable=False,
server_default=sa.text("'{}'::jsonb"),
doc="Reserved for the user to store custom metadata",
),
# TIME STAMPS ----ß
column_created_datetime(timezone=True),
column_modified_datetime(timezone=True),
sa.PrimaryKeyConstraint("project_uuid"),
)


register_modified_datetime_auto_update_trigger(projects_metadata)
Loading