Feature/project ownership

[teemukataja]

Description

Folders and templates were originally owned by users, now they will be owned by projects that users belong to, allowing multiple users to work on the same folders and templates at the same time.

Related issues

Fixes #282
Frontend issue CSCfi/metadata-submitter-frontend#625

Type of change

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update (API changes)

Changes Made

• deprecated folders and templates keys from user
• added new key projects to user
• added new key projectId to folder and template collections
• added new project collection
• new mandatory userinfo value from AAI sdSubmitProjects
• if user has no sdSubmitProjects they are redirected to /noproject
• new mandatory query parameter projectId for GET /folders
• new endpoint GET /templates?projectId=... that replaces GET /users/current "templates":[...]
• projectId is not mandatory when requesting specific resource, e.g. GET /folders/{folderId}, the permissions are read from session
• new mandatory key projectId for POST /folders and /templates
• deprecated PATCH /user
• new request body keys index and tags in PATCH /templates/schema/templateId which are the same values as previously used in the deprecated PATCH /user

Other Points

• user project affiliations are re-validated on every login, if project affiliations have changed, user data is updated
• breaking change that requires fresh database, because "project" is new information that did not exist before, and it can't be migrated to existing user-owned hierarchy due to missing relationship information

Testing

• Unit Tests
• Integration Tests
• Manual Testing

[teemukataja]

The previous commit 00cec8f

1. updated integration tests,
2. they also showed some bugs in the code which were fixed in the same commit
3. also removed some deprecated features, and will continue to prune through deprecated features in next commit

[teemukataja]

Added integration test case for GET /templates. I tried to copy folder unit tests (query_folders) for templates (query_templates_by_project) (because they are almost identical), but because the query loops over collections, it raises a StopAsyncIteration, and I'm not sure how to deal with that.

Would this be usful:

try:
    ....    
except StopIteration:
    pass

metadata-submitter/tests/integration/run_tests.py

Line 549 in 135af96

except StopIteration:

I thought about this also initially and tested it, but it still errored, and I think it's also not the correct way to approach this, because it would stop the test

This is the function
https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L308-L365
It queries multiple collections

It's based on this
https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L812-L854

[blankdots]

It's based on this https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L812-L854

I think this: https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L395 might be more appropriate as you want object query not folder query and maybe pay attention to: https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/conf/conf.py#L113

[blankdots]

wait you want query_templates_by_project not to filter the templates content ... 🤔 maybe it will be easier if you put them all in one collection templates and add the type for each document in the collect e.g. study, sample etc. I am not sure we should invest the effort in making cross collection queries at this point for templates

[blankdots]

however if you really want JOIN like from SQL i think https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/ is what you need.

[teemukataja]

It's based on this https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L812-L854

I think this: https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L395 might be more appropriate as you want object query not folder query and maybe pay attention to: https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/conf/conf.py#L113

That is also the same, it uses self.db_service.do_aggregate() which is all this function needs

For the query constructor, I don't think that is needed for this use case (seems overly complicated), because we only want to read 2 values (accessionId and title)

wait you want query_templates_by_project not to filter the templates content ... 🤔 maybe it will be easier if you put them all in one collection templates and add the type for each document in the collect e.g. study, sample etc. I am not sure we should invest the effort in making cross collection queries at this point for templates

I didn't want to create new functionality, but re-use existing data to keep it simple. This is an endpoint that replaces GET /users/current's "templates": [...] key

however if you really want JOIN like from SQL i think https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/ is what you need.

I will look into this if it makes it possible to request the data in one query

[teemukataja]

however if you really want JOIN like from SQL i think https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/ is what you need.

I will look into this if it makes it possible to request the data in one query

Looks like lookup is used for joining data together into one object from separate collections, but we want to get a list of all objects from multiple collections that have the same project id

[teemukataja]

Returned old object assigning code into use with templates-in-project (as opposed to templates-in-user).