Merge pull request #225 from CSCfi/develop

release 0.10.0

.github/workflows/int.yml

@@ -30,15 +30,17 @@ jobs:
         sleep 30
   
     - name: Run Integration test
-      run: python tests/integration/run_tests.py
+      run: |
+        python tests/integration/clean_db.py
+        python tests/integration/run_tests.py
 
     - name: Collect logs from docker
       if: ${{ failure() }}
       run: docker-compose logs --no-color -t > tests/dockerlogs || true
 
     - name: Persist log files
       if: ${{ failure() }}
-      uses: actions/upload-artifact@v1
+      uses: actions/upload-artifact@v2.2.4
       with:
         name: test_debugging_help
         path: tests
@@ -57,15 +59,17 @@ jobs:
         sleep 30
 
     - name: Run Integration test
-      run: python tests/integration/run_tests.py
+      run: |
+        python tests/integration/clean_db.py --tls
+        python tests/integration/run_tests.py
 
     - name: Collect logs from docker
       if: ${{ failure() }}
       run: docker-compose logs --no-color -t > tests/dockerlogs || true
 
     - name: Persist log files
       if: ${{ failure() }}
-      uses: actions/upload-artifact@v1
+      uses: actions/upload-artifact@v2.2.4
       with:
         name: test_debugging_help
         path: tests

.gitignore

@@ -118,3 +118,4 @@ venv.bak/
 .DS_Store
 
 metadata_backend/frontend/*
+config/*

Dockerfile

@@ -9,7 +9,7 @@ ARG BRANCH=master
 RUN git clone -b ${BRANCH} https://github.com/CSCfi/metadata-submitter-frontend.git
 
 WORKDIR /metadata-submitter-frontend
-RUN npm install -g npm@7.6.0 \
+RUN npm install -g npm@7.20.0 \
     && npx --quiet pinst --disable \
     && npm install --production \
     && npm run build --production

docs/conf.py

@@ -11,7 +11,7 @@
 author = "CSC Developers"
 
 # The full version, including alpha/beta/rc tags
-release = "0.9.0"
+release = "0.10.0"
 
 
 # -- General configuration ---------------------------------------------------

docs/index.rst

@@ -16,6 +16,9 @@ The application's inteded use is with `NeIC SDA (Sensitive Data Archive) <https:
 consists out of the box includes the `ENA (European Nucleotide Archive) <https://ena-docs.readthedocs.io>`_ metadata model,
 model which is used also by the  `European Genome-phenome Archive (EGA) <https://ega-archive.org/>`_.
 
+.. image:: /_static/metadata-app.svg
+   :alt: Metadata Submitter Architecture and Metadata Overview
+
 Out of the box the ``metadata-submitter`` offers:
 
 * flexible REST API for working with metadata; 
@@ -34,6 +37,7 @@ see :ref:`validate`.
 
    Backend <submitter>
    Frontend <frontend>
+   Metadata <metadata>
    Deployment <deploy>
    Testing  <test>
    Validator CLI Tool <validator>

docs/metadata.rst

@@ -0,0 +1,60 @@
+Metadata Model
+==============
+
+ENA Metadata Model
+------------------
+
+The object schemas that are used for rendering the forms and validating the information submitted
+to the application are based on the `ENA (European Nucleotide Archive) Metadata Model <https://ena-docs.readthedocs.io/en/latest/submit/general-guide/metadata.html>`_.
+
+The source XML schemas are from `ENA Sequence Github repository <https://github.com/enasequence/schema/tree/master/src/main/resources/uk/ac/ebi/ena/sra/schema>`_.
+The XML schemas are converted to JSON Schemas so that they can be both validate the submitted data as well as be rendered as forms in the User Interface.
+For this reason the translation from XML Schema to JSON schema is not a 1-1 mapping, but an interpretation.
+
+.. image:: /_static/metadata-model.svg
+   :alt: Metadata ENA Model
+
+The ENA model consists of the following objects:
+
+- ``Study``: A study groups together data submitted to the archive. A study accession is typically used when citing data submitted to ENA. Note that all associated data and other objects are made public when the study is released.
+- ``Project``: A project groups together data submitted to the archive. A project accession is typically used when citing data submitted to ENA. Note that all associated data and other objects are made public when the project is released.
+- ``Sample``: A sample contains information about the sequenced source material. Samples are typically associated with checklists, which define the fields used to annotate the samples.
+- ``Experiment``: An experiment contain information about a sequencing experiment including library and instrument details.
+- ``Run``: A run is part of an experiment and refers to data files containing sequence reads.
+- ``Analysis``: An analysis contains secondary analysis results derived from sequence reads (e.g. a genome assembly).
+- ``DAC``: An European Genome-phenome Archive (EGA) data access committee (DAC) is required for authorized access submissions.
+- ``Policy``: An European Genome-phenome Archive (EGA) data access policy is required for authorized access submissions.
+- ``Dataset``: An European Genome-phenome Archive (EGA) data set is required for authorized access submissions.
+
+Relationships between objects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each of the objects are connected between each other by references, usually in the form of an ``accessionId``.
+Some of the relationships are illustrated in the Metdata ENA Model figure, however in more detail they are connected as follows:
+
+- ``Study`` - usually other objects point to it, as it represents one of the main objects of a ``Submission``;
+- ``Analysis`` - contains references to:
+
+    - parent ``Study`` (not mandatory);
+    - zero or more references to objects of type: ``Sample``, ``Experiment``, ``Run``;
+
+- ``Experiment`` - contains references to exactly one parent ``Study``. It can also contain a reference to ``Sample`` as an individual or a Pool;
+- ``Run`` - contains reference to exactly one parent ``Experiment``;
+- ``Policy`` - contains reference to exactly one parent ``DAC``;
+- ``Dataset`` - contains references to:
+
+    - exactly one ``Policy``;
+    - zero or more references to objects of type: ``Analysis`` and ``Run``.
+
+
+EGA/ENA Metadata submission Guides
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Related guides for metadata submission:
+
+- EGA Metadata guides:
+
+    - `Submitting array based metadata <https://ega-archive.org/submission/array_based/metadata>`_
+    - `Submitting sequence and phenotype data <https://ega-archive.org/submission/tools/submitter-portal>`_
+
+- ENA Data Submission `general Guide  <https://ena-docs.readthedocs.io/en/latest/submit/general-guide.html>`_

docs/specification.yml

@@ -754,6 +754,22 @@ paths:
       tags:
         - Query
       summary: List of folders available for the user.
+      parameters:
+        - in: query
+          name: page
+          schema:
+            type: string
+          description: Page number
+        - in: query
+          name: per_page
+          schema:
+            type: string
+          description: Results per page
+        - in: query
+          name: published
+          schema:
+            type: string
+          description: Return folders based on the folder published value
       responses:
         200:
           description: OK
@@ -974,6 +990,21 @@ paths:
           schema:
             type: string
           required: true
+        - in: query
+          name: page
+          schema:
+            type: string
+          description: Page number
+        - in: query
+          name: per_page
+          schema:
+            type: string
+          description: Results per page
+        - in: query
+          name: items
+          schema:
+            type: string
+          description: Item type name
       responses:
         200:
           description: OK
@@ -1324,9 +1355,12 @@ components:
     FolderList:
       type: object
       required:
+        - page
         - folders
       additionalProperties: false
       properties:
+        page:
+          $ref: "#/components/schemas/Pagination"
         folders:
           type: array
           items: