diff --git a/docs/release-notes/1.59/DeleteFile.png b/docs/release-notes/1.59/DeleteFile.png new file mode 100644 index 0000000..c3ef39d Binary files /dev/null and b/docs/release-notes/1.59/DeleteFile.png differ diff --git a/docs/release-notes/1.59/INFO.png b/docs/release-notes/1.59/INFO.png new file mode 100644 index 0000000..fbb636e Binary files /dev/null and b/docs/release-notes/1.59/INFO.png differ diff --git a/docs/release-notes/1.59/Import.png b/docs/release-notes/1.59/Import.png new file mode 100644 index 0000000..db9a5d4 Binary files /dev/null and b/docs/release-notes/1.59/Import.png differ diff --git a/docs/release-notes/1.59/ManageData.png b/docs/release-notes/1.59/ManageData.png new file mode 100644 index 0000000..4f0bd3f Binary files /dev/null and b/docs/release-notes/1.59/ManageData.png differ diff --git a/docs/release-notes/1.59/Menu.png b/docs/release-notes/1.59/Menu.png new file mode 100644 index 0000000..a5edca2 Binary files /dev/null and b/docs/release-notes/1.59/Menu.png differ diff --git a/docs/release-notes/1.59/OutdatedEndpoints.png b/docs/release-notes/1.59/OutdatedEndpoints.png new file mode 100644 index 0000000..a127831 Binary files /dev/null and b/docs/release-notes/1.59/OutdatedEndpoints.png differ diff --git a/docs/release-notes/1.59/RefGenome.png b/docs/release-notes/1.59/RefGenome.png new file mode 100644 index 0000000..1511563 Binary files /dev/null and b/docs/release-notes/1.59/RefGenome.png differ diff --git a/docs/release-notes/1.59/exQuery.png b/docs/release-notes/1.59/exQuery.png new file mode 100644 index 0000000..9aafdb1 Binary files /dev/null and b/docs/release-notes/1.59/exQuery.png differ diff --git a/docs/release-notes/1.59/fileManager.png b/docs/release-notes/1.59/fileManager.png new file mode 100644 index 0000000..1e99d68 Binary files /dev/null and b/docs/release-notes/1.59/fileManager.png differ diff --git a/docs/release-notes/1.59/variantFeature.png b/docs/release-notes/1.59/variantFeature.png new file mode 100644 index 0000000..81955e2 Binary files /dev/null and b/docs/release-notes/1.59/variantFeature.png differ diff --git a/docs/release-notes/v1.50-v1.59.md b/docs/release-notes/v1.50-v1.59.md index 158f8cb..394e477 100644 --- a/docs/release-notes/v1.50-v1.59.md +++ b/docs/release-notes/v1.50-v1.59.md @@ -10,6 +10,224 @@ **Admins** - users who manage organization in ODM, its user and groups. +## 1.59 Open Data Manager 2024 Winter Release + +This release focuses on optimizing workflows with expanded filtering options for VCF files, streamlined reference genome management, and improved API-based file management, all of which increase flexibility and precision in data handling. + +### New Features + +#### [Researchers] + +* **Enhancements for VCF functionality: Filtering by the exact match of key values in INFO fields** + +From this release, ODM now enables precise filtering of variant data by exact matches in **INFO** fields, improving search specificity. Variant data can now be retrieved from studies primarily associated with different data types (e.g., transcriptomics) based on exact **INFO** field matches. +Support has been added for exact match searches within the variantInfo parameter. + +* ??? abstract "INFO fields considerations" + * info.key=value returns all variants with matching values for the specified INFO field. + * info.key\!=value excludes the variants with the value from the search results. + * Multiple values can be provided for a key, separated by a comma. Equal to OR operator. + +* ??? danger "Affected endpoints (for both as-curator and as-users) that include vxQuery parameter" + * GET /api/v1/as-curator/omics/expression/data + * GET/api/v1/as-curator/omics/expression/group + * GET /api/v1/as-curator/omics/flow-cytometry/data + * GET /api/v1/as-curator/omics/flow-cytometry/group + * GET /api/v1/as-curator/omics/samples + * GET /api/v1/as-curator/omics/variant/data + * GET /api/v1/as-curator/omics/variant/group + * GET /api/v1/as-curator/omics/variant/streamed-data + * GET /api/v1/as-user/omics/expression/data + * GET /api/v1/as-user/omics/expression/group + * GET /api/v1/as-user/omics/flow-cytometry/data + * GET /api/v1/as-user/omics/flow-cytometry/group + * GET /api/v1/as-user/omics/samples + * GET /api/v1/as-user/omics/variant/data + * GET /api/v1/as-user/omics/variant/group + * GET /api/v1/as-user/omics/variant/streamed-data + + ![INFO.png](1.59/INFO.png) +
The parameter INFO allows retrieval of data by exact match and is included in the GET /api/v1/as-curator/omics/ and GET /api/v1/as-user/omics/ endpoints
+ +* **Enhancements for VCF functionality: Adding exQuery to variant streaming endpoint** + +With this release, ODM expands its GET endpoint capabilities to facilitate locating objects connected to expression data originally uploaded in TSV or GCT formats within VCF variant studies. + +* ??? danger "Affected endpoints (for both as-curator and as-users) that include exQuery parameter" + * GET /api/v1/as-curator/omics/variant/streamed-data + * GET /api/v1/as-user/omics/variant/streamed-data + +![exQuery.png](1.59/exQuery.png) +
The GET /api/v1/as-user/omics/variant/streamed-data and GET /api/v1/as-curator/omics/variant/streamed-data endpoints now support streaming expression data
+ +* **Enhancements for VCF functionality: Filtering variant data based on genes: variantFeature** + +From release 1.59, ODM allows users to retrieve all gene variants within a specified gene region using the **variantFeature** parameter, facilitating the discovery of data within targeted genomic locations. + +* ??? abstract "variantFeature considerations" + * Multiple values are provided as a string array. + * Gene(s) should be supplied (e.g. feature=TP53 ) + * The system searches for **exact matches** for the supplied gene(s) in the reference genomes. + * Each gene is resolved to a region for each Reference genome in the system. + +* ??? danger "Affected endpoints (for both as-curator and as-users) that include the variantFeature parameter" + * GET /api/v1/as-user/omics/expression/data + * GET /api/v1/as-user/omics/expression/group + * GET /api/v1/as-user/omics/flow-cytometry/data + * GET /api/v1/as-user/omics/flow-cytometry/group + * GET /api/v1/as-user/omics/samples + * GET /api/v1/as-user/omics/variant/data + * GET /api/v1/as-user/omics/variant/group + * GET /api/v1/as-user/omics/variant/streamed-data + * GET /api/v1/as-curator/omics/expression/data + * GET /api/v1/as-curator/omics/expression/group + * GET /api/v1/as-curator/omics/flow-cytometry/data + * GET /api/v1/as-curator/omics/flow-cytometry/group + * GET /api/v1/as-curator/omics/samples + * GET /api/v1/as-curator/omics/variant/data + * GET /api/v1/as-curator/omics/variant/group + * GET /api/v1/as-curator/omics/variant/streamed-data + +![variantFeature.png](1.59/variantFeature.png) +
The feature search capability using variantFeature is added, aiding targeted gene searches
+ +#### [Curators] + +* **New endpoints for importing, and retrieving Reference Genomes** + +Users can now import different versions and builds of reference genomes. This feature allows users to significantly enhance flexibility, accuracy, and relevance in variant analysis. + +Uploading reference genomes was already supported in ODM but only via .py script. From this release (1.59), uploading of reference genomes is now available via a new API endpoint: POST api/v1/reference-genomes, part of the new definition **referenceGenome**. + +* ??? abstract "Important considerations of the Reference Genome endpoints" + * Only users that are part of the Curator Group can execute the endpoint api/v1/reference-genomes to upload and query reference genomes + * Each uploaded reference genome should be unique according to a combination of mandatory fields specified in a request body + * Reference genome requests should include the following parameters: + + * | Field Name | Required | Description | + | :---- | :---- | :---- | + | annotationUrl | No | link to the non-vertebrae organism or any custom reference genome in .gtf format, archived in \`.gz\` format. Note: if any of the parameters below do not correspond to the available Ensembl data (for example, organism name or assembly is changed), the link must be provided. | + | species | Yes | Scientific name with the first uppercase letter and underscore as a delimiter (e.g. Homo\_sapiens). | + | assembly | Yes | Major version (e.g. \`GRCh38\`) | + | release | Yes | Ensembl reference genome minor version (for example, \`75\` or \`109\`) | + | name | No | Customized reference genome title that is used for linkage variant files. By default is combined from species, assembly, and release parameters: \`{species} reference genome {assembly}.{release}\` | + +* ??? danger "New Endpoints to retrieve and import Reference Genomes" + * GET /api/v1/reference-genomes + * POST /api/v1/reference-genomes + * GET /api/v1/reference-genomes/{id} + + ![RefGenome.png](1.59/RefGenome.png) +
The definition referenceGenome has been created. This definition includes endpoints to retrieve and upload reference genomes
+ +* **Improved Support for Files (formerly known as Attachments): Add POST (job) endpoint to upload Files** + +On this release, the endpoint POST /api/v1/jobs/import/file, which allows the importing of Files (attachments) via API, has been incorporated to import file jobs efficiently. + +* ??? abstract "Important considerations" + * Upon calling the endpoint + * An attachment object is created in ODM. + * The reference between the specified study and the created object is created + * Request body: + * The field “source” is optional (if no source is provided, the default is used) + * Allowed values are "S3", "HTTP" and “LOCAL”. + * The link to the file, “dataLink”, is mandatory + * The ID (studyAccession) of the study to which the file will be attached must be provided + +* ??? danger "Endpoint to import Files" + * POST /api/v1/jobs/import/file + +![Import.png](1.59/Import.png) +
The newly introduced POST /api/v1/jobs/import/file endpoint, users can efficiently import file attachments via APIs
+ +#### [Admins] + +* **Delete Reference Genome via API** + +In addition to the previously mentioned enhancements to create and query reference genomes, the release 1.59 of ODM also includes the ability to delete reference genomes that are no longer needed. This feature aims to provide control over data relevance, accuracy, and compliance, ensuring the tool’s infrastructure stays up-to-date and aligned with project requirements + +The endpoint to delete reference genomes is now part of the endpoint DELETE /api/v1/manage-data/data. + +* ??? abstract "Important considerations to delete Reference Genomes" + * Only users assigned as Administrators can execute the DELETE /api/v1/manage-data/data endpoint to delete data, including reference genomes + * If the Reference Genome is linked to the VCF group, then the reference genome should be unlinked. As a reference genome is required for filtering by genes, variantFeature parameter is expected to not work for the VCF group without a reference genome + * In case the selected object has more than one version, the endpoint deletes all of them + * If a Reference Genome is linked to any other data and the Reference Genome is deleted, the endpoint will also delete the linked data associated with it then searching for variant by variantFeature (genes) won't be available for linked VCF groups. + +* ??? danger "Endpoint to delete Reference Genomes" + * DELETE /api/v1/manage-data/data + +![ManageData.png](1.59/ManageData.png) +
The user can now remove reference genomes if no longer needed. The endpoint DELETE /api/v1/manage-data/data now also includes the option to delete Reference Genome (accession number is required)
+ +* **Improved Support for Files (formerly known as Attachments): Delete Files via API** + +Admins can now manage imported files through the DELETE /api/v1/manage-data/data endpoint (**manageData**), improving the control over file lifecycle management. + +* ??? abstract "Important considerations" + * Manage organization AND Access all data permissions are required. + * The whole stack of versions of the mentioned attachment is to be deleted. + +* ??? danger "Endpoints to delete files" + * DELETE /api/v1/manage-data/data + +![DeleteFile.png](1.59/DeleteFile.png) +
The endpoint DELETE /api/v1/manage-data/data has been enhanced, allowing the deletion of (attached) files from ODM
+ +#### [All Users] + +* **Updates in the Study Menu interface: Menu accessible on both sides of the menu** + +The Study Menu now offers enhanced access options, making it easier to navigate ODM. Users can access the menu from both the top and right sides of the metadata editor, streamlining data management workflows + +![Menu.png](1.59/Menu.png) +
Updated view of the metadata editor. The menu can easily be accessed from the top part of the study, or by clicking on the right side, next to the Export button
+ +* **Updates in the Study Menu interface: File Manager** + +The dashboard now includes a dedicated File Manager accessible from the sidebar menu. This feature lets users view and filter files by type, such as Reference Genome, Flow Cytometry group, Library, and Sample Metadata. + +* ??? abstract "Important considerations" + * Filters will be displayed in ascending order + * If no filters are applied, the types selected as default will be Reference Genomes and Dictionaries. + +![fileManager.png](1.59/fileManager.png) +
Access the File Manager section by clicking on the top left menu
+ +### Fixed Issues + +* **Updates on DELETE and POST Endpoints DELETE endpoints: Eliminating sync POST and DELETE endpoints** + +In previous releases, posting and deleting data within ODM was available to all curator users. Given the importance of these functions, access will now be limited to users with specific permissions. Release 1.59 refines the ODM APIs by removing outdated endpoints, making the API functionality more reliable, streamlined, and user-friendly + +* ??? abstract "Important considerations" + * To import data, users need to use the Job endpoints (e.g. POST /api/v1/jobs/import/variant) + * To delete data, users need to access the Manage Data endpoints (DELETE /api/v1/manage-data/data) + * Deletion of data is exclusively available to users classified as Administrators + +* ??? danger "Endpoints removed" + * POST /api/v1/as-curator/expressions/group/{id} + * DELETE /api/v1/as-curator/expressions/gct + * DELETE /api/v1/as-curator/variants/group/{id} + * POST /api/v1/as-curator/variants/vcf + * DELETE /api/v1/as-curator/flow-cytometries/group/{id} + * POST /api/v1/as-curator/flow-cytometries/facs + * DELETE /api/v1/as-curator/preparations/{id} + * DELETE /api/v1/as-curator/preparations/group/{id} + * POST /api/v1/as-curator/preparations + * DELETE /api/v1/as-curator/libraries/{id} + * DELETE /api/v1/as-curator/libraries/group/{id} + * POST /api/v1/as-curator/libraries + * DELETE /api/v1/as-curator/samples/{id} + * POST /api/v1/as-curator/samples + * DELETE /api/v1/as-curator/studies/{id} + * POST /api/v1/as-curator/studies + +![OutdatedEndpoints.png](1.59/OutdatedEndpoints.png) +
Outdated and less effective endpoints to POST and DELETE (e.g. Variant SPoT as Curator endpoints) have been assigned to Job (POST) and Manage Data (DELETE) definitions
+ +Genestack ODM Version 1.59 is designed to enhance user experience, providing comprehensive tools for managing omics data with precision and flexibility. These updates ensure that customers have access to improved data management capabilities, enabling more effective data integration and retrieval processes. + ## 1.58 Open Data Manager 2024 Summer Release: VCF functionality ### New Features @@ -116,6 +334,7 @@ Please review the documentation, as these changes may impact your integrations. 1. **Tabular Data Endpoint Changes** * ??? abstract "Introduced changes to Tabular Data" + | Introduced change | Changes Description | |:----------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Deprecating old response format | The old response format for tabular data will be deprecated. The **"multi_value"** response format (introduced in July 2023) will be the default option. It allows retrieval of data with one or more features and suits any type of tabular data, including **.gct** files. | @@ -125,6 +344,7 @@ Please review the documentation, as these changes may impact your integrations. ![RN](1.58/1-58-2.png) * ??? danger "Affected endpoints (for both as-curator and as-users)" + | Endpoint Group | Endpoints Affected | |:---------------------|:----------------------------------------------------------------------------------------------------------------------------------------| | expressionUser | GET `/api/v1/as-user/expressions` GET `/api/v1/as-user/expressions/{id}` | @@ -135,6 +355,7 @@ Please review the documentation, as these changes may impact your integrations. 2. **Variants Endpoint Changes** * ??? abstract "Introduced changes to Variants" + | Introduced change | Changes Description | |:----------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Introducing **“variant”** section | A new section **“variant”** contains all fields related to a specific variation from the original VCF file: CHROM, POS, ID, REF, ALT, QUAL, FILTER, INFO. | @@ -143,6 +364,7 @@ Please review the documentation, as these changes may impact your integrations. ![RN](1.58/1-58-3.png) * ??? danger "Affected endpoints (for both as-curator and as-users)" + | Endpoint Group | Endpoints Affected | |:---------------------|:----------------------------------------------------------------------------------------------------------------------------------------| | variantUser | GET `/api/v1/as-user/variants` GET `/api/v1/as-user/variants/{id}` | @@ -153,6 +375,7 @@ Please review the documentation, as these changes may impact your integrations. 3. **Flow Cytometry Endpoint Changes** * ??? abstract "Introduced changes to Flow Cytometry" + | Introduced change | Changes Description | |:----------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Introducing **“feature”** and **“value”** sections | Flow-cytometry data response now has section **“feature”** including **"readoutType"**, **"cellPopulation"** and **"marker"** fields. **“Value”** section contains the expression field which is renamed to **“value”**.| @@ -162,6 +385,7 @@ Please review the documentation, as these changes may impact your integrations. ![RN](1.58/1-58-4.png) * ??? danger "Affected endpoints (for both as-curator and as-users)" + | Endpoint Group | Endpoints Affected | |:---------------------|:----------------------------------------------------------------------------------------------------------------------------------------| | flowCytometryUser | GET `/api/v1/as-user/flow-cytometries` GET `/api/v1/as-user/flow-cytometries/{id}` |