Metabolic pathways visualisation

[mzueva]

Background

Metabolic pathways are series of chemical reactions that start with a substrate and finish with an end product.
These pathways are being displayed as certain kind of infographics, often called "maps".
Maps can be as public ones received from different sources like BioCyc database and as well as internal, produced by pathway modelling software.
NGB shall support visualization of both types of pathway maps.

Approach

• Pathways panel
  • Databases sub-tab
  • Pathways sub-tab
• Internal pathways
• External database integration
• Pathways annotation
  • Annotation file
  • Manual colorization
• Additional details

Pathways panel

A new Pathways panel shall be added.
This panel shall be available:

• from the VIEWS item of the main menu - to open an empty Pathways panel, e.g.:
  
• from the context menu at the "Browser" panel when clicks the certain gene/feature - to open the Pathways panel with search results by the feature name, e.g.:
  
• from the context menu at the "Genes" panel when clicks the certain gene/feature - to open the Pathways panel with search results by the feature name, e.g.:
  

In all described cases, a new Pathways panel should be opened that is placed as other side panels - at the right side of the page.
This panel should contain:

• pathway search input field
• button to start the search
• two sub-tabs:
  • "Databases" (opened by default) - for displaying search results over pathways from external pathway databases
  • "Pathways" - for displaying search results over internal pathways (inner NGB storing)

Example of view:

Notes:

• if the Pathways panel was opened from the VIEWS item of the main menu - this panel should be empty as shown on the image above (search field is empty, sub-tabs are empty too)
• if the Pathways panel was opened from the context menu when clicking any gene/feature - the name of that gene/feature should be specified to search field and the search should be performed over external pathways database by selected gene/feature name (example of the Pathways panel view in this case see below)

Databases sub-tab

After user has specified the search keyword and click the Search button or just opened the Pathways panel from the context menu when clicking any gene/feature, the search should be performed over external pathways databases by the keyword/gene or feature name.
Results of the search should be displayed in the "Databases" sub-tab, e.g.:

Search results should contain a list of found maps from external pathways database.
Each record should contain the following fields:

• Map - map ID
• Thumbnail - thumbnail image of the map
• Name - map name
• Description - map description. Note: as often that descriptions are rather long, there should be only short part of the description is shown, e.g. first 150 symbols

Note: list of found records should support pagination

The "Map" column should contain hyperlinks - user can click any record in this list to open details of the specific metabolic pathways map.
The new form with map details should be opened in the same sub-tab instead previous list, e.g.:

This form shall contain:

• button to return to the previous table (with all found maps)
• map name label
• "Info" button near the map name label - by click this button a tooltip should appear with the full map description
• panel with additional capabilities that can be applied to the opened map:
  • search control - for the search over the current pathway objects
  • annotation section - for adding annotation file(s) to the current pathway
• full map image:
  • the map image should be shown (PNG-format image received from KEGG)
  • by default, image should be autoscaled - fitted by width
  • there should be "Zoom-in"/"Zoom-out" buttons - to change scale
  • there should be the ability to change scale by mouse wheel
  • there should be the ability to move over the image on large scales by mouse hold+move

Search over the pathway
User shall have the ability to search over pathway objects - via the search control at the additional capabilities panel:

• search results should be highlighted at the map image
• no auto-scaling to the search results are required
• each new search should highlight only new found objects, previous search results should be hide
  

Add annotation to the pathway
User shall have the ability to add annotation(s) to the displayed map.
This annotation should allow to colorize the map.
The following annotations shall be supported:

• heatmap file, previously registered in NGB
• variation file, previously registered in NGB (TBD)
• data in TSV/CSV format, uploaded from the local workstation
• manually specified annotation in NGB GUI

To add an annotation, user shall click the "Add+" control at the additional capabilities panel.
A pop-up shall appear, e.g.:

This pop-up shall contain:

• text input field for specifying annotation name (optional)
• dropdown list with heatmap and variation files registered in NGB, multiselect is disabled - only one file from the list can be selected
• button to load annotation file from the local workstation (in TSV/CSV format)
• text input field for specifying annotation in plain text format
• controls to switch between options from which the annotation will be received. There shall be the ability to use only one option at the moment - load from the dropdown/upload from the local workstation/manual configuration
• button to save the annotation

If user selects any annotation from the dropdown list or load from the local TSV/CSV file, the view of that pop-up shall be changed:

• annotation name field is automatically filled by selected file name. Then, user can manually change this name, if needs
• the control shall appear - to set where the pathway object IDs (names/labels) are expected - in first column (row headers) or in first row (column headers)
• for the annotation, the ability to config the colorizing shall be support. Here the same settings as for heatmaps should be:
  • ability to select between continuous (default) and discrete colorizing
  • for continuous colorizing, there should be the ability to set colors for max, min, average values
  • for discrete colorizing, there should be the ability to set a color for each specific value or configure color(s) for values range(s)
    

If user selects "Manual colors config" - no additional controls appear, in this case user should manually specify the colorization scheme and annotation name if needs, e.g.:

When the annotation was saved:

• the annotation pop-up shall be closed
• saved annotation should appear in the additional capabilities panel. For each added annotation, there should appear:
  • annotation name (as hyperlink). If it was not specified - it should be set automatically, e.g. annotation_<index_number>
  • remove button near the annotation name - to remove the annotation from the list
  • checkbox near the annotation name - to enable (apply) it or disable
• just saved annotation should be applied to the map
• when the annotation is applied to the map - objects according to the annotation should be colorized to the specified colors
  • if no objects are found with the specified identifiers - nothing should be colorized

Note: about annotation possible formats and colors applying see below.

Manage the annotations:

• user can add and apply several annotations for the map
• if any object is subject to several annotations - it should be colorized only according to the first applied annotation
• user can click any annotation name to open the annotation details and change them if needs - in the annotation pop-up (similar as described above)
• user can remove any annotation by the remove button near it

Pathways sub-tab

Pathways sub-tab can be opened manually.
Here, the search over internal NGB pathways should be performed - by the keyword/gene or feature name.
Search should be performed over the following fields of pathway maps:

• name
• description
• map objects

Results of the search should be displayed in the same sub-tab - similar to Databases sub-tab, but without map IDs and thumbnails, e.g.:

Each record should contain the following fields:

• Map - map name
• Description - map description

Notes:

• List of found records should support pagination
• Table should supports sorting by "Map" column

The "Map" column should contain hyperlinks - user can click any record in this list to open details of the specific metabolic pathways map.
The new form with map details should be opened in the same sub-tab instead previous list.
Content and functionality for internal maps details should be fully similar to external database maps, the single difference - the map image should be generated and drawn from pathway map file in SBGN-xml format, e.g.:

Internal pathways

• NGB shall support registration and management of pathway map files in one of the following formats: SBML, SBGN, Biopax (TBD with GUI team)
• Pathway data shall not be stored in the internal database, server just stores path to the original file (same as for BAM files and similar), but some lucene index may be required to allow gene search over the pathway
• NGB CLI shall provide standard register, list, delete operations for pathway map files
• GUI shall support drawing of pathway maps in the selected format

BioCYC Pathways integration

• NGB shall support registration of BioCYC pathway files in biopax format
• NGB shall support gene search over registered BioCYC pathway files
• NGB shall support visualization of BioCYC biopax pathways

Pathways annotation

• Pathway annotation shall be supported for both internal and external pathways
• NGB shall support annotation of pathway map based on heatmap or custom TSV/CSV (table) data like different pathway tools e.g. Pathview
• NGB shall support annotation of pathway map based on variations data (TBD: how to map variation to pathway element)

Map colorizing shall be different according to the annotation source.

Annotation file

When user selects heatmap/TSV or CSV file as annotation, the matrix format of the annotation is expected.
In this case:

• pathway objects (their IDs, names or labels) should be placed in the first column (row headers) or the first row (column headers) of the matrix. Note: this should be specified by the user when create/edit annotation in GUI
• all other matrix values should be used for colorizing pathways objects

If the continuous scheme is selected, the algorithm of the colorizing should be the following:

• 3 colors should be configured - for minimum value (from all matrix), for maximum value (from all matrix) and middle value (average between minimum and maximum). Colors for all other values in matrix are selected from a gradient according to these three
• for each pathway object from the matrix - if this object is presented at the map, it should be colorized:
  • color(s) for the object should be determined from the value corresponding to that object in the matrix and from the defined gradient
  • only object background color should be changed
  • if for the object, there are several values in the matrix:
    • for each value, the color should be determined
    • object should be colorized in all these colors equally (from left to right in the order as values are specified in matrix)

For example, if the annotation file looks like:

          Sample1
Gene1     -0.87
Gene2     0.1
Gene3     0.93
Gene4     0.99
Gene5     -0.8

Then, the following example map with applied annotation shall be colorized in the way (if the gradient colors were selected correspondingly):

For example, if the annotation file looks like:

          Sample1       Sample2
Gene1     -0.87         0.02
Gene2     0.1           -0.45
Gene3     0.93          0.88
Gene4     0.99
Gene5     -0.8          -0.9

Then, the following example map with applied annotation shall be colorized in the way (if the gradient colors were selected correspondingly):

If the discrete scheme is selected, the algorithm of the colorizing should be the following:

• for each specific value of the matrix the color should be configured (or color(s) for value range(s))
• for each pathway object from the matrix - if this object is presented at the map, it should be colorized in the defined color
  • defined color(s) for the object should be determined from the value corresponding to that object in the matrix and from the defined gradient
  • only object background color should be changed
  • if for the object, there are several values in the matrix:
    • object should be colorized in all these colors equally (from left to right in the order as values are specified in matrix)

For example, if the annotation file looks like:

          Sample1       Sample2
Gene1     -0.87         0.02
Gene2     0.1           -0.45
Gene3     0.93          0.88
Gene4     0.99
Gene5     -0.8          -0.9

Then, the following example map with applied annotation shall be colorized in the way (if colors were selected correspondingly):

Manual colorization

Manual colorization annotation is expected in the following format:

<object_identifier_1>    <background_color1>[,<foreground_color1>]
<object_identifier_2>    <background_color2>[,<foreground_color2>]
...

Note: color can be specified in HEX format (in this case, it should start with # symbol, e.g. #ffffff) or as HTML color name (e.g. White)

The algorithm of the colorizing should be the following:

• for each pathway object from the list - if this object is presented at the map, it should be colorized in the defined color:
  • <background_color> (if specified) should be used as background color
  • <foreground_color> (if specified) should be used as border color and text color

For example, if the manual annotation config looks like:

Gene1     #121212
Gene2     Yellow,#ffab00
Gene3     Green,#cca837
Gene4     Red
Gene5     #000000,White

Then, the following example map with applied annotation shall be colorized in the way:

Additional details

• Created annotations should be stored globally
• Users shall have the ability to save an opened map with applied annotation(s) in the session

[Tatyana2022]

Bug was found:
Missing pathway map for 'Glycolysis 2'

To reproduce:

1. Go to NGB
2. Open VIEWS->PATHWAYS
3. Go to PATHWAYS panel
4. Enter 1 value in Search keyword field
5. Click on Glycolysis2 hyperlink in 'Name' column
6. Look at PATHWAYS sub-tab

Expected result:
Specific pathways map should be displayed in the PATHWAYS sub-tab

Actual result:
Text data outside of root node. Line: 0 Column: 63 Char: } text displays instead Glycolysis2 pathways map

@mzueva @okolesn

Fixed and verified

[Tatyana2022]

Bug was found:
Apply button is disabled after switching between checkboxes in annotations

To reproduce:

1. Go to NGB
2. Open PATHWAYS panel
3. Find and open BRCA1
4. Click A (annotations) button
5. Set checkbox in Upload CSV/TSV field
6. Unset checkbox in Upload CSV/TSV field
7. Set checkbox in Manual colors config field
8. Enter any value
9. Look at Apply button

Expected result:
Apply button should be active

Actual result:
Apply button is disabled

Extra details:
Same behavior for all toggle options between checkboxes

@rodichenko @mzueva @DmitriiKrasnov

Fixed and verified

[Tatyana2022]

Bug was found:
The map slides to the right or left when zooming in or out

To reproduce:

1. Go to NGB
2. Open PATHWAYS panel in main menu
3. Find and open pathway BRCA1
4. Click on + button
5. Double click on - button

Expected result:
The pathway map should maximize and minimize in the center of the PATHWAYS panel

Actual result:
5) The pathway's map slides to the bottom right corner when zooming in
6) The pathway's map slides to the left upper corner when zooming out

@rodichenko @mzueva @DmitriiKrasnov

Fixed and verified

[Tatyana2022]

Bug was found:
An error displays in the console after opening PATHWAYS panel from track

Prerequisites
Open console (Ctrl+F12)

To reproduce:

1. Go to NGB
2. Open PATHWAYS panel in main menu
3. Find and open Pwycollage E.Coli
4. Search by ybha in the Search field
5. Click on highlighted value
6. Go to gene's track on BROWSER panel
7. Open Context menu
8. Click Show pathways
9. Look at console

Expected result:
No errors in the console.
Pathways panel is opened

Actual result:
Pathways panel is opened
An error displays in the console

app.bundle.0e08a0baaf769ff20edb.js:1049 Uncaught TypeError: Cannot read properties of null (reading 'isHeadless')
    at Core.headless (app.bundle.0e08a0baaf769ff20edb.js:1049:80163)
    at app.bundle.0e08a0baaf769ff20edb.js:1047:52662
    at cachedBoundingBoxImpl (app.bundle.0e08a0baaf769ff20edb.js:1047:56462)
    at Collection.elesfn$k.boundingBox (app.bundle.0e08a0baaf769ff20edb.js:1047:58057)
    at update (app.bundle.0e08a0baaf769ff20edb.js:1047:45276)
    at Element.elesfn$k.updateCompoundBounds (app.bundle.0e08a0baaf769ff20edb.js:1047:47584)
    at Object.beforeGet (app.bundle.0e08a0baaf769ff20edb.js:1047:40870)
    at Element.<anonymous> (app.bundle.0e08a0baaf769ff20edb.js:1047:8991)
    at Element.<anonymous> (app.bundle.0e08a0baaf769ff20edb.js:1041:33596)
    at _loop2 (app.bundle.0e08a0baaf769ff20edb.js:1047:67984)
    at app.bundle.0e08a0baaf769ff20edb.js:1047:68151
    at forEachEventObj (app.bundle.0e08a0baaf769ff20edb.js:1047:65787)
    at Emitter.p.emit.p.trigger (app.bundle.0e08a0baaf769ff20edb.js:1047:67148)
    at Core.emit (app.bundle.0e08a0baaf769ff20edb.js:1049:9858)
    at app.bundle.0e08a0baaf769ff20edb.js:1047:68248
    at forEachEventObj (app.bundle.0e08a0baaf769ff20edb.js:1047:65695)

@rodichenko @mzueva @DmitriiKrasnov

[Tatyana2022]

Bug was found:
Pop-up Annotations window is not displayed after creation annotation

To reproduce:

1. Go to NGB
2. Open PATHWAYS panel in main menu
3. Find and open pathway BRCA1
4. Click on A(Annotation) button
5. Fill any field by any values
6. Click Apply button

Expected result:
Pop-up Annotation window with created annotation should be displayed in the right upper corner

Actual result:
Pop-up Annotations window is not displayed after creation annotation

@rodichenko @mzueva @DmitriiKrasnov

Fixed and verified

[Tatyana2022]

Bug was found:
It is not possible reopen Add Annotation pop-up with manual colorization

To reproduce:

1. Go to NGB
2. Open PATHWAYS panel in main menu
3. Find and open pathway BRCA1
4. Click on A(Annotation) button
5. Set checkbox in Manual colors config field
6. Set any values (e.g. Laclp red blue)
7. Click Apply button
8. Click on A(Annotation) button
9. Click on link with created annotation

Expected result:
Add annotation pop-up should be opened with filled Manual colors config field

Actual result:
Add Annotation pop-up with manual colorization is not opened

Extra details:
An error displays in the console:

app.bundle.0e08a0baaf769ff20edb.js:60 TypeError: Cannot read properties of null (reading 'copy')
    at ngbPathwaysAnnotationAddDlgController (app.bundle.0e08a0baaf769ff20edb.js:1049:399004)
    at Object.invoke (app.bundle.0e08a0baaf769ff20edb.js:60:40742)
    at extend.instance (app.bundle.0e08a0baaf769ff20edb.js:60:105109)
    at Object.link (app.bundle.0e08a0baaf769ff20edb.js:639:27641)
    at app.bundle.0e08a0baaf769ff20edb.js:639:43464
    at app.bundle.0e08a0baaf769ff20edb.js:639:44048
    at app.bundle.0e08a0baaf769ff20edb.js:60:183790
    at app.bundle.0e08a0baaf769ff20edb.js:60:183933
    at Scope.$eval (app.bundle.0e08a0baaf769ff20edb.js:60:195953)
    at Scope.$digest (app.bundle.0e08a0baaf769ff20edb.js:60:193358)
    at Scope.$apply (app.bundle.0e08a0baaf769ff20edb.js:60:196379)
    at HTMLAnchorElement.<anonymous> (app.bundle.0e08a0baaf769ff20edb.js:60:249603)
    at HTMLAnchorElement.dispatch (app.bundle.0e08a0baaf769ff20edb.js:39:58996)
    at HTMLAnchorElement.elemData.handle (app.bundle.0e08a0baaf769ff20edb.js:39:55827)

@rodichenko @mzueva @DmitriiKrasnov

Fixed and verified

[Tatyana2022]

Bug was found:
Server error is displayed when searching for certain values

To reproduce:

1. Go to NGB
2. Open PATHWAYS panel in main menu
3. Enter pw* value in the Search keyword field
4. Click Search button

Expected result:
All matched results started with "pw" value should be displayed in the table

Actual results:
Your operation has been aborted due to NGB has encountered a server problem. text displays instead table with results

Extra details:
The same behavior for:
Glycolysis_Homo_sapiens, Glycolysis description...

Finds results for following pathways:
pwycollage.graph, BRCA1...

@mzueva @okolesn

Fixed and verified

[Tatyana2022]

Bug was found:
Switching to the gene track does not match the result

To reproduce:

1. Go to NGB
2. Open PATHWAYS panel in main menu
3. Find and open Pwycollage human pathway
4. Apply search with following values: fbp
5. Click on found node
6. Go to gene's track on BROWSER panel
7. Look at found gene

Expected result:
The gene should not have been found because there is no match

Actual result:
FBP1 gene is displayed in the gene track

@rodichenko @DmitriiKrasnov @mzueva

[Tatyana2022]

Bug was found:
The Apply button is activated after double-clicking the checkbox in annotations

To reproduce:

1. Go to NGB
2. Open PATHWAYS panel
3. Find and open BRCA1
4. Click A (annotations) button
5. Set checkbox in Upload CSV/TSV field
6. Unset checkbox in Upload CSV/TSV field
7. Look at Apply button

Expected result:
Apply button should be disabled

Actual result:
Apply button is active

@rodichenko @mzueva @DmitriiKrasnov

[Tatyana2022]

Bug was found:
Incorrect coloring of nodes if the node consists of more than 1 value

To reproduce:

1. Go to NGB
2. Open PATHWAYS panel
3. Find and open Pwycollage human pathway
4. Click A (annotations) button
5. Set checkbox in Manual colors config field
6. Enter enolase: eno red blue
7. Click Apply button
8. Look at enolase: eno node

Expected result:
enolase: eno node should have blue background with red text

Actual result:
enolase: eno text has red text only without background

@rodichenko @mzueva @DmitriiKrasnov