Merge pull request #2877 from kbase/DATAUP-737_merge_truss_into_develop

DATAUP-737: merge truss into develop (!!!!)

docs/adrs/0004-bulk-import-output-cells.md

@@ -29,7 +29,7 @@ Bulk Import cell importer apps that employ an output cell:
 | Importer | Output cell |
 |---|---|
 | Import GenBank as genome from staging | Genome viewer |
-| Import GFF/FASTA as genome from staging | Genome viewer |
+| Import GFF+FASTA as genome from staging | Genome viewer |
 
 Other importer apps with output cells:
 

docs/developer/data_import.md

@@ -0,0 +1,48 @@
+# Data Importing in the KBase Narrative
+
+This document highlights some development details about importing data into the KBase Narrative. For user documentation on how to use it, see the [KBase Data Upload and Download Guide](https://docs.kbase.us/data/upload-download-guide).
+
+## Introduction
+
+The general flow of data goes from an external data source, to files uploaded to the Staging Area, to data objects imported to the Workspace service and available in the Narrative. The two main user actions here - uploading and importing - are performed by accessing the Staging Service, and running one or more Import apps, respectively. Both of these are started by opening the Import tab of the data slideout.
+
+## Staging Service
+
+The Staging Service acts as a first stop for importing data. This service provides an API to a KBase-hosted file system, where each user has a separate file storage directory under their username. The Narrative front end uses a [REST client](../../kbase-extension/static/kbase/js/api/StagingServiceClient.js) to access that service and manipulate files.
+
+## Staging Service Viewer
+
+Users can interact with the Staging Service through the Import tab of the data slideout. The top portion of this tab is controlled by the [FileUploadWidget](../../kbase-extension/static/kbase/js/widgets/narrative_core/upload/fileUploadWidget.js). This uses [Dropzone](https://www.dropzone.dev/js/) to provide a drag-and-drop interface for uploading files. If the user's account is linked to Globus, a link is also provided. Another option is to upload via URL. This requires running an app that imports data to the Staging Service from an external, public URL.
+
+The bottom portion of the Import tab acts as a file browser and selector for starting the Import process. The staged files get rendered by the [StagingAreaViewer](../../kbase-extension/static/kbase/js/widgets/narrative_core/upload/stagingAreaViewer.js) module. This viewer fetches the list of files from the Staging Service, updating every 30 seconds.
+
+## Data Import
+
+Data import is initiated by selecting one or more data files from the Import tab, then running the Importer app for that data type. When at least one file has a data type chosen and its checkbox selected, the `Import Selected` button will activate. Clicking that will close the slideout and open a Bulk Import cell and/or at least one App Cell with the relative file path entered. File paths are given as the relative path from the root of the Staging Area without the username - the username is assumed to be the root, and will be added at app run time.
+
+### Importer App Configuration
+
+The data import app selection is configured in [`staging_upload.json`](../../kbase-extension/static/kbase/config/staging_upload.json). This JSON file has the following top-level keys:
+* `dropdown_order`: this describes the order of the file selection dropdown. Each item here has an `id` field, which must be unique, and acts as the app descriptor (more in the `app_info` field below), and a `name` field. The `name` field is displayed to the user in the dropdown.
+* `bulk_import_types`: This array contains data type `id`s that should trigger a Bulk Import cell. If a user imports at least one file with a data type in this list, it'll map that file and import app into a bulk import cell.
+* `app_info`: This object has a set of app ids for keys that map onto info about each app. These keys are used as the app `id`s elsewhere in the configuration.
+
+Each `app_info` object has the following accepted keys:
+* `app_id`: the app to use for importing the file in the format `module_name/app_name` (e.g. `kb_uploadmethods/import_genbank_as_genome_from_staging`). This is the only required field.
+* `app_input_param`: this is the parameter for setting the file name in the app. App specs don't provide an unambiguous parameter for a file input, hence the need for this clarifying element.
+* `app_input_param_type`: describes what the `app_input_param` is expected to be. Accepts either `string` or `list` (list of strings). If the given app spec has `allow_multiple` as an option for the input, then this should be of type `list`.
+* `app_static_params`: an object with key-value pairs of other app parameters that should be preset when creating the app cell.
+* `app_output_param`: this should be the app's parameter id for the created object. This gets automatically filled with a suggested name based on the file name and the `app_output_suffix` field below. Note that this does not take into account whether the parameter has the `is_output_name` option set.
+* `app_output_suffix`: an automatic suffix to add to the data file name for a created object. E.g. if the file name is "e_coli.gbk", and the `app_output_suffix` is "_genome", then the automatically set output name will be "e_coli.gbk_genome".
+* `multiselect`: (deprecated) originally intended to denote files with multiple inputs, this is not currently implemented and will be removed in the future.
+
+### Configuring a new Bulk Import app
+The steps for adding a bulk import app to the configuration are fairly simple.
+1. Add the `app_info` object field as described above, with a unique id.
+2. Add the new data uploader's id to `bulk_import_types`.
+
+Be aware that as of Narrative 5.0.0, there are several issues to be aware of before adding a new bulk import app:
+* Apps that parallelize themselves (e.g. make use of the KBParallel module, or otherwise create extra jobs in the Execution Engine) may not work, or may cause a deadlock where no further app runs may be submitted by that user. These are not recommended for use.
+* Apps that make use of Grouped Parameters (the `"parameter-groups"` field in an app spec) will not work correctly in a Bulk Import cell.
+* Apps that require subobject selection may not work, and have not been tested in the Bulk Import cell.
+* Apps that require the use of a dynamic dropdown (a select box that gets its available selections from a service) may not work as expected in a Bulk Import cell.

kbase-extension/scss/modules/_colours.scss

@@ -147,6 +147,7 @@ $job_status_colors: (
   // error already exists, but we need it in our job status list
   "error": map-get($kbase-palette, 'error'),
   "does_not_exist": map-get($kbase-palette, 'error'),
+  "ee2_error": map-get($kbase-palette, 'warning-dark'),
 );
 
 $palette: map-merge($kbase-palette, $job_status_colors);

kbase-extension/scss/partials/_appCell.scss

@@ -141,7 +141,6 @@
 
   .kb-app-footer {
     background-color: rgb(245 245 245);
-    overflow: none;
     padding: 10px;
     width: 100%;
   }

kbase-extension/scss/partials/_buttons.scss

@@ -76,6 +76,7 @@
   &.active {
     &:focus,
     &.focus {
+      outline: 5px auto Highlight;
       outline: 5px auto -webkit-focus-ring-color;
       outline-offset: -2px;
     }

kbase-extension/scss/partials/_cellToolbar.scss

@@ -29,6 +29,7 @@ $default-icon-size: 56px;
   &__title-container {
     flex: 1 1 10rem;
     text-overflow: ellipsis;
+    overflow: hidden;
 
     // .cell.unselected .kb-cell-toolbar__title-container
     .cell.unselected & {
@@ -44,6 +45,7 @@ $default-icon-size: 56px;
     margin-top: 8px;
     min-width: 0;
     text-overflow: ellipsis;
+    overflow: hidden;
   }
 
   // .kb-cell-toolbar__subtitle
@@ -53,12 +55,13 @@ $default-icon-size: 56px;
     height: 20px;
     min-width: 0;
     text-overflow: ellipsis;
+    overflow: hidden;
   }
 
   // .kb-cell-toolbar__buttons-container
   &__buttons-container {
-    flex: none;
-    min-width: 9.5rem;
+    float: right;
+    min-width: 2rem;
     white-space: nowrap;
 
     // unselected cells
@@ -80,14 +83,13 @@ $default-icon-size: 56px;
       color: use_color('accent-warm');
     }
 
-    // .kb-cell-toolbar__icon--notch-spin
-    &--notch-spin {
-      color: use_color('primary');
-    }
-
     // .kb-cell-toolbar__icon--outdated
     &--outdated {
+      border: 0;
       color: use_color('accent-warm');
+      margin: 0 5px;
+      padding: 0;
+      vertical-align: top;
     }
 
     // .kb-cell-toolbar__icon--plus
@@ -104,8 +106,7 @@ $default-icon-size: 56px;
 
     /* .kb-cell-toolbar__icon--times,
     .kb-cell-toolbar__icon--triangle */
-    &--times,
-    &--triangle {
+    &--times {
       color: use_color('red');
     }
   }

kbase-extension/scss/partials/_editorCell.scss

@@ -103,7 +103,6 @@
 
   .kb-app-footer {
     background-color: rgb(245 245 245);
-    overflow: none;
     padding: 10px;
     width: 100%;
   }

kbase-extension/scss/partials/_icons.scss

@@ -9,7 +9,6 @@
   font-style: normal !important;
   font-variant: normal !important;
   font-weight: 400 !important;
-  speak: none;
   text-transform: none !important;
 }
 

kbase-extension/scss/partials/_jobLog.scss

@@ -50,6 +50,9 @@ $counter-name: code-list-counter;
     &--expanded {
       max-height: $line-height-spacious*30;
     }
+    p {
+      margin: 1rem;
+    }
   }
   // .kb-log__log_line_container
   &__log_line_container {

kbase-extension/scss/partials/_landingPages.scss

@@ -71,7 +71,7 @@ table {
 
 .scroll-pane {
   height: 200px;
-  overflow-x: hi;
+  overflow-x: hidden;
   overflow-y: scroll;
 }
 

kbase-extension/scss/partials/_narrative.scss

@@ -568,7 +568,7 @@ p#site-title { /* FIXME */
 
 #data-add-btn {
   font-size: 0.8em;
-  padding: 0 0 -10px;
+  padding: 0;
   position: absolute;
   right: 7px;
 
@@ -920,7 +920,7 @@ img.remove {
 }
 
 .ui-state-highlight {
-  background-color: fbfaed;
+  background-color: #fbfaed;
 }
 
 p span.notification {
@@ -1197,7 +1197,6 @@ div#notebook_panel { /* FIXME Jupyter css */
       border-radius: 5px;
       color: rgb(51 51 51) !important;
       cursor: pointer;
-      *cursor: hand;
       margin: 0 3px;
       padding: 2px 5px;
     }
@@ -1208,7 +1207,6 @@ div#notebook_panel { /* FIXME Jupyter css */
       border-radius: 5px;
       color: rgb(51 51 51) !important;
       cursor: pointer;
-      *cursor: hand;
       margin: 0 3px;
       padding: 2px 5px;
 
@@ -1459,7 +1457,7 @@ div#notebook_panel { /* FIXME Jupyter css */
 }
 
 .unselected .kb-toolbar-open {
-  background-color: none;
+  background: none;
 }
 
 .selected.kb-error .kb-toolbar-open {
@@ -1863,7 +1861,6 @@ control with padding, and then scooting the icon back into its column.
 
 .kb-method-footer {
   background-color: rgb(245 245 245);
-  overflow: none;
   padding: 10px;
   width: 100%;
 }

kbase-extension/scss/partials/_reportViewer.scss

@@ -68,7 +68,6 @@
     border-bottom: 1px solid use-color('base-lighter');
   }
 
-
   &__item_toggle {
     @include font-h4;
     padding: 0.5rem 0;

kbase-extension/static/kbase/config/job_config.json

@@ -0,0 +1,43 @@
+{
+    "params": {
+        "BATCH_ID": "batch_id",
+        "CELL_ID": "cell_id",
+        "CELL_ID_LIST": "cell_id_list",
+        "JOB_ID": "job_id",
+        "JOB_ID_LIST": "job_id_list"
+    },
+    "message_types": {
+        "CANCEL": "cancel_job",
+        "CELL_JOB_STATUS": "cell_job_status",
+        "ERROR": "job_error",
+        "INFO": "job_info",
+        "LOGS": "job_logs",
+        "RETRY": "retry_job",
+        "RUN_STATUS": "run_status",
+        "START_UPDATE": "start_job_update",
+        "STATUS_ALL": "job_status_all",
+        "STATUS": "job_status",
+        "STOP_UPDATE": "stop_job_update"
+    },
+    "requests": [
+        "CANCEL",
+        "CELL_JOB_STATUS",
+        "INFO",
+        "LOGS",
+        "RETRY",
+        "START_UPDATE",
+        "STATUS",
+        "STATUS_ALL",
+        "STOP_UPDATE"
+    ],
+    "responses": [
+        "CELL_JOB_STATUS",
+        "ERROR",
+        "INFO",
+        "LOGS",
+        "RETRY",
+        "RUN_STATUS",
+        "STATUS",
+        "STATUS_ALL"
+    ]
+}

kbase-extension/static/kbase/config/staging_upload.json

@@ -18,11 +18,11 @@
         },
         {
             "id": "gff_genome",
-            "name": "GFF/FASTA Genome"
+            "name": "GFF+FASTA Genome"
         },
         {
             "id": "gff_metagenome",
-            "name": "GFF/FASTA Metagenome"
+            "name": "GFF+FASTA Metagenome"
         },
         {
             "id": "expression_matrix",
@@ -38,7 +38,7 @@
         },
         {
             "id": "assembly",
-            "name": "Assembly"
+            "name": "FASTA Assembly"
         },
         {
             "id": "phenotype_set",

kbase-extension/static/kbase/custom/custom.js

@@ -109,7 +109,7 @@ define([
     'notebook/js/cell',
     'common/cellUtils',
     'common/jupyter',
-    'kb_common/html',
+    'common/html',
 
     'components/requirejs/require',
     'narrative_paths',

kbase-extension/static/kbase/js/common/cellComponents/tabs/jobStatus/jobActionDropdown.js

@@ -182,9 +182,7 @@ define([
         }
 
         function updateState() {
-            const jobCountsByStatus = Jobs.getCurrentJobCounts(
-                jobManager.model.getItem('exec.jobs')
-            );
+            const jobCountsByStatus = Jobs.getCurrentJobCounts(jobManager.getIndexedJobs());
 
             actionArr.forEach((action) => {
                 const result = action.target.some((status) => {