Stable URLs, s3 link improvements

README.md

@@ -1,6 +1,6 @@
 # MAAP Jupyter Interface
 
-This repo includes JupyterLab extensions that have been built for the MAAP project (https://che-k8s.maap.xyz/)
+This repo includes JupyterLab extensions that have been built for the MAAP project (https://www.maap-project.org/)
 
 In order to use each of these extensions they must be installed and enabled in your environment. Instructions for each extension can be 
 found in the respective folder. Make sure each extension's dependencies are installed first.
@@ -16,8 +16,8 @@ Create a new Conda environment and install common dependencies and versions:
 conda create --name maap-ade
 conda activate maap-ade
 conda install conda=4.7.12 jupyterlab=2.1.4 nodejs=10.13.0 gitpython=3.0.2
-jupyter labextension install jupyterlab_toastify@3.3.0 --no-build
-npm i jupyterlab_toastify@3.3.0
+jupyter labextension install jupyterlab_toastify@3.0.0
+npm i jupyterlab_toastify@3.0.0
 pip install plotly==4.0.0
 jupyter labextension install @jupyter-widgets/[email protected]
 ```
@@ -68,143 +68,35 @@ Some Jupyter Extensions/Resources we have found helpful:
 In JupyterLab's update to the stable 1.0 version, they have also updated and added lots of documentation on extension 
 development. I recommend taking a look at [this](https://jupyterlab.readthedocs.io/en/stable/developer/extension_dev.html).
 
-## Deploying Extensions as Part of Eclipse Che
-### Dockerizing
-Our development process involves building and running an extension locally in jupyterlab using a conda env before 
-installing it on the che server. To enable an extension in Che, it must be included in the base docker image/stack that a 
-Che workspace is launched with. The dockerfile that extensions are included in is the `Dockerfile` and the highest level
-in this repo. At the point of adding your extension into the Docker image, some minor changes may have to be made 
-(mainly path issues). This will be explained in the bullets below.
-
-An instance of this repository lives on the Che server under `~/che/dockerfiles/maap-jupyter-ide`. Once an extension has been tested locally, rebuild the docker 
-image with your new extensions.
-
-
-- Add your install to the Dockerfile. For example:
-    ```bash
-    # jlab pull projects into /projects directory
-    COPY pull_projects /pull_projects
-    RUN cd /pull_projects && npm run build
-    RUN cd /pull_projects && jupyter labextension link .
-    RUN cd /pull_projects && pip install -e .
-    RUN cd /pull_projects && jupyter serverextension enable --py pull_projects --sys-prefix
-
-    ```
-- If your extension includes a server extension you also need to modify `entrypoint.sh`. This is because jupyter 
-server extensions function off of having a standard base url, but in the context of che the url is not what jupyter 
-thinks it is.
-    - Here is some magic that fixes it (add this line and replace with the path to where 
-your `load_jupyter_server_extension` function is)
+## Steps for Contributing
+### Each Contribution
+- Clone repository locally to test and develop extensions. Create your own branch and once you have tested it and are ready to contribute, put a PR into the develop branch.
+- After you have validated your code is working locally, the next step is to test it in the MAAP ADE in the devlopment environment.
+- When there is an update to the develop branch, such as merging in a PR, the CI will be kicked off automatically and will build the image necessary for use in the MAAP ADE. This image will end in the `:develop` tag. This is intended to be used to test in the development environment.
+- When updates in develop have been tested in the MAAP ADE and deemed stable, merge them into master.
+### Ready for Release
+- Once there have been changes that warrant a new release to the operations environment, create a new Github release from the master branch. Name the release the next corresponding number following the `v3.0.1` schema. In the notes of the release include a changelog of the updates included in that release
+- Then, update the existing `stable` tag to point to the same place as the release that you just created. This can easily be done on the command line by checking out the latest code on master (or whatever commit the release was created from) and running the following:
+  ```
+  git tag -f stable
+  git push -f origin stable
+  ```
+- The CI will also be kicked off for `stable` and `v` tags. The `stable` tag is used in the devfiles in the operations environment and automatically pulled in each new workspace, so it is very important pushing changes to this tag have been thuroughly tested. The `v` tag is not used in any environment but is used to track versions and the changes we role out. If there is an issue found in the `stable` tag we can roll it back to the previous `v` release by checking out that release and running the lines above.
+- The CI trigger information can be found .github/workflows/gitlab-trigger.yml and in the actions tab
+
+## Docker Images and CI Process
+- The CI and images built off of this code live on MAAP's hosted GitLab (https://mas.maap-project.org/root).
+- Triggered from maap-jupyter-ide
+    - Will build corresponding maap-jupyter-ide branch on all of the base images listed in the base_images.txt file
+- Triggered from Gitlab
+    - If triggered from a [base image](https://mas.maap-project.org/root/ade-base-images) being updated, it will only rebuild that base image with map-jupyter-ide’s stable tag on top of it.
+    - If triggered from an update in the [jupyter-image](https://mas.maap-project.org/root/jupyter-image) repo, it will rebuild all of the base images listed in base_iamges.txt with maap-jupyter-ide’s stable tag.
+- Troubleshooting
+    - If the image failed to build for an unexpected reason, it is likely because the gitlab runner ran out of space. Ssh onto the runner, clean up the docker images (you can get rid of anything - everything important is on s3 backed registries) and restart the failed job.
+
+
+## Troubleshooting
+- If your extension includes a server extension you also need to modify `entrypoint.sh`. This is because jupyter server extensions function off of having a standard base url, but in the context of che the url is not what jupyter thinks it is.
+  - Here is some magic that fixes it (add this line and replace with the path to where your `load_jupyter_server_extension` function is)
         ```bash
         perl -pi -e "s|web_app.settings\['base_url'\]|'/'|g" /show_ssh_info/show_ssh_info/__init__.py
-        ```
-- Then rebuild the docker image. `microk8s.docker build -t localhost:32000/che-jupyter-lab-ide .` 
-- Push! `microk8s.docker push localhost:32000/che-jupyter-lab-ide `
-- Now when you build a new workspace with the `localhost:32000/che-jupyter-lab-ide` image it will automatically 
-fetch the new image. (found in the stack's `Recipe` or `Raw Configuration`)
-    - you can also specify the image tag to use in your build on the stack if you want to use a previous build
-- Any change pushed to `microk8s.docker push localhost:32000/che-jupyter-lab-ide ` will affect the default stacks
-on all user accounts. If you are testing something, you can create your own image and your own stack to play around with.
-
-
-### Che Stacks
-To make your custom docker image available to users in Che, you need to make a new stack that creates workspaces using your image and make it available to users.
-Below is an example stack configuration using our locally built dockerized juptyer image with MAAP extensions installed.  
-
-Make sure to replace the image name in `workspaceConfig.environments.default.recipe.image` with the location of your image.
-In order for SSH-ing into the workspace to be possible, the `org.eclipse.che.exec` and `org.eclipse.che.ssh` installers must be enabled under `workspaceConfig.environments.default.machines.ws/jupyter.installers`.
-
-#### Che Stack Raw Configuration
-```
-{
-  "scope": "general",
-  "description": "Use this one. Stable jupyter. No extra packages",
-  "creator": "b07e3a58-ed50-4a6e-be17-fcf49ff8b242",
-  "tags": [
-    "MAAP",
-    "JUPYTER",
-    "STABLE"
-  ],
-  "workspaceConfig": {
-    "defaultEnv": "default",
-    "environments": {
-      "default": {
-        "recipe": {
-          "contentType": "text/x-yaml",
-          "type": "kubernetes",
-          "content": "kind: List\nitems:\n - \n  apiVersion: v1\n  kind: Pod\n  metadata:\n   name: ws\n   labels:\n    name: ws\n  spec:\n   containers:\n    - \n     name: jupyter\n     image: 'localhost:32000/stable-ide:latest'\n     resources:\n      limits:\n       memory: 1024Mi\n     securityContext:\n       privileged: true\n - \n  apiVersion: v1\n  kind: Service\n  metadata:\n   name: ws\n  spec:\n   type: NodePort\n   ports:\n    - \n     port: 22\n   selector:\n    name: ws\n     \n    "
-        },
-        "machines": {
-          "ws/jupyter": {
-            "env": {
-              "MACHINE_NAME": "WS_JUPYTER"
-            },
-            "servers": {
-              "jupyter": {
-                "path": "/",
-                "attributes": {
-                  "cookiesAuthEnabled": "true",
-                  "type": "ide",
-                  "secure": "true"
-                },
-                "protocol": "http",
-                "port": "3100"
-              }
-            },
-            "installers": [
-              "org.eclipse.che.exec",
-              "org.eclipse.che.ssh"
-            ],
-            "volumes": {
-              "projects": {
-                "path": "/projects"
-              }
-            },
-            "attributes": {}
-          }
-        }
-      }
-    },
-    "projects": [],
-    "commands": [],
-    "name": "default",
-    "attributes": {},
-    "links": []
-  },
-  "components": [],
-  "name": "maap-jupyter-ide",
-  "id": "stacktdo2q0ixhv7cge00"
-}
-```
-#### Enabling Privileged Docker Containers
-1. Cluster Privileges
-- in `/var/snap/microk8s/current/args/kubelet` and `/var/snap/microk8s/current/args/kube-apiserver`, append `--allow-privileged`
-- restart both services:
-```
-sudo systemctl restart snap.microk8s.daemon-apiserver
-sudo systemctl restart snap.microk8s.daemon-kubelet
-```
-
-2. Che Permissions
-- in `che/dockerfiles/init/manifest/che.env`, set `CHE_DOCKER_PRIVILEGED=true` under the Privileged Mode section
-- restart Che
-
-#### Creating and Sharing Stacks
-To create a stack, you write a raw configuration with all the che and docker settings your workspace will require, including installers, volumes, docker run tags, docker images, etc. See the example above.
-
-To share a stack, you will need to be the owner (creator) of the stack.
-Go to the homepage of where your Che instance is hosted and add `/swagger` to the end of the url for an interface with Che's API.   Under the `permissions`section, make a POST request with the users you want to share with and the id of your stack (shows up at the bottom of the configuration after creation).
-POST body:
-```
-{
-"userId": "*",
-  "domainId": "stack",
-  "instanceId": "${STACK_ID}",
-  "actions": [
-    "read",
-    "search"
-  ]
-}
-```
-reference: https://www.eclipse.org/che/docs/che-6/stacks.html#sharing-stacks-and-system-stacks
-.

dps_info/src/jobinfo.ts

@@ -141,14 +141,12 @@ export class JobTable extends Widget {
     }
 
     async _getJobList(me:JobTable) {
-        // console.log(this._username);
         const res:RequestResult = await getJobs(me._state, me._username);
         if(res.ok){
             let json_response:any = res.json();
-            INotification.success("Get user jobs success.");
-            // console.log(json_response);
             if (json_response['status_code'] === 200){
-                // let resp = json_response['result'];
+                INotification.success("Get user jobs success.");
+
                 me._table = json_response['result'];
                 JOBS = json_response['jobs'];
                 DISPLAYS = json_response['displays'];
@@ -1130,8 +1128,9 @@ export class JobWidget extends Widget {
         const res:RequestResult = await getJobs(this._state, this._username);
             if(res.ok){
                 let json_response:any = res.json();
-                INotification.success("Get user jobs success.");
                 if (json_response['status_code'] === 200){
+                    INotification.success("Get user jobs success.");
+
                     me._table = json_response['table'];
                     JOBS = json_response['jobs'];
                     DISPLAYS = json_response['displays'];

edsc_extension/edsc_extension/handlers.py

@@ -3,9 +3,25 @@
 import nbformat
 from notebook.base.handlers import IPythonHandler
 import subprocess
+import functools
+import json
 import maap
 from maap.maap import MAAP
 
+@functools.lru_cache(maxsize=128)
+def get_maap_config(host):
+    path_to_json = os.path.join(os.path.dirname(os.path.abspath(__file__)), '../..', 'maap_environments.json')
+
+    with open(path_to_json) as f:
+        data = json.load(f)
+
+    match = next((x for x in data if host in x['ade_server']), None)
+    maap_config = next((x for x in data if x['default_host'] == True), None) if match is None else match
+
+    return maap_config
+
+def maap_api(host):
+    return get_maap_config(host)['api_server']
 
 class GetGranulesHandler(IPythonHandler):
     def printUrls(self, granules):
@@ -18,7 +34,7 @@ def printUrls(self, granules):
 
     def get(self):
 
-        maap = MAAP()
+        maap = MAAP(maap_api(self.request.host))
         cmr_query = self.get_argument('cmr_query', '')
         limit = str(self.get_argument('limit', ''))
         print("cmr_query", cmr_query)
@@ -35,7 +51,7 @@ def get(self):
 
 class GetQueryHandler(IPythonHandler):
     def get(self):
-        maap = MAAP()
+        maap = MAAP(maap_api(self.request.host))
         cmr_query = self.get_argument('cmr_query', '')
         limit = str(self.get_argument('limit', ''))
         query_type = self.get_argument('query_type', 'granule')