Issue with pangeo kernelspec

[grallewellyn]

Unable to open a notebook if you only have the default kernelspec of /opt/conda/envs/pangeo/share/jupyter/kernels/python3
even though this environment is present

Creating a new kernelspec is a temporary workaround

Need to verify:

• is this issue just happening in pangeo? Maybe because it is in a separated jupyterlab/environment.yml?
• Is this issue happening in DIT?

[grallewellyn]

The problem is the nb_conda and nb_conda_kernels which are just installed in the jupyterlab image for pangeo
Options in order in which they should be attempted:

1. Correct the python paths if that is the issue Python paths wrong in pangeo  #956

2. Do we really need nb_conda and nb_conda_kernels? Alex suggested we might not here: Install missing packages to pangeo #898 (comment) but also nb_conda_kernels is present here: https://github.com/pangeo-data/pangeo-docker-images/blob/master/pangeo-notebook/environment.yml

3. Is there a way to correct nb_conda_kernels by removing the kernelspec and readding it again?

[grallewellyn]

Removing the jupyterlab-s3-browser package fixes the bug not being able to open notebooks and this package was already being removed in the 4.1.0 platform release which will come out next week
The issue with the console warning "Default kernel not found, using 'conda-env-pangeo-py'" is being caused by the nb_conda and nb_conda_kernels packages which are only present in the pangeo image
The purpose of the nb_conda package is to add a conda tab where you can manage your conda environments, but even when I do the developer’s install for nb_conda, I am still not seeing this conda tab. This package not working is a separate issue and I created a ticket
The purpose of the nb_conda_kernels package is to allow users to select different kernels based on their existing conda environments which is why we are seeing different kernel options in the pangeo image. Using nb_conda_kernels, I believe I am opening the right kernel in the notebook by attempting imports before and after installing packages into that conda env, but something about this package is messing up/ not setting the default kernel hence the console warning "Default kernel not found, using 'conda-env-pangeo-py'". I could not resolve the default kernel bug locally, so I don’t believe this is a python paths bug. The README for this package gives no additional installation instructions besides conda install
Do you think we should be concerned about the default kernel error messages if the correct kernel is still being used by the notebook?

[grallewellyn]

cc @wildintellect

[wildintellect]

@maxrjones @weiji14 have any experience with this?

@grallewellyn on a similar note, yesterday when working in the R image I couldn't figure out how to register a new conda env with an R kernel (custom not the default).
I'll need to read up on nb_conda , do we support similar features in other workspaces with some alternate method?

[grallewellyn]

I can register a new conda env with an R kernel by creating a kernel.json file in a test folder in /opt/conda/envs/r/share/jupyter/kernels/test like

{"argv": ["R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
 "display_name":"test",
 "language":"R"
}

then registering with jupyter kernelspec install /opt/conda/envs/r/share/jupyter/kernels/test. Then I can run R commands in a notebook started with this kernel. Is that what you meant?
I am not sure if this is the best way to do this, but creating an environment with packages like r-base, r-irkernel, r-essentials is not enough to have the language field in the kernel.json to be set to R

nb_conda is for viewing and managing packages in your conda environments, the only means we have right now for viewing and managing packages in a conda env, is just conda terminal commands

[weiji14]

I'm not familiar with the differences between https://github.com/anaconda/nb_conda_kernels and https://github.com/anaconda/nb_conda (there is a question on this at anaconda/nb_conda#88 which remains unanswered). I do see both installed in MAAP pangeo image at https://github.com/MAAP-Project/maap-workspaces/blob/77a270b61ef6aacd92658d65525287b63877c9a3/jupyterlab/pangeo/environment.yml#L21-L22, is it ok to have both installed like this?

@grallewellyn, what are the default kernels you see with a fresh start from Jupyter, i.e. provide the output of jupyter kernelspec list --json? Is the intention to set up R as the default kernel instead of Python?

[wildintellect]

We were copying pangeo-notebook https://github.com/pangeo-data/pangeo-docker-images/blob/f9d461494dce5c1589625b7e94b33c397a44b659/pangeo-notebook/environment.yml#L71
but I'll admit I didn't know about this feature for GUI management of packages. I think it's ok to drop for now, I doubt users knew about it either. I'll note only nb_conda_kernels is in pangeo.

@weiji14 yes on the R workspaces we want the R kernel as the default when a new R notebook is opened. The case I was mentioning was tangentially related when a user creates their own conda env and needs it to use R primarily.

[grallewellyn]

nb_conda- Conda tab to the Jupyter file browser that lets you manage conda environments and the packages within them
nb_conda_kernels- automatically shows all conda environments with a kernel installed in the launch page for jupyterlab

nb_conda_kernels is a dependency of nb_conda

jupyter kernelspec list --json with a fresh start gives

{
  "kernelspecs": {
    "ir": {
      "resource_dir": "/opt/conda/envs/r/share/jupyter/kernels/ir",
      "spec": {
        "argv": [
          "R",
          "--slave",
          "-e",
          "IRkernel::main()",
          "--args",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "R",
        "language": "R",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "python3": {
      "resource_dir": "/opt/conda/envs/r/share/jupyter/kernels/python3",
      "spec": {
        "argv": [
          "/opt/conda/envs/r/bin/python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3 (ipykernel)",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {
          "debugger": true
        }
      }
    }
  }
}

Users didn't know about nb_conda because it wasn't working, I also couldn't get this package to work locally

nb_conda_kernels also allows you to create a conda environment with r-irkernel (and maybe also ipykernel- I am seeing inconsistent behavior with this locally and on the ADE) and that conda environment will automatically show up as an option on the jupyterlab launch page with an R kernel so that could help with this problem