hpc instructions to use mamba and small fixes #826
Conversation
|
|
||
| :: | ||
|
|
||
| conda update conda | ||
| conda config --add channels conda-forge --force | ||
| conda config --remove channels defaults --force |
There was a problem hiding this comment.
I'm not sure why you need to use only conda-forge? This is to ensure the best compatibility between all the dependencies. This config might prevent using other channels like pytorch, don't this?
But at this point if a user needs other channels he knows what he's doing, so maybe that's OK.
There was a problem hiding this comment.
Any kind of channel mixing is problematic. We always recommend to use only 1 and, if it is extremely necessary, add the extra channel in the env file configuration. However, of keeping defaults and conda-forge in the condarc is usually problematic, specially without strict channel policy. That is b/c those two channels overlap a lot. It is not like getting pytorch from another channel, it is like mixing from two different sources for most, if not all, packages.
The other big advantage, besides avoiding compatibility issues, is to speed up the solver time by reducing the amount of packages it has to work with. Single channel reduces the scope by at least half and speed things up considerably.
There was a problem hiding this comment.
Thanks for the clarification Filipe! This has been a moving target for many of us, so good to know that one channel is the way to go now!
| echo "Step 2: Copy this ssh command into a terminal on your" | ||
| echo " local computer:" | ||
| echo "" | ||
| echo " ssh -N -L 8889:`hostname`:$JPORT [email protected]" |
There was a problem hiding this comment.
poseidon.whoi.edu: maybe replace with a generic name.
| developed a `start-notebook <https://www2.cisl.ucar.edu/resources/computational-systems/cheyenne/software/jupyter-and-ipython#notebook>`__ | ||
| utility that wraps the following steps into a single execution. You should | ||
| check with your system administrators to see if they have something similar. | ||
| If not, you'll need to take the following steps: | ||
|
|
||
| Copy this line into your terminal. It will echo a command you'll want to use | ||
| later. | ||
| If not, you can easily create your own start_jupyter script. In the script below, we choose a random port | ||
| on the server (to reduce the chance of conflict with another user), but we use port 8889 on the client, as port 8888 is | ||
| the default client port if you are running Jupyter locally. We can also change to a starting directory: | ||
|
|
||
| :: | ||
|
|
||
| (pangeo) $ echo "ssh -N -L 8888:`hostname`:8888 [email protected]" | ||
| ssh -N -L 8888:r8i4n0:8888 [email protected] | ||
|
|
||
| Now we can launch the notebook server: | ||
|
|
||
| (pangeo) $ more ~/bin/start_jupyter | ||
| cd /home/data/username | ||
| JPORT=$(shuf -i 8400-9400 -n 1) | ||
| echo "" | ||
| echo "" | ||
| echo "Step 1: Wait until this script says the Jupyter server" | ||
| echo " has started. " | ||
| echo "" | ||
| echo "Step 2: Copy this ssh command into a terminal on your" | ||
| echo " local computer:" | ||
| echo "" | ||
| echo " ssh -N -L 8889:`hostname`:$JPORT [email protected]" | ||
| echo "" | ||
| echo "Step 3: Browse to https://localhost:8889 on your local computer" | ||
| echo "" | ||
| echo "" | ||
| sleep 2 | ||
| jupyter lab --no-browser --ip=`hostname` --port=$JPORT | ||
|
|
||
| Now we can launch the Jupyter server: | ||
| :: | ||
|
|
||
| (pangeo) $ jupyter lab --no-browser --ip=`hostname` --port=8888 | ||
| (pangeo) $ ~/bin/start_jupyter | ||
|
|
||
| Step 1:... | ||
| Step 2:... | ||
| Step 3:... | ||
| ... | ||
| [I 13:36:52.321 LabApp] The Jupyter Notebook is running at: | ||
| [I 13:36:52.321 LabApp] http://r8i4n0:8888/ | ||
| [I 13:36:52.321 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). | ||
|
|
||
| Now, connect to the server using an ssh tunnel from your local machine | ||
| (this could be your laptop or desktop). | ||
|
|
||
| :: | ||
|
|
||
| $ ssh -N -L 8888:r8i4n0:8888 [email protected] | ||
|
|
||
| [I 2021-04-06 06:33:57.962 ServerApp] Jupyter Server 1.5.1 is running at: | ||
| [I 2021-04-06 06:33:57.962 ServerApp] http://pn009:8537/lab | ||
| [I 2021-04-06 06:33:57.963 ServerApp] or http://127.0.0.1:8537/lab | ||
| [I 2021-04-06 06:33:57.963 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). | ||
|
|
||
| You'll want to change the details in the command above but the basic idea is | ||
| that we're passing the port 8888 from the compute node `r8i4n0` to our | ||
| local system. Now open http://localhost:8888 on your local machine, you should | ||
| find a jupyter server running! | ||
| Just follow the Steps 1,2,3 printed out by the script to get connected. |
There was a problem hiding this comment.
📣 Shameless plug 📣 ;) : You could replace all of this with a single command via jupyter-forward - Jupyter-forward launches Jupyter Lab over SSH (on any computing cluster) with a single command !
Here are some examples:
- Launching Jupyter Lab on a remote host’s login node: https://jupyter-forward.readthedocs.io/en/latest/examples.html#launching-jupyter-lab-on-a-remote-host-s-login-node
- Launching Jupyter Lab on a remote host’s compute node: https://jupyter-forward.readthedocs.io/en/latest/examples.html#launching-jupyter-lab-on-a-remote-host-s-compute-node
There was a problem hiding this comment.
@andersy005 , what do you think about leaving these steps in here, and then saying that many of them can be automated by jupyter-forward as long as you are not on windows?
I'm hesitant to just replacing this with jupyter-forward recommendation until ncar-xdev/jupyter-forward#81 is figured out.
There was a problem hiding this comment.
@andersy005 , what do you think about just merging this and following it with something about jupyter forward as another PR?
|
I'll just go ahead and merge this one! |
fixes #825