use cylc CLI and config dir

[oliver-sanders]

closes #123

• put commands behind the cylc console script
• source the base config by default
• source the user config from the .cylc dir when present

The cylc hub script just calls Jupyterhub sourcing our configuration module which loads the default and user configuration files. This command can be called from anywhere on the filesystem and appears in the cylc help output:

$ cylc help all
broadcast ............. Override [runtime] configuation items in a running suite.
cat-log ............... View Cylc suite and job log files.
check-versions ........ Check that Cylc versions match on different platforms.
...
hub ................... Launch the Cylc hub for running the Cylc Web GUI.
...

I've added a preliminary user config location for now, will need to add a site configuration too, will raise the config locations (#111) at the next meetup.

closes #161
addresses #111 (full solution requires resolution of outstanding questions)
addresses #80 (the dependence on having cylc-ui and cylc-uiserver in the same dir remains for now)

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Already covered by existing tests.
• Does not need tests (why?).
• Appropriate change log entry included.
• No documentation update required.
• No dependency changes.

[oliver-sanders]

Will need to pip install from the git repo in GH actions when implemented.

[oliver-sanders]

Note: travis dead, will need to test locally until convterted to GH Actions - #174

[dwsutherland]

Works fine for me 👍 .. Happy to merge after comment addressed

[dwsutherland]

Had to setup a link here:

(uiserver) sutherlander@cortex-vbox:python3.7$ pwd
/home/sutherlander/.envs/uiserver/lib/python3.7
(uiserver) sutherlander@cortex-vbox:python3.7$ ls -l
total 12
lrwxrwxrwx   1 sutherlander sutherlander    31 Feb 24 01:09 cylc-ui -> /home/sutherlander/cylc/cylc-ui
drwxr-xr-x 142 sutherlander sutherlander 12288 Feb 24 00:56 site-packages

I suppose there's a better way of doing this..

[oliver-sanders]

Ah, ok, I'm guessing you've not installed the UIS in editable mode so cylc.uiserver.__file__ is showing up as being within the python lib rather than the UIS working copy.

I'm not sure how we are going to install the UI into the UIS yet, I think DIST should be configurable somehow.

[oliver-sanders]

You should now be able to set the static path for your UI installation using the user config like so:

#  ~/.cylc/hub/config.py
c.Spawner.cmd = ['cylc', 'uiserver', '-s', '<path>']

[oliver-sanders]

I've now made the path to the UI files a configurable traitlet. The UIS now loads in the same config file as the hub but has it's own UIServer namespace which so far contains just one config, ui_path.

• For editable installations (pip install -e .) you do not need to set anything, it loads ../cylc-ui.
• For non-editable installations (pip install .) you will need to point the UIS at the UI files from your ~/.cylc/hub/config.py file using the c.UIServer.ui_path configuration.

I'm not sure I'm working with traitlets correctly, let me know if I'm going wrong. If it's more involved, happy to drop the last commit for now.

[kinow]

I think it didn't pick my config? (ah, now

[I 2021-02-25 09:16:02.207 JupyterHub spawner:1451] Spawning jupyterhub-singleuser --port=57197
Failed to set groups [Errno 1] Operation not permitted
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/bin/jupyterhub-singleuser", line 6, in <module>
    from jupyterhub.singleuser import main
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/jupyterhub/singleuser/__init__.py", line 5, in <module>
    from .app import main
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/jupyterhub/singleuser/app.py", line 16, in <module>
    App = import_item(JUPYTERHUB_SINGLEUSER_APP)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/traitlets/utils/importstring.py", line 30, in import_item
    module = __import__(package, fromlist=[obj])
ModuleNotFoundError: No module named 'notebook'

Does it mean users must create the config elsewhere first? I assume that means users that have a jupyterhub_config.py already in use, will have to link/copy it where cylc hub looks for, or deploy it to a folder and start jupyterhub?

[oliver-sanders]

I think Jupyterhub only ever looks for the jupyerhub_config.py in $PWD so to pick up a config you must always cd to the right place or specify the path to the config with the -f option.

If you already have a jupyterhub_config for running regular jupyterhub then your options are:

1. As there is is no global Jupyterhub location you might as well move it to the Cylc location (~/.cylc/hub/config.py).
2. Copy it to the Cylc location as configuration values are likely to differ due to the highly distributed nature of Cylc.
3. Specify it on the CLI i.e cylc hub -f /path/to/config/file.
4. Symlink it to ~/.cylc/hub/config.py (will be automatically loaded by cylc hub).
5. Import it from ~/.cylc/hub/config.py (will be automatically loaded by cylc hub).
6. Run plain jupyterhub for full control.

Not sure how you got that traceback, is that from running jupyterhub?

[kinow]

Not sure how you got that traceback, is that from running jupyterhub?

Yes, running jupyterhub raises this error. So far the only differences I could tell after this PR were having to install cylc-flow first, and not being able to use jupyterhub directly.

Running cylc hub works. I think users who want/prefer to use jupyterhub may find it confusing, and have to search for a way to extract the configuration used by Cylc for Jupyterhub (e.g. if they are deploying to k8s in a HA env following the jupyterhub guide). But we probably won't have users with that set up initially, so not a blocker IMO.

If I install with pip install . (not editable), then I think I get the same error as @dwsutherland after I launch the UIS.

2021-03-02 14:17:00,839 tornado.application ERROR    Uncaught exception GET /user/kinow/?redirects=1 (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:8000', method='GET', uri='/user/kinow/?redirects=1', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/tornado/web.py", line 1905, in wrapper
    return method(self, *args, **kwargs)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/tornado/web.py", line 3173, in wrapper
    return method(self, *args, **kwargs)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/cylc/uiserver/handlers.py", line 71, in get
    self.render(index, python_base_url=base_url)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/tornado/web.py", line 863, in render
    html = self.render_string(template_name, **kwargs)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/tornado/web.py", line 1009, in render_string
    t = loader.load(template_name)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/tornado/template.py", line 446, in load
    self.templates[name] = self._create_template(name)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/site-packages/tornado/template.py", line 477, in _create_template
    with open(path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.8/cylc-ui/dist/index.html'
2021-03-02 14:17:00,841 tornado.access ERROR    500 GET /user/kinow/?redirects=1 (127.0.0.1) 2.72ms

[oliver-sanders]

and not being able to use jupyterhub directly.

If you want to bypass cylc hub just specify the path to the config file manually:

# run the cylc hub
# NOTE: does not source your user config automatically
$ jupyterhub -f cylc/uiserver/config_defaults.py

I think users who want/prefer to use jupyterhub may find it confusing, and have to search for a way to extract the configuration used by Cylc for Jupyterhub

The jupyterhub_config.py file we have on master would not have been included with the pypi or conda-forge packages so this would only ever work for dev and was never an option for production.

In order to package the config it has to go into the source library somehow so the user was always going to have to search for the config. This way there is a command they can use which does this for them, otherwise they can locate the file using cylc.uiserver.config_defaults.__file__.

If I install with pip install . (not editable), then I think I get the same error as @dwsutherland after I launch the UIS.

For a non-editable install you will need to specify the ui path in your user config:

#  ~/.cylc/hub/config.py
c.Spawner.cmd = ['cylc', 'uiserver', '-s', '<path>']

[kinow]

If you want to bypass cylc hub just specify the path to the config file manually:

For a non-editable install you will need to specify the ui path in your user config:

Thanks!