Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pilot hubs to infrastructure #107

Merged
merged 2 commits into from
Oct 18, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ This repository serves as the user-facing documentation and communication space

**Note: This documentation is a work in progress, so please do not hesitate to provide feedback [by opening an issue](https://github.com/2i2c-org/pilot/issues/new).

Most of the infrastructure that we discuss in the documentation is deployed [in the `pilot-hubs/` repository](https://github.com/2i2c-org/pilot-hubs).
Most of the infrastructure that we discuss in the documentation is deployed [in the `infrastructure/` repository](https://github.com/2i2c-org/infrastructure).

See [the pilot documentation](https://2i2c.org/pilot) for more information.
4 changes: 2 additions & 2 deletions about/distributions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,8 @@ If you wish to access data that exists outside of your 2i2c Hub, it is your resp

## Where are hubs configured and deployed?

All of the configuration and deployment scripts for the 2i2c JupyterHub can be found at [the `pilot-hubs/` repository](https://github.com/2i2c-org/pilot-hubs). This repository contains both the deployment code as well as documentation that explains how it works. It should be treated as "for advanced users only", and is provided for transparency and as a guide for the community to follow if they wish to manage their own infrastructure similar to 2i2c JupyterHub.
All of the configuration and deployment scripts for the 2i2c JupyterHub can be found at [the `infrastructure/` repository](https://github.com/2i2c-org/infrastructure). This repository contains both the deployment code as well as documentation that explains how it works. It should be treated as "for advanced users only", and is provided for transparency and as a guide for the community to follow if they wish to manage their own infrastructure similar to 2i2c JupyterHub.

To learn about how the `pilot-hubs/` repository works, we recommend checking out the [`pilot-hubs` documentation](ph:index).
To learn about how the `infrastructure/` repository works, we recommend checking out the [`infrastructure` documentation](infra:index).

See the next sections for more information about each hub distribution.
4 changes: 2 additions & 2 deletions about/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ Below we describe major initiatives that are currently active in the Managed Hub
## Hub infrastructure launch

The Managed Hubs Service relies heavily on infrastructure that centralizes the configuration and deployment of many JupyterHub instances.
Our first major project is to use [our Pilot JupyterHubs](https://pilot-hubs.2i2c.org/en/latest/reference/hubs.html) to drive development on this infrastructure stack.
Our first major project is to use [our Pilot JupyterHubs](https://infrastructure.2i2c.org/en/latest/reference/hubs.html) to drive development on this infrastructure stack.

:::{note}
We are also [using an issue](https://github.com/2i2c-org/pilot-hubs/issues/610) to track long-term infrastructure needs for this service across all cloud providers.
We are also [using an issue](https://github.com/2i2c-org/infrastructure/issues/610) to track long-term infrastructure needs for this service across all cloud providers.
That is more comprehensive and bigger in scope than any one initiative.
:::

Expand Down
4 changes: 2 additions & 2 deletions about/strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ We'll focus on the following cloud providers:

In the short term, we favor deploying hubs on Google Cloud Platform.
This is because GCP has the most stable Kubernetes offering of all of the cloud providers.
We follow [team guidelines for when to deploy new Kubernetes clusters](ph:cluster:when-to-deploy).
We follow [team guidelines for when to deploy new Kubernetes clusters](infra:cluster:when-to-deploy).
For new hubs that don't require their own Kubernetes cluster, we plan to run them on Google Cloud until our team has capacity to run more infrastructure across Azure and AWS.

### Why Jupyter and JupyterHub?
Expand Down Expand Up @@ -104,4 +104,4 @@ Below are a few major aspects of the service that we believe provide a good chan
All of the work done in this pilot should be open and public by default, leveraging workflows that are common to open source communnities.
We will need to create some private channels for communication for conversations with sensitive or private information, but will strive to do everything that we can in public.

Currently, all of our deployment infrastructure and development can be found at [the `pilot-hubs/` repository](https://pilot-hubs.2i2c.org).
Currently, all of our deployment infrastructure and development can be found at [the `infrastructure/` repository](https://infrastructure.2i2c.org).
2 changes: 1 addition & 1 deletion admin/howto/control-user-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,6 @@ By default, kernels will be checked for activity **every `5 minutes`**.
All kernels that haven't shown activity in **in the last hour** will be stopped by the [jupyterhub-idle-culler](https://github.com/jupyterhub/jupyterhub-idle-culler).

This window can be configured if you'd like to change the window of inactivity needed before user kernels will be stopped.
See the [Hub Engineer's guide](ph:configure:culling) for some documentation on this.
See the [Hub Engineer's guide](infra:configure:culling) for some documentation on this.

% TODO: Add link to SRE guide on how to configure this, once it exists
8 changes: 4 additions & 4 deletions admin/howto/environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ without having to install packages themselves.

## Default user environment

The default environment for all pilot hubs is defined [in this
folder](https://github.com/2i2c-org/pilot-hubs/tree/master/images/user).
The default environment for all community hubs is defined [in this
folder](https://github.com/2i2c-org/infrastructure/tree/master/images/user).
It is configured with the following:

- Python packages defined in [this `requirements.txt`
file](https://github.com/2i2c-org/pilot-hubs/blob/master/images/user/requirements.txt). Many common scientific python packages are installed here.
file](https://github.com/2i2c-org/infrastructure/blob/master/images/user/requirements.txt). Many common scientific python packages are installed here.
- R packages installed from [this `install.R`
file](https://github.com/2i2c-org/pilot-hubs/blob/master/images/user/install.R).
file](https://github.com/2i2c-org/infrastructure/blob/master/images/user/install.R).
- Many popular data science user interfaces installed:
- [Classic Jupyter Notebook](https://github.com/jupyter/notebook/)
- [JupyterLab](https://github.com/jupyterlab/jupyterlab/)
Expand Down
4 changes: 2 additions & 2 deletions admin/howto/manage-users.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Users can prove who they are by logging in via an *authentication provider*. Cur

3. Username / Password via [auth0](https://auth0.com/docs/connections/database).
A traditional username / password interface where users can sign up. There are currently [limited
options](https://github.com/2i2c-org/pilot-hubs/issues/421) for limiting who
options](https://github.com/2i2c-org/infrastructure/issues/421) for limiting who
can sign up, so this should be only used in limited circumstances.

4. `<a different provider>`. We may be able to support other authentication providers, depending on your specific needs and the provider's complexity. Please reach out to us if none of these 3 work for your use-case.
Expand All @@ -40,7 +40,7 @@ Authorizing regular users
educational institution.

Authorizing admin users
: Admin users are authorized [in a hub's YAML config](https://github.com/2i2c-org/pilot-hubs/blob/c1d06be1eed2d748a4d39e4cba76436cffe89fb2/config/hubs/2i2c.cluster.yaml#L50-L55), with support from 2i2c staff.
: Admin users are authorized [in a hub's YAML config](https://github.com/2i2c-org/infrastructure/blob/c1d06be1eed2d748a4d39e4cba76436cffe89fb2/config/hubs/2i2c.cluster.yaml#L50-L55), with support from 2i2c staff.

% TODO: Link to SRE docs on how to do this once we have it

Expand Down
4 changes: 2 additions & 2 deletions admin/howto/new-hub.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ In order to set up a new JupyterHub with 2i2c, there are a few steps that you sh

The Community Representative is the main point of contact between the community and the 2i2c Engineering Team.
There can be up to two Community Representatives per hub.
See [the pilot hubs documentation](tc:roles:community-representative) for more details on this role.
See [the infrastructure documentation](tc:roles:community-representative) for more details on this role.

## Fill in the "New Hub" GitHub template

We use a GitHub issue template to ask a few questions about your hub deployment that will help us deploy it.
Click the button below to go to the form:

```{button-link} https://github.com/2i2c-org/pilot-hubs/issues/new?assignees=&labels=type%3A+hub&template=new-hub.yml&title=New+Hub%3A+%3CHub+name%3E
```{button-link} https://github.com/2i2c-org/infrastructure/issues/new?assignees=&labels=type%3A+hub&template=new-hub.yml&title=New+Hub%3A+%3CHub+name%3E
:color: primary
Go to new Hub Form
```
Expand Down
6 changes: 3 additions & 3 deletions admin/howto/replicate.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,15 +105,15 @@ Below we'll cover how you can deploy your own JupyterHub using your 2i2c Jupyter
2i2c JupyterHubs use the [Zero to JupyterHub](https://z2jh.jupyter.org) guide for their configuration and deployments.
We recommend familiarizing yourself with it, as it will be invaluable in helping you navigate how to run a JupyterHub that replicates the 2i2c JupyterHub service.

All of the configuration for a 2i2c JupyterHub exists at the [`pilot-hubs/` repository](ph:index). This is a "meta" repository that centralizes configuration and deployment of many 2i2c JupyterHubs.
All of the configuration for a 2i2c JupyterHub exists at the [`infrastructure/` repository](infra:index). This is a "meta" repository that centralizes configuration and deployment of many 2i2c JupyterHubs.

There are two main things you'll need from this repository to deploy your hub:

1. **Your hub-specific configuration**. `pilot-hubs/` contains [configurations for each JupyterHub in YAML files](https://github.com/2i2c-org/pilot-hubs/tree/master/config/hubs) (one YAML file per cluster).
1. **Your hub-specific configuration**. `infrastructure/` contains [configurations for each JupyterHub in YAML files](https://github.com/2i2c-org/infrastructure/tree/master/config/hubs) (one YAML file per cluster).
This file has an entry for each hub, with contains a **Zero to JupyterHub configuration** for your hub. You should find this configuration under `config/jupyterhub:`.
2. **Your hub template configuration**. In addition to your hub-specific configuration, your hub also has a "template configuration" that defines the basic setup of your hub infrastructure.
Each template has a name (e.g., `dask-hub`) and maps onto a Helm configuration.
You can [find each template in this folder](https://github.com/2i2c-org/pilot-hubs/tree/master/hub-templates).
You can [find each template in this folder](https://github.com/2i2c-org/infrastructure/tree/master/hub-templates).
Look for the `values.yaml` file, as this contains the actual template configuration that you'll use with your Zero to JupyterHub configuration.

You should merge these two configuration files into a single one, for use later.
Expand Down
2 changes: 1 addition & 1 deletion conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@

intersphinx_mapping = {
"tc": ('https://team-compass.2i2c.org/en/latest', None),
"ph": ('https://pilot-hubs.2i2c.org/en/latest', None),
"infra": ('https://infrastructure.2i2c.org/en/latest', None),
"jb": ('https://jupyterbook.org', None),
"z2jh": ('https://z2jh.jupyter.org/en/latest', None),
}
Expand Down
2 changes: 1 addition & 1 deletion index.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ admin/howto/create-billing-account
The 2i2c Team DevOps guide is what we use to document how to develop and operate the cloud infrastructure that is run by 2i2c.
It contains all the details about how our infrastructure is deployed, how we make changes to it, and team processes around ensuring site reliability.

[**Our DevOps Guide**](https://pilot-hubs.2i2c.org)
[**Our DevOps Guide**](https://infrastructure.2i2c.org)


## Our Team Compass
Expand Down