Migrate UToronto hub to our pilot-hubs repository

[yuvipanda]

Problem statement

We currently deploy the University of Toronto hub via a dedicated GitHub organization and repository (https://github.com/utoronto-2i2c/jupyterhub-deploy/), using hubploy for deployment. This pre-dates our pilot-hubs/ infrastructure which is why it is special-cased.

The problem is that, because it is on a dedicated repository / deployment infrastructure, it:

• doesn't run the same infrastructure setup as our main CI/CD, which makes it harder to understand
• is a duplication of some machinery which means we have more code to maintain
• is an information and access silo (e.g., only @yuvipanda and @GeorgianaElena could resolve issues Incident report: U. Toronto credentials expired #637 currently

Solution

We should move this infrastructure to be deployed from pilot-hubs/, the same as any other Azure deployment (see the #288 for one example). This will involve a few steps, described below

Step 1: Setup new hub Infra and prepare

We now have Azure terraform support in this repo (#800) and can set up a new hub infra with that, to run in parallel with current setup. The primary infra difference will be the use of Azure File for home directories rather than the hand-spun NFS VM we have there right now - I think this is a big positive change.

Step 1 is all the things we can do that don't involve any downtime, and can be done early on.

• Set up a repo2docker-action repository for the U. Toronto Hub #854
• Deploy a new cluster for the U. Toronto community #853
• Create a staging hub for the UToronto cluster #867
• Support nbgitpuller pulling from private repositories #851
• Copy UToronto home dirs into the new cluster #876

When these tasks are completed, I think we should have a staging hub that can be used by UToronto, but runs out of this cluster.

Step 2: Migrate production hub

This should be done between Dec 13-15.

• Migrate the U Toronto production hub to our new U Toronto cluster #868

TADA! New hub!

Step 3: Clean up any loose ends

• UToronto resource limits #873
• UToronto hub homepage customization #877
• Copy the old jupyterhub.log ➡️ gzip it, and leave it in the new hub db dir. We can do usage analysis on this later, just gotta save it.
• Figure out what to do about user placeholders ➡️ let's discuss this here Speed up node spin-up with placeholders #643?
• Update docs on how to migrate a hub
• Merge code that parallelizes rsync for copying user folders
• Setup CI/CD for UToronto hub from this repo
• Reduce the size of the UToronto Azure FIle instance, as we don't need high performance anymore
• Cleanup the old cluster and NFS VM, preserving anything we want from there

Appetite

2 sprints

The final hub transition should be completed on Dec 13th or so, which means that we should do all of the prep work needed before then. It is hard to know exactly how much work this will entail until we dig into the current U. Toronto hub setup and see what is different from our infrastructure/ setup, so we'll give 2 sprints to this process.

Information

• Champion: @GeorgianaElena
• Assistant: @yuvipanda

Issues board for this project

[choldgraf]

We've had another incident on this hub, that reminded us again that we have an imbalance of team access to this infrastructure. I think we should try and make this move ASAP or this will keep on happening.

I've added some to-dos to the top, and we can prioritize this on the backlog.

I've also reached out to the U.T people to ask if there's a time window when there will be reduced activity where we could try to make the switch.

[choldgraf]

I've heard back from U. Toronto, they said that Canadian Thanksgiving is on October 11th (a Monday), so if we wanted to, we could plan a migration at that time. Otherwise we can wait until later

They had a few questions:

• will there be a service interruption?
  • I think the answer is "yes", but likely not for long.
• how long will the work take to perform?
  • I suspect the answer is "it will take a day or two to get set up properly, but that can be front-loaded before an actual migration begins"
• what time of day works for you? It's obviously easier for us (less disruptive) if work can happen early morning or late evening (or night) eastern time.
  • We should try to do this early in the morning Toronto time, since that's when most of our team will be awake.

Questions

So two questions for you all:

1. Are my answers above correct?
2. Should we shoot for migrating this hub on or around October 11th? Or should we post-pone to later in the semester?

[damianavila]

We had a recent request from them about shared directories that are natively supported in the new infrastructure and missing from the current deployment.

[choldgraf]

I've updated this issue with some new migration steps, as well as some things that we could do sooner than later since they likely won't be as disruptive.

[choldgraf]

Update: preferred date for migration

Just FYI, it sounds like the preferred date for migration (if we assume there will be down time) would be between December 13th and 15th.

[damianavila]

Just FYI, it sounds like the preferred date for migration (if we assume there will be down time) would be between December 13th and 15th.

This is pretty close to the "winter break" I presume a lot of @2i2c-org/tech-team members will take. We should keep that in mind at the time to execute on this one, IMHO.

[jsg-uoft]

@damianavila similar for UofT. We want to get the update sorted before most of our staff take their winter break as well. Hope the timing lines up.

[choldgraf]

I think that if we can take some baby steps in this direction before doing a full migration (like setting up a repo2docker image repository) it might also reduce the uncertainty and effort around the hub migration itself)

[yuvipanda]

I've went through the old repo's config, and figured out a more concrete set of actions. I've updated the issue with a task list, one that can be done before Dec 13 and then a migration during that time.

[sgibson91]

@GeorgianaElena is going to give deploying the Azure infra a shot in this sprint
@yuvipanda will take on the nbgitpuller for private repositories aspect and write a nice blog post :)

[GeorgianaElena]

The migration happened this morning 🎉

There are still a few boxes that need to checked and I also added some additional ones in the Step 3: Clean up any loose ends section in the original issue ⬆️ as per @yuvipanda's suggestion.

[choldgraf]

Amazing - thanks for the update @GeorgianaElena and bravo on all of the progress we've made on migration already!

Do you think we are on track to have all of these done before next Wednesday? Trying to figure out if our original sprint estimate is reasonable.

I also opened up 2i2c-org/team-compass#331 to track some blog post / docs that I'd like to work on that will pull from the stuff we did here as well, so I may ask y'all for some help giving me guidance there!

[GeorgianaElena]

I also opened up 2i2c-org/team-compass#331 to track some blog post / docs that I'd like to work on that will pull from the stuff we did here as well, so I may ask y'all for some help giving me guidance there!

This is so great @choldgraf! Happy to help with anything I can.

Do you think we are on track to have all of these done before next Wednesday? Trying to figure out if our original sprint estimate is reasonable.

My response comes a bit late, sorry about that. But I believe the only things remaining to do now are these two items:

• Figure out what to do about user placeholders
• Cleanup the old cluster and NFS VM, preserving anything we want from there

I was thinking that maybe the placeholder discussion can happen as part of #643 so we can close this issue once we've deleted the old cluster and NFS VM. What do you think?

[choldgraf]

Question: are we still running the old cluster VMs?

We got some feedback from the U.Toronto folks that they saw a noticeable increase in cloud costs after the migration. I looked at the grafana but don't see anything out of the ordinary there. To that extent, a few questions:

• Could the increase in costs be because we haven't yet shut down some of the old cluster?
• Is there any reason not to just shut down all the old stuff? (e.g., as part of Archive original UofT hub repository #917)?
• Any other reason that their costs might have gone up?

[yuvipanda]

I think there are two possible causes for the cost increase:

1. Old cluster and NFS VM + disks still being up
2. Azure File maybe more expensive than our NFS VM + disks setup

(2) is hopefully not a huge component, and we can get by with just finishing up (1)

[GeorgianaElena]

@yuvipanda, are you aware of anything important that we'd need to save before destroying the cluster and NFS VM?
I remember copying over the hub logs before the winter break, but was wondering if there's anything else important left to back up?

[yuvipanda]

@GeorgianaElena nah let's bust it

[GeorgianaElena]

Thanks @yuvipanda! I will hit delete on the cluster in a few mins then the NFS VM 🚀

[choldgraf]

I believe that the old cluster has now been deleted, and I've updated the Toronto team with this information. If so, can we consider this project complete??? 🚀

[GeorgianaElena]

Yes! I just deleted the last remaining bit of the original Azure resource group! So let's close this 🎉 🚀

[choldgraf]

congrats @GeorgianaElena :-)

[damianavila]

Indeed, huge congratulations @GeorgianaElena!