Add utoronto staging hub #866

GeorgianaElena · 2021-12-02T15:44:01Z

Closes #867

Deploy the support chart for the utoronto cluster per @yuvipanda's comment.
Edit the A record for grafana.utoronto.2i2c.cloud to point to the nginx-ingress service's external IP
If everything goes ok in step 1, merge this PR and check if the deploy goes well.
Edit the DNS A record of staging.utoronto.2i2c.cloud to point to the external IP of the new hub (which is the nginx-ingress service's external IP)
Open an issue about the differences in resource allocation that we have for the utoronto hub and discuss there if we should stick with those or try use the defaults we have in the infra repo ➡️ UToronto resource limits #873

for more information, see https://pre-commit.ci

GeorgianaElena · 2021-12-02T15:48:04Z

config/hubs/utoronto.cluster.yaml

+                cpu: 0.1
+                memory: 512Mi
+              limits:
+                memory: 512Mi
+        proxy:
+          chp:
+            resources:
+              requests:
+                cpu: 0.1
+                memory: 128Mi
+              limits:
+                memory: 512Mi
+          traefik:
+            resources:
+              requests:
+                cpu: 0.1
+                memory: 256Mi
+              limits:
+                memory: 512Mi


These are some resource limits from the old utoronto config. I wasn't sure whether to still use those or go with the default values in basehub template.

IMO we should stick with the current UToronto configuration values

Mmm... can I suggest otherwise, maybe? 😉
I mean, do we know if these values are the proper ones for the current incantation of our hubs from this repo?
If we set the values without validation (and only because we inherit those values), we may be setting conditions that are not actually needed (nor optimal) and makes the whole thing more complex than it should be...
Maybe we can trace back why these values were needed and infer if we need to tweak them here as well before actually doing it?

That was my feeling too @damianavila, esp since I believe the utoronto hub was the first version of our infra.

Maybe we can trace back why these values were needed and infer if we need to tweak them here as well before actually doing it?

I will dig into it to try find out if these were tweaked afterwards or were part of the initial deployment.

Ah - I was making an assumption here, let me re-phrase my suggestion:

If they have explicitly requested those limits then we should keep them.

If they have not and this was just set by us without explicit rationale for special-casing it then we should just go with whatever our other configurations use

From this conversation it sounds like we were the ones that chose those limits, not UToronto, so if that's true then I'm good w/ option 2

Looking at the commit history, it seems that the limits were set by us.
But I'm not 100% sure this means that there wasn't a request from utoronto or a limitation observed at some point.

l would love if @yuvipanda could confirm this 👀

This is a side-by-side resource differences. Hope this makes it more easier to follow

choldgraf

This looks awesome! A couple questions to clarify:

Now that Automating deployments to Azure hubs #841 is done will this auto-deploy when we merge?
This probably doesn't quite close Deploy a new cluster for the U. Toronto community #853 , because we'll need to deploy a prod hub as well, right? Since nobody will be using this hub yet anyway, why don't we deploy both staging and prod in this PR?

choldgraf · 2021-12-02T20:57:57Z

config/hubs/utoronto.cluster.yaml

+                cpu: 0.1
+                memory: 512Mi
+              limits:
+                memory: 512Mi
+        proxy:
+          chp:
+            resources:
+              requests:
+                cpu: 0.1
+                memory: 128Mi
+              limits:
+                memory: 512Mi
+          traefik:
+            resources:
+              requests:
+                cpu: 0.1
+                memory: 256Mi
+              limits:
+                memory: 512Mi


IMO we should stick with the current UToronto configuration values

yuvipanda · 2021-12-03T10:00:18Z

config/hubs/utoronto.cluster.yaml

+    grafana:
+      ingress:
+        hosts:
+          - grafana.utoronto.2i2c.cloud


DNS change needs to be done for this

yuvipanda

This lgtm, @GeorgianaElena. You'll have to make DNS changes for staging.utoronto.2i2c.cloud to point to the IP of the nginx-ingress for this to work. But once that's done, hopefully everything (including auth!) should work.

Do deal with DNS issues, my suggested workflow is:

Run deploy-support locally to deploy nginx-ingress
Change DNS to point domains to the nginx-ingress service's external IP
Then deploy the hub after 5-10 minutes, to allow time for the DNS change to propogate

This should avoid problems with let's encrypt trying to get HTTPS working properly.

damianavila · 2021-12-03T15:03:11Z

Now that #841 is done will this auto-deploy when we merge?

AFAIK, nop, this one is using a raw kubeconfig and auto-deploy needs setting up some Service Principal. AFAIK, the terraform stuff for that was merged quite recently, so I am not sure the cluster supporting this hub actually was built before or after that PR was merged. In addition, I think there are still some permission issues to be figured out yet accordingly to this comment: #864 (comment)

GeorgianaElena · 2021-12-03T15:25:26Z

Now that #841 is done will this auto-deploy when we merge?

@damianavila is right, #841 won't work for the utoronto hub because of not enough permissions to create a service principal (#864 (review)) :( Also, the cluster was built before the PR

This probably doesn't quite close #853 , because we'll need to deploy a prod hub as well, right? Since nobody will be using this hub yet anyway, why don't we deploy both staging and prod in this PR?

I believe we should deploy the prod hub after we make sure that everything works on this staging hub. It should be a fairly straightforward task, that can happen as part of Step 2 of #638. Also, we might need to take over the prod hub URL too (like we are doing here with the staging one).

GeorgianaElena · 2021-12-03T15:39:25Z

Ah, oops! I just noticed that in the original issue #638, there this piece of context from @yuvipanda:

We now have Azure terraform support in this repo (#800) and can set up a new hub infra with that, to run in parallel with current setup. The primary infra difference will be the use of Azure File for home directories rather than the hand-spun NFS VM we have there right now - I think this is a big positive change.

I believe that NFS is enabled by default rather than Azure File, so that needs to change before merging this.

for more information, see https://pre-commit.ci

GeorgianaElena · 2021-12-06T11:31:25Z

I would like to move this forward since we have a migration due date.
I was thinking about the following plan:

Deploy the support chart for the utoronto cluster per @yuvipanda's comment.
Edit the A record for grafana.utoronto.2i2c.cloud to point to the nginx-ingress service's external IP
If everything goes ok in step 1, merge this PR and check if the deploy goes well.
Edit the DNS A record of staging.utoronto.2i2c.cloud to point to the external IP of the new hub (which is the nginx-ingress service's external IP)
Open an issue about the differences in resource allocation that we have for the utoronto hub and discuss there if we should stick with those or try use the defaults we have in the infra repo

GeorgianaElena · 2021-12-06T14:23:16Z

Since I have Yuvi's approval and we can tweak resources later, I will try a merge and check how the deploy goes 🤞

damianavila · 2021-12-06T14:44:11Z

Open an issue about the differences in resource allocation that we have for the utoronto hub and discuss there if we should stick with those or try use the defaults we have in the infra repo

Totally OK with me (but let's not forget about that discussion!! 😉 )

damianavila · 2021-12-06T14:44:45Z

Btw, this is great progress @GeorgianaElena, thanks for pushing forward on it!!

GeorgianaElena · 2021-12-06T15:10:38Z

Totally OK with me (but let's not forget about that discussion!! )

Absolutely, @damianavila!

GeorgianaElena and others added 2 commits December 2, 2021 17:41

Add utoronto staging hub

509d976

[pre-commit.ci] auto fixes from pre-commit.com hooks

6b16eae

for more information, see https://pre-commit.ci

GeorgianaElena commented Dec 2, 2021

View reviewed changes

Use the right callback url for the toronto staging hub

07ec87a

choldgraf reviewed Dec 2, 2021

View reviewed changes

yuvipanda reviewed Dec 3, 2021

View reviewed changes

config/hubs/utoronto.cluster.yaml

grafana:

ingress:

hosts:

- grafana.utoronto.2i2c.cloud

Copy link

Member

yuvipanda Dec 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DNS change needs to be done for this

yuvipanda approved these changes Dec 3, 2021

View reviewed changes

GeorgianaElena and others added 3 commits December 3, 2021 21:45

Add some missing bits about home dirs, https grafana and hub resource

91ce3a2

Fix config indentation

7d25afc

[pre-commit.ci] auto fixes from pre-commit.com hooks

db79697

for more information, see https://pre-commit.ci

GeorgianaElena merged commit 0ed463c into 2i2c-org:master Dec 6, 2021

This was referenced Dec 6, 2021

Consider the case when there is no staff id list #872

Merged

UToronto resource limits #873

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add utoronto staging hub #866

Add utoronto staging hub #866

GeorgianaElena commented Dec 2, 2021 •

edited

Loading

GeorgianaElena Dec 2, 2021

choldgraf Dec 2, 2021

damianavila Dec 3, 2021

GeorgianaElena Dec 3, 2021

choldgraf Dec 3, 2021 •

edited

Loading

GeorgianaElena Dec 3, 2021

choldgraf left a comment •

edited

Loading

choldgraf Dec 2, 2021

yuvipanda Dec 3, 2021

yuvipanda left a comment

damianavila commented Dec 3, 2021

GeorgianaElena commented Dec 3, 2021

GeorgianaElena commented Dec 3, 2021

GeorgianaElena commented Dec 6, 2021 •

edited by damianavila

Loading

GeorgianaElena commented Dec 6, 2021

damianavila commented Dec 6, 2021 •

edited

Loading

damianavila commented Dec 6, 2021

GeorgianaElena commented Dec 6, 2021

Add utoronto staging hub #866

Add utoronto staging hub #866

Conversation

GeorgianaElena commented Dec 2, 2021 • edited Loading

GeorgianaElena Dec 2, 2021

Choose a reason for hiding this comment

choldgraf Dec 2, 2021

Choose a reason for hiding this comment

damianavila Dec 3, 2021

Choose a reason for hiding this comment

GeorgianaElena Dec 3, 2021

Choose a reason for hiding this comment

choldgraf Dec 3, 2021 • edited Loading

Choose a reason for hiding this comment

GeorgianaElena Dec 3, 2021

Choose a reason for hiding this comment

choldgraf left a comment • edited Loading

Choose a reason for hiding this comment

choldgraf Dec 2, 2021

Choose a reason for hiding this comment

yuvipanda Dec 3, 2021

Choose a reason for hiding this comment

yuvipanda left a comment

Choose a reason for hiding this comment

damianavila commented Dec 3, 2021

GeorgianaElena commented Dec 3, 2021

GeorgianaElena commented Dec 3, 2021

GeorgianaElena commented Dec 6, 2021 • edited by damianavila Loading

GeorgianaElena commented Dec 6, 2021

damianavila commented Dec 6, 2021 • edited Loading

damianavila commented Dec 6, 2021

GeorgianaElena commented Dec 6, 2021

GeorgianaElena commented Dec 2, 2021 •

edited

Loading

choldgraf Dec 3, 2021 •

edited

Loading

choldgraf left a comment •

edited

Loading

GeorgianaElena commented Dec 6, 2021 •

edited by damianavila

Loading

damianavila commented Dec 6, 2021 •

edited

Loading