Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for additional cloud providers #88

Closed
aculich opened this issue Jun 14, 2017 · 81 comments
Closed

Add support for additional cloud providers #88

aculich opened this issue Jun 14, 2017 · 81 comments
Labels
documentation help wanted

Comments

@aculich
Copy link
Collaborator

aculich commented Jun 14, 2017

If you're interested in support for this software on AWS, Jetstream, or other cloud providers, please let us know here... or even better, send us a Pull Request with your contributions to getting the code working on your desired cloud provider!

We so far have heard interest in supporting Jetstream using the OpenStack Magnum API, as well as using kubeadm.

We also have heard interest in supporting AWS. Here are some links provided to us by our AWS reps:

https://kubernetes.io/docs/getting-started-guides/aws/
https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/
@willingc
Copy link
Collaborator

Thanks @aculich. For those that wish to help by submitting a PR, please limit changes that are vendor/cloud provider specific to its own section within https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/doc/source/create-k8s-cluster.rst file. We would like to keep the remainder of the documentation vendor agnostic. Thanks. Please let us know if you have questions.

@willingc willingc changed the title supporting additional cloud providers Add support for additional cloud providers Jun 20, 2017
@aculich
Copy link
Collaborator Author

aculich commented Aug 7, 2017

@choldgraf and I tested Heptio based on pointers from our AWS rep, and @yuvipanda mentioned kops as a direction the open source community is moving, however it relies on having a DNS name already registered for its discovery process which can get in the way for quick testing on an IP address.

note that we also had to disable RBAC (which is not desirable in the long-term) with our Heptio install: https://kubernetes.io/docs/admin/authorization/rbac/#permissive-rbac-permissions

There is more to do.... and we'll ask for input from folks at the UCCSC AWS User Group meeting today.

@willingc
Copy link
Collaborator

willingc commented Aug 7, 2017

Nice to see work happening with Heptio, @aculich and folks.

@rdodev, do you know who would be a good contact if we have additional questions? ☀️

@rdodev
Copy link

rdodev commented Aug 7, 2017

Hey @willingc happy to help and can be point person with any questions or issues relating to our AWS quickstart.

@willingc
Copy link
Collaborator

willingc commented Aug 7, 2017

Thanks @rdodev. Good stuff happening at Heptio 😄

@choldgraf
Copy link
Member

FWIW I really need to get something like this working on AWS within a week or so...otherwise we'll need to switch to something else for the bootcamp in early September. @aculich do you have time to give it another go with me this week?

@choldgraf
Copy link
Member

@rdodev would you have a chance to do a live-chat with @aculich and I as we try to get k8s running on AWS? I'm helping teach a bootcamp to a buncha neuroscientists in early September and was hoping to run a k8s-based jupyterhub on AWS!

@willingc
Copy link
Collaborator

@choldgraf I got the heptio tutorial https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/ up and running the other day with no issues. I haven't had time to try with JupyterHub but kubectl and helm were working. Heptio's friday podcasts on YouTube are really good too. The first one basically walks you through the tutorial install.

@choldgraf
Copy link
Member

choldgraf commented Aug 10, 2017 via email

@willingc
Copy link
Collaborator

FYI. I used the new VM option FWIW.

@rdodev
Copy link

rdodev commented Aug 10, 2017

Great to see things are working as expected @willingc one thing worth highlighting is the fact that AWS QS clusters are not "production-grade" and are only meant for testing/staging. Would be glad to help productionize (sic) your environment if and when you folks are ready.

@choldgraf
Copy link
Member

I've got things running up to the point of the helm install. I followed the heptio guide and got my kubernetes machines running. Helm + kubectl are also installed. Here's the error that I'm getting:

helm install jupyterhub/jupyterhub --version=v0.4 --name=kube --namespace=kube -f config.yaml

    Error: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "kube". (get namespaces kube)

helm version
    Client: &version.Version{SemVer:"v2.5.1", GitCommit:"7cf31e8d9a026287041bae077b09165be247ae66", GitTreeState:"clean"}
    Server: &version.Version{SemVer:"v2.5.1", GitCommit:"7cf31e8d9a026287041bae077b09165be247ae66", GitTreeState:"clean"}

Any ideas?

@yuvipanda
Copy link
Collaborator

yuvipanda commented Aug 11, 2017 via email

@choldgraf
Copy link
Member

choldgraf commented Aug 11, 2017 •
edited
Loading

Oh you mean from that PR that I created and have already forgotten that I created? whoops ;-)

that fixes the namespace error...now helm is hanging on install:

helm install jupyterhub/jupyterhub --version=v0.4 --name=kube --namespace=kube -f config.yaml --debug
   [debug] Created tunnel using local port: '61697'

   [debug] SERVER: "localhost:61697"

   [debug] Original chart version: "v0.4"
   [debug] Fetched jupyterhub/jupyterhub to /home/choldgraf/.helm/cache/archive/jupyterhub-v0.4.0+fb6fc47.tgz

   [debug] CHART PATH: /home/choldgraf/.helm/cache/archive/jupyterhub-v0.4.0+fb6fc47.tgz

been stuck on that last one for like 10 minutes, ended with:

Error: timed out waiting for the condition

@choldgraf
Copy link
Member

UPDATE: I got it working by running this command: https://kubernetes.io/docs/admin/authorization/rbac/#permissive-rbac-permissions

which @yuvipanda mentions makes the cluster insecure. I think there's a better solution coming soon but just putting this here for reference

@choldgraf
Copy link
Member

OK I think I am close. Got jupyterhub deployed and everything with one snag:

It's not generating a public-facing IP address:

kubectl --namespace=kube get svc
   NAME           CLUSTER-IP      EXTERNAL-IP        PORT(S)        AGE
   hub            10.109.128.19   <none>             8081/TCP       3m
   proxy-api      10.96.110.230   <none>             8001/TCP       3m
   proxy-public   10.100.36.195   a72d589697ecd...   80:31656/TCP   3m

I'd assume that EXERNAL-IP would have a proper IP address. I wonder if this is something about how my AWS instance is set up? Do I need to configure something special to allow public access?

@yuvipanda
Copy link
Collaborator

yuvipanda commented Aug 11, 2017 via email

@choldgraf
Copy link
Member

boosh! a72d589697ecd11e7b8e202ffae2b2ec-945672095.us-west-2.elb.amazonaws.com

@choldgraf
Copy link
Member

getting PersistentVolumeChain is not bound errors...I think there's a fix for that in the guide IIRC

@yuvipanda
Copy link
Collaborator

yuvipanda commented Aug 11, 2017 via email

@choldgraf
Copy link
Member 

apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

@choldgraf
Copy link
Member

So @yuvipanda and I chatted and it seems like this could be an issue for AWS. We need users to be able to have their own disks and it looks like this isn't something that comes by default.

@willingc when you got this up and running did you figure out a way to allow for people to have disks in their jupyterhub instance? @rdodev any thoughts on how one might enable this w/ the current setup?

@rdodev
Copy link

rdodev commented Aug 12, 2017

@choldgraf I guess I'm not fully abreast what the use case architecture is for jupyterhub. Is it similar to tmpnb.org? If you have literature or diagrams would be greatly helpful.

@choldgraf
Copy link
Member

hmmm, well there's lotsa docs describing JupyterHub and the tools it utilizes here:

https://zero-to-jupyterhub.readthedocs.io/en/latest/

As an example, a common use-case is a classroom setting. A teacher puts together a Docker image that contains all the requirements/dependencies/code/data etc needed for the class, and that image is served to students via JupyterHub. When students log in, kubernetes spins up a pod for them and attaches it to a persistent disk that contains the student's files (so that they can modify their notebooks and those changes will persist in time). It sounds like we're having trouble with the persistent-disk-attaching part.

@rdodev
Copy link

rdodev commented Aug 12, 2017

@choldgraf great, thanks for the info. Let me look into it.

@rdodev
Copy link

rdodev commented Aug 12, 2017

@choldgraf are the manifest files you've used in the master branch of the repo?

@choldgraf
Copy link
Member

which repo? at this point I'm not actually working from any repo. just following the instructions post-kubernetes-install from here: https://zero-to-jupyterhub.readthedocs.io/en/latest/

(also just FYI I think that @yuvipanda will be of more help than I here, he's a lot better at debugging kube stuff)

@choldgraf
Copy link
Member

pinging you @rdodev in case you're only paying attention to parts of this thread in which you're mentioned ;-)

@rdodev
Copy link

rdodev commented Aug 14, 2017

@choldgraf no, never seen consistent failures w/ any type of instance. Those types of errors are usually on AWS' side.

@choldgraf
Copy link
Member

ok, I'll give it a shot again...

@choldgraf
Copy link
Member

hmmm...I got the same failure to create + rollback. @aculich have you experienced any issues like this on AWS before?

@rdodev
Copy link

rdodev commented Aug 14, 2017

Strange. Are you trying to launch into an existing VPC? What's the exact errors you're seeing?

@choldgraf
Copy link
Member

nope - I'm creating a new one (the button on the left in the guide). It was hard to pin down a specific error message, but it seemed like a subset of the machines being requested didn't succeed (like 3 out of 7) so the whole thing failed and rolled back...

One theory is that this is related to some kind of limit on my AWS account...not sure how to test that out though. This works fine for all the tN machines

@rdodev
Copy link

rdodev commented Aug 14, 2017 •
edited
Loading

@choldgraf A lot of people bump on this issue:

https://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2

@choldgraf
Copy link
Member

hmm - we were requesting r3.large, which isn't listed on that page, so not sure what kind of limits it has. :-/

@rdodev
Copy link

rdodev commented Aug 14, 2017

@choldgraf "All Other Instance Types | 20" this is total per region so if you have any other deployed in a different AZ will count against quota.

@choldgraf
Copy link
Member

Gotcha - yeah we were only requesting 7 so I guess this isn't the issue...hmmm, I can try and ask someone in a different part of the country to deploy w/ heptio and the same computational config

@rdodev
Copy link

rdodev commented Aug 14, 2017

Let me give it a try :D

@choldgraf
Copy link
Member

:-)

@rdodev
Copy link

rdodev commented Aug 14, 2017

Spinning up a cluster with 7 x r3.larges as we speak. Will update when done (or error).

@rdodev
Copy link

rdodev commented Aug 14, 2017 •
edited
Loading

@choldgraf
awsparams
awsresult

Region: Oregon (us-west-2)

@choldgraf
Copy link
Member

damnit!

@choldgraf
Copy link
Member

I mean.....that's great! :-)

hmmm, OK I can give it another shot with us-west-2b. This makes me wonder if it is something with my account...

@rdodev
Copy link

rdodev commented Aug 14, 2017

If your account is a child/sub account it's possible other users under the same umbrella account have VMs running in that region and are invisible to you (thus bumping on the quota).

@choldgraf
Copy link
Member

well either way, that's good news - let me send these instructions to another guy we're working with at UW and see if he can get the machines set up...I'm trying to do this so that we can use AWS + JupyterHub for a training camp in early September...so really it just needs to work for him :-)

@rdodev
Copy link

rdodev commented Aug 14, 2017

@choldgraf so it worked, I presume? Please ping me if need be. Though I'm on Eastern time so probably won't check until tomorrow morning.

@choldgraf
Copy link
Member

I still haven't got it working with r3 but it's working with the two machines... I'll let you know if my colleague can get it working. Thanks so much for your help! I'll report back w an update but either way I owe ya a 🍺 or two!

@choldgraf
Copy link
Member

hey @rdodev - I wonder if you're still around for a quick question!

First off - the AWS deployments are working quite well, I think...thanks so much for the great guide/template and all the help!

A question: somebody is asking about how to rescale thier AWS cluster after deploying (specifically the "1-20" nodes). I looked through the guide but couldn't find a clear way to do this. Do you have any intuition for how to do this?

@choldgraf
Copy link
Member

ping @arokem since he's interested in this

@rdodev
Copy link

rdodev commented Aug 28, 2017

@choldgraf looking into this. Give me 1/2 hour or so to test solution.

@rdodev
Copy link

rdodev commented Aug 28, 2017 •
edited
Loading

The most graceful way is:

  1. log into aws console and go to CloudFormation
  2. Find the stack that you want to scale out (name ends in 12 uppercase alphanumeric string, both stacks share the same prefix name)
  3. Select above mentioned stack, then from Actions menu, click on Update Stack
  4. Click Next
  5. In parameters, change value of Node Capacity to desired value.
  6. Click Next twice
  7. Confirm change and click on Update.

//cc @choldgraf

@arokem
Copy link
Contributor

arokem commented Aug 28, 2017

Thanks! I will give this a try later today. I assume that other parameters can also be changed? For example, instance type, etc.?

@rdodev
Copy link

rdodev commented Aug 28, 2017

@arokem it is possible, but that's a bit more complicated since changing instance type will nuke existing nodes and any data or workloads therein will be lost.

@yuvipanda yuvipanda mentioned this issue Mar 20, 2018
Closed
@choldgraf
Copy link
Member

Hey all - as we now have more mature docs for a number of providers, I'm going to close this. If people would like to re-open, please feel free to do so! Though I think it'll be more useful if we have issues for specific cloud providers we haven't supported, rather than one-catch all (especially since this one is quite long already!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@yuvipanda @arokem @aculich @rdodev @choldgraf @willingc