-
-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature request] Support several Docker configurations in one spotty.yaml (to better support auxiliary on-demand instances) #44
Comments
Hi Vadim, Thank you for this feature request, I actually had a similar problem before. Sometimes you just want to run a CPU instance to do some work that doesn't require GPU, for example, analyze your results with Jupyter notebooks, and then you may need a different image that was compiled for CPU. I was thinking to extend the configuration file to be able to specify several containers and then use them for the instances. So that instead of the
And then in the instance parameters, you can redefine the default container:
As for your case: by design, you're actually not supposed to do any work from the host OS :). All the work should be done through your custom environment - a Docker container. And the I believe (but need to double-check) that even a Did you try to use some lightweight container with a P.S. Next 2 weeks I'm on vacation, so will be able to give proper thought and work on this feature only when I come back :). Best regards, |
@apls777 Thanks for the detailed response! I did try using a smaller instance. It even managed to boot up with ubuntu:18.04 container and Deep Learning AMI. Unfortunately it was unbearable slow (too little memory, I guess). Even with GPU instances, Docker sometimes takes 5-10 minutes to start-up (DockerReadyWait event). For this usecase of launching a light CPU instance, probably an option to launch a regular On-Demand instance would be useful (downloading a huge dataset on EBS can take many hours). A few other points I noticed: for some reason you do not support Nitro-based instances. I commented out the check and everything still worked OK. Also the list of instance types that you hardcoded became out-of-date, some instance types are missing. Also sometimes, Docker just wouldn't start (no useful error messages on CloudFormation). As a remedy I disabled the docker EBS volume, and things run normally then (for the CPU instance it's probably OK). |
Was it too slow to work with or it took a lot of time to start-up?
I think it's a "bug" that I found recently as well. If you have a lot of files on one of your EBS volumes, it may take time to start. When Spotty is mounting volumes, it's changing ownership of all files from The instance should start-up much faster.
Nitro-based instances have different device names for attached EBS volumes: see here vs here. At the moment, a Nitro-based instance would fail to start if you attached any volume. So I decided to disable this functionality at all for now, but it's in the TODO list.
Thanks for pointing out, I'll try to load this list dynamically using AWS API, otherwise, it's difficult to keep it up-to-date.
Can it be the issue with changing ownership that I described above? |
The t2.micro machine took many minutes to start a Docker and then ssh was super unresponsive. Bigger machines also took long time to start Docker, but then there were no problems with ssh. About Docker not starting, it's hard to say, CloudFormation log just spits some "unique error ID" which is ungoogleable without an additional error message. I'll try commenting out the |
Also, do you use your custom Dockerfile or already built image? Because if you're using a Dockerfile, Spotty builds this image every time it starts an instance. Then you may want to consider caching it using a dedicated EBS volume: see here. |
Prebuilt image: just |
Okay, thanks. I'll check what I can do about micro instances once I'll come back from vacation.
If you have issues when the instance is launched, but Docker container is not started for some reason, try to connect to the host OS using |
Thanks! The errors I referred to were from the CloudFormation web console, maybe the local logs have more information. |
It would be super useful if the CloudFormation logs were automatically downloaded and offered to the user (no hassle with manual ssh'ing as in #48) |
Btw removing My fork is at https://github.com/vadimkantorov/spotty |
Hi @apls777! Any news about this one? |
Hi @vadimkantorov, Unfortunately, I don't have time to work on this feature at the moment, but maybe I will have it in April. |
Added support for multiple container configurations. |
Downloading the dataset on a EBS volume can take many hours. I don't want to use a GPU machine for that.
Unfortunately, currently I cannot instruct spotty to not run any Docker image or specify a Docker image per instance (the main Docker image fails to start on the t2.micro machine).
Currently I'm using a separate
spotty_preprocess.yaml
to achieve this goal.The text was updated successfully, but these errors were encountered: