support vagrant AWS provider

[craigfurman]

Hi,

We've just started using this to provision swift hosts for testing a backups tool. We've got it up and running on AWS, using an ubuntu base image and your existing chef cookbooks.

The only cookbook change we had to make was adding the vagrant user if it does not already exist, to support use of community AMIs that were not explicitly produced for use as a vagrant box (such as the official Canonical Ubuntu 14.04 image).

cc @mariantalla

[clayg]

super cool! cookbook changes look fine, i'm happy/surprised you were able to make it work in this context!? I think this would be a useful addition, be happy to see it merged.

I'd like to give it a go - can you help me with some of the AWS_* env options? Can any of them be defaulted or can you provide generic examples? Maybe add some commenting to https://github.com/swiftstack/vagrant-swift-all-in-one/blob/master/localrc-template?

[bloodeagle40234]

Cool, that's what I was trying to in my repo [1]. And this looks better than mine. I'll take a time to try with this. Just a small question, once I was trying, I hit a aws provider bug that we have to remove .vagrant/aws/synced_folders for each time of provision. Has it already resolved?

1: https://github.com/bloodeagle40234/vagrant-swift-all-in-one/tree/aws

[craigfurman]

@clayg It was actually pretty easy to do because of your use of the chef solo provisioner on top of a bare ubuntu box for the virtualbox provider. We added AWS provider boilerplate and let the cookbooks do their thing :)

You should be able to get it up and running with the following:

1. Log into an AWS account
2. Go to EC2
3. Go to elastic IPs subsection, and allocate a new elastic IP
4. Go to keypairs section, and generate a new keypair. The private key will download. Chmod it to 400, and keep it somewhere
5. Go to security groups section, and create a security group. The rules are up to you. We restricted inbound traffic to only allow our office's public IP, and our CI server's IP.
6. Use existing / create new IAM credentials that have the permission to provision VMs
7. set up your environment as shown below, using resources you have just created on AWS. Note VAGRANT_BOX=dummy is to prevent a virtualbox disk image being downloaded when it is not needed. Also note the below example is for eu-west-1 region, and to switch region you will need the AMI ID of the ubuntu AMI for your nearest region.
8. vagrant up --provider aws

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export SSH_PRIVATE_KEY_PATH=PATH_TO_PRIVATE_KEY_COMPONENT_OF_AWS_KEYPAIR
export AWS_REGION=eu-west-1
export AWS_AMI=ami-b265c7c1
export AWS_INSTANCE_TYPE=m3.medium
export AWS_ELASTIC_IP=...
export AWS_KEYPAIR_NAME=...
export AWS_SECURITY_GROUPS=...
export VAGRANT_BOX=dummy

Let me know how you get on!

@bloodeagle40234 we haven't noticed that problem, we have run vagrant up --provision repeatedly with idempotent results. I'm not sure if you can spin up multiple vagrant boxes on different providers from the same vagrantfile, we ran vagrant destroy while trying to set up swift alternately on virtualbox and aws. We've been using Vagrant 1.8.1 on OS X

[clayg]

ok, I got this working with the following in my localrc.aws

export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export AWS_KEYPAIR_NAME=private
export SSH_PRIVATE_KEY_PATH=/Users/clayg/.ssh/private

export AWS_REGION=us-east-1
export AWS_AMI=ami-03d2116e
export AWS_SECURITY_GROUPS=ssh-only

export VAGRANT_BOX=dummy
export AWS_INSTANCE_TYPE=m3.medium
export AWS_ELASTIC_IP=true

I installed the plugin with:

vagrant plugin install vagrant-aws

And launched the instance with:

vagrant up --provider=aws

I'm running vagrant 1.8.4

Some things I'd like to add:

• localrc-template updated with explanations of these options
• some way to default VAGRANT_BOX=dummy, AWS_INSTANCE_TYPE=m3.medium, AWS_ELASTIC_IP=true when provider is aws
• a reference to VAGRANT_DEFAULT_PROVIDER somewhere, because I kept forgetting --provider=aws when I vagrant up.
• a link to https://cloud-images.ubuntu.com/locator/ec2/ along with some words around the ami stuff because I had a rough time finding a good ami- id? Kept getting some junk about instance types and VPC.

I think I can cram most of it into localrc-template - but all of that feels like follow on improvements.

One thing I'm a little more curious about is the mis-match between vagrant ssh logging in, and everything getting rsynced as ubuntu - but everything gettingconfigured/started as vagrant. One example of the mis-match is autodoc - when you try to run it as vagrant you can't build sphinx docs in the ubuntu owned /vagrant directory - vagrant user can't even sudo (usermod would help, but also need/want to add something like ubuntu ALL=(ALL) NOPASSWD:ALL for vagrant)? OTOH, if you run as ubuntu you can't read the pids owned by vagrant in /var/run/swift. Do you think it'd be possible to parametrize the user and configure everything as ubuntu? I'm guessing you must be doing everything as root, correct? I'm thinking if down the road it gets easier to use aws builds as the user ubuntu and we can eventually drop the vagrant user on those builds altogether - that would still work for you? If so - let's merge it!

[craigfurman]

Yes, it is a little weird that we create the vagrant user rather than parameterising the user, that is something that would be nice to add at a later date.

All those additions sound reasonable!

[clayg]

cool!

I wrote issue #40 to track fixing the vagrant user and issue #41 for adding some docs.

[craigfurman]

Cool, thanks @clayg !