Amazon Machine Instance (AMI) Overview
In an effort to provide users with an easier alternative to setting up and running the model via the Docker-Anaconda workflow and to leverage cloud computing resources the dGen team has developed several Amazon Machine Instances (AMIs) that will be public facing for users to copy.
Users will need to make their own AWS account to copy the AMI and will be responsible for all costs. Costs depend on several factors including:
- Size of the instance (e.g. medium or large) and storage (GB)
- Status of the instance (stopped vs started)
- Dynamic costs based on compute resources used (e.g. number of model runs, number of agents being run, number of years modeled)
Generally the cost of just hosting an instance in a stopped state will be between $0.20-1.50/day depending on the size/storage.
There are several public AMIs available for users to copy and use. These include versioned AMIs and template AMIs.
"Plug and Play" AMI (search for "osdgen_national_ami" in Community AMIs tab in AWS):
- A specific version of the open source model code and compatible database is already setup and ready to be run based on user configuration.
- Model code is cloned from a particular version of the model on github, but the code is not connected to github.
- All of the data in the database and agents needed to run the model for all geographic resolutions (e.g. national, ISO, and or state) and sector (commercial and or residential) is already present (i.e., the database is the national level database).
- Users can clone a specific fork of the model if they'd like to use an adapted version and connect it to their github.
- One size: 128GiB
"Custom" AMI (search for "osdgen_template_ami" in Community AMIs tab in AWS):
- Postgres and Anaconda are already setup and all of the agent files are present.
- Users will need to secure copy the specific database they're wanting to work with and restore it.
- Requires the user to clone their fork of the model code version so it is connected to their github if they don't want to use the base codebase. Users are responsible for making sure their fork of the model code and database are compatible.
- One size: 32GiB
Users can resize these AMIs as needed. See the instructions below for resizing AMIs.
Only two AMIs are publicly available at the moment, one with the national database restored and a template one without a restored database. Both have Anaconda, PostgreSQL, the codebase, and all agents already setup. Please reach out if you'd like a custom one made.
Navigate to https://aws.amazon.com/ and create an account if you don't already have one.
Once logged in, navigate to the top left and click on EC2 under the "Compute" service. Then click the orange "Launch instance" button.
Search for either "osdgen_national_ami" or "osdgen_template_ami" in Community AMIs tab in AWS. (see image below).
Select the box on the left and then right click the "Actions" tab and select "Copy AMI".
Next, choose a t2.medium instance and move on to configuring the instance. All default configurations can be kept as is except the Auto-assign Public IP which should be set to "Use subnet setting (Enable)" (see image below).
Next, add storage. The copied AMI should have 32 GiB of storage already (see image below), however, you can add more if desired.
Next, make a new security group (name it whatever you want) with the same specifications as pictured below.
Finally, review the instance and click "Launch" in the bottom right corner. A screen will pop up prompting you to either create or select a key pair. If you already have a key pair and the corresponding .pem file then you can use an existing one. Otherwise create a new one (see image below) and then download the key pair to your .ssh directory (you'll need to create a .ssh directory if you don't already have one on your local machine). See these further instructions for enabling the key pair on your machine as the steps differ for Mac/Linux vs Windows. https://docs.aws.amazon.com/quickstarts/latest/vmlaunch/step-2-connect-to-instance.html.
Select the box on the left and then right click the "Actions" tab and select "Copy AMI".
For more information please see the AWS documentation on copying AMIs: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/CopyingAMIs.html.
If you are the owner of the AMI and haven't already downloaded the ".pem" file from AWS then you will need to do so. Please see these instructions for how to do so: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html. It is recommended that you download this to your .ssh directory.
Once downloaded, run the following command in gitbash (Windows) or terminal (Mac) to ssh into the instance:
$ ssh -i ~/.ssh/osdgen.pem [email protected]
Where osdgen.pem is the name of your .pem file and 172.18.1.114 is the public IP address of your instance. Note, the AMI must be running in order to connect to it.
If you are the AMI owner then you can add users and give them admin privileges so that your team members can also use the AMI.
First, connect to the AMI:
$ ssh -i ~/.ssh/osdgen.pem [email protected]
Next run the following in the AMI terminal, making the username and password whatever you want:
$ sudo adduser <username>
To give the new user sudo privileges run:
$ sudo visudo
and under "# Members of the admin group may gain root privileges" add:
%<username> ALL=(ALL) ALL
Lastly the user's SSH keys need to be setup:
$ sudo su <username> // switch from ubuntu to the user's account
$ cd .. // go back/up a dir to home dir
$ cd <username> // go into the user's dir
$ mkdir .ssh // make a .ssh directory
$ chmod 700 .ssh // give permissions
$ touch .ssh/authorized_keys // makes new file named authorized_keys
$ chmod 600 .ssh/authorized_keys // give permissions
$ nano /.ssh/authorized_keys // open file
Now you'll need to copy the public RSA id for the user you're adding and paste it into the /.ssh/authorized_keys file during the nano session.
You can copy this by running $ cat /.ssh/id_rsa.pub (or wherever you've saved the id_rsa.pub file) on your local computer in a mac terminal or git bash. Once you have gotten this from the user you're adding past it into the /.ssh/authorized_keys file during the nano session and Ctrl + O to save the file.
Once these steps are completed the user can now connect to the AMI (if the AMI is running and hasn't been stopped) by running:
$ ssh <username>@<AMI_Public_IP> e.g. $ ssh [email protected]
If you have trouble with these steps please refer to the AWS documentation:
- https://aws.amazon.com/premiumsupport/knowledge-center/new-user-accounts-linux-instance/
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html
If you haven't already installed PgAdmin then you will need to do so here https://www.pgadmin.org/download/ (ignore all of the options for docker, python, os host, etc.).
In PgAdmin create a new server by right clicking on Servers > Create > Server... and ideally add something like ‘ami’ in the name. Then fill out the ‘connection’ tab and ‘ssh tunnel’ tab like in the attached screenshot. The username and password on the connection tab will both just be ‘ubuntu’. You shouldn’t need to specify a password on the ssh tunnel tab, but you will need to specify your username. If no other accounts have been added to the instance then the username will just be 'ubuntu'. If other accounts have been added then the username of the account (e.g. dgen_ts) will need to be put in this field. Select the ‘identity file’ option for authentication and then put the proper path to that file (it’s your .ssh/<filename.pem> if you're the account owner or your .ssh/id_rsa.pub file if you were added to the AMI as an additional user). See the below pictures detailing this process.
You will need to start a linux screen session by running $ screen -S <name_of_screen_session> so that the model keeps running even when you detach from the SSH connection to the instance. You can re-attach to the screen session by running $ screen -r <name_of_screen_session>. For more information on using linux screen please see https://kapeli.com/cheat_sheets/screen.docset/Contents/Resources/Documents/index.
Once you have a screen session you will run the model in the same way outlined in the README by activating the dg3n_ami environment, configuring the model inputs/parameters and supplying the proper agent file, and then by running $ python dgen_model.py.
In AWS go to services in the top left and click on EC2 under the "Compute" service. Then click the orange "Launch instance" button.
Scroll down and choose the "Ubuntu Server 20.04 LTS (HVM), SSD Volume Type" AMI. Unless making an image for a state with a small number of agents you'll need to use the "t2.medium" instance type or larger.
Network & auto IP (forthcoming)
Next add storage. You'll need to add at least 20 GiB of storage to accommodate for the Anaconda install and environment, the database, and other meta-data along with the dgen codebase. It's recommended that you allocate 32 GiB, especially if running for a larger state or an ISO region. If building the AMI to run national level scenarios then allocate 128 GiB.
Adding tags can be skipped unless your organization has specific billing IDs/organizations to use.
Configure the security group of the instance such that it looks like the below image:
Once the instance is made you can click on it in the Instances dashboard in AWS and see the "Private IPv4 addresses" address number. It will be something like 172.18.1.118. Assuming you've already downloaded the .pem file, you can simply launch a terminal window (for Mac) or git bash window (for Windows users, though PowerShell may work as well) and run: $ ssh -i ~/.ssh/osdgen.pem [email protected].
If you'd like to clone the dGen repo to the instance then you can simply run git clone https://github.com/NREL/dgen.git (or your fork of the repo) in the /home directory in the instance.
Alternatively, if you want the codebase but don't want it connected to github then you can run: $ scp -r /path/to/dgen_repo/dgen [email protected]:.. after having already run $ sudo chmod 777 . in the /home directory. This will just copy the repo directory on your local machine to the instance.
In /home run: $ sudo apt install postgis postgresql-12-postgis-3 to install postgresql and postgis.
Once installed run:
$ createuser exampleuser
$ psql -c "alter user exampleuser with password 'yourPassword'"
$ psql -U postgres -c "ALTER USER exampleuser WITH SUPERUSER;"
$ createdb dgen_db -O exampleuser
$ psql -d dgen_db
$ dgen_db=# CREATE EXTENSION postgis;
Then configure postgresql:
$ sudo nano /etc/postgresql/12/main/pg_hba.conf and add: host all all 0.0.0.0/0 md5 directly below host replication all ::1/128 md5. Press Ctrl + O and then enter to save the file. Press Ctrl + X to exit.
Then run $ sudo nano /etc/postgresql/12/main/postgresql.conf and under "Connection Settings" uncomment listen_addresses = 'localhost' and replace localhost with * (keep the single quotes around the asterisk).
Run $ sudo chmod 777 . in /home and then copy over the database.sql file to the EC2 instance using the following command: $ scp i ~/.ssh/id_rsa.pub </path_to_where_you_saved_the_db/dgen_db.sql> [email protected]:.. (don't include the '<>'). This will take a while to copy over depending on the size of the database file.
Once the database file is copied over run the following in the /home directory:
$ sudo systemctl start postgresql@12-main
$ sudo pg_ctlcluster 12 main start
$ psql -U ubuntu -d dgen_db -f dgen_db.sql
The database will take some time to restore.
In the EC2 instance terminal in the /home directory run: $ sudo apt update and then run $ sudo wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh followed by $ sudo bash Anaconda3-2020.02-Linux-x86_64.sh.
Click enter to read through the terms and conditions and type yes to agree. When prompted specify the location for anaconda3 to be /home/anaconda3.
Once the install is finished run the following commands:
$ sudo addgroup conda_group
$ sudo chgrp -R conda_group /home/anaconda3
$ sudo chmod 770 -R /home/anaconda3
$ sudo adduser <user> conda_group
Next, navigate to the ubuntu directory (cd ubuntu) and run nano .bashrc. Add the following code to the very bottom of the file
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/home/anaconda3/etc/profile.d/conda.sh" ]; then
. "/home/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/home/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
Press Ctrl + O and enter to save the .bashrc file. Press Ctrl + X to exit and then run source .bashrc (making sure you're in /home/ubuntu.
Close the terminal/connection to the instance and then log back in using the ssh command. In /home/ubuntu run conda init --all. Several packages and settings should have been modified and you'll need to close and open up the terminal/connection again.
To finalize configuring Anaconda run conda config --set ssl_verify False followed by conda update conda.
Finally, to build the environment run: $ conda env create -f /dgen/dgen_os/python/dg3n_ami.yml in the /home directory.