Clouds

Table of Contents

• Clouds
  • Table of Contents
  • Overview
  • Features
  • Requirements
  • Installation
    • 1. Clone the repository
    • 2. Change directory into repository folder
    • 3. Setup AWS credentials for artifact download from S3
    • 4. Install required packages (required for local implementation)
  • Usage
    • 1. Local
      • Pipeline only
      • Unit Test
    • 2. Docker
      • Pipeline only
        • Build the Docker image
        • Run the application
      • Unit Test
        • Build the Docker image for unit test
        • Run the tests
  • Customization
    • AWS

Overview

This repository contains the Clouds project, an interactive web application built with Streamlit, which allows users to select and run different versions of trained machine learning models. This document includes a guide for setting up, installing, retrieving models, setting up the Streamlit application, usage, customization, and troubleshooting.

Features

The main feature of this project is the Streamlit-based web application, which allows users to load, select and run different versions of trained machine learning models. The application also includes a comprehensive testing framework, logging and error handling mechanisms to ensure a smooth and robust user experience. Each module can be easily customized to accommodate specific requirements or preferences using config.yaml. The entire application and its unit tests can be run inside a docker container. The application has also been deployed on AWS ECS, but has been stopped mainly for cost reasons.

Requirements

• Python 3.7 or higher
• Streamlit
• Docker
• AWS credentials
• See requirements.txt.
• The application assumes that an S3 bucket with trained model objects and training data has already been setup. It will not set one up for you.

Installation

1. Clone the repository

git clone https://github.com/MSIA/423-2023-hw3-hwl6390.git

2. Change directory into repository folder

cd 423-2023-hw3-hwl6390

3. Setup AWS credentials for artifact download from S3

This guide assumes you have installed the AWS CLI. If you have not configured an AWS profile, run the following.

aws configure sso --profile my-sso

For the purposes of this guide, the name of the AWS profile will be my-sso. The user can name it however they like.

After configuring the sso, run the following to login.

aws sso login --profile my-sso

After logging in, export the profile as an environment variable.

export AWS_PROFILE=my-sso

If you run aws configure list and are able to see my-sso in the list of profiles, the environment variable has been set correctly.

4. Install required packages (required for local implementation)

pip install -r requirements.txt

Usage

1. Local

Application only

Verify you are in the same directory as main.py. Then, run

streamlit run main.py

in the terminal.

Unit Test

Run

pytest

in the terminal.

2. Docker

Application only

Build the Docker image

docker build -t clouds-app -f dockerfiles/docker-main .

Run the application

docker run -p 80:80 -v ~/.aws:/root/.aws -e AWS_PROFILE=my-sso clouds-app

Unit Test

Build the Docker image for unit test

docker build -t unittest-cloud -f dockerfiles/docker-test .

Run the tests

docker run unittest-cloud

Customization

To customize settings within the pipeline, edit config.yaml.

AWS

Modify aws section of config.yaml to achieve desired bucket name and prefixes.