Update executor to use EFS, add logs

.gitignore

@@ -0,0 +1,2 @@
+data-container
+node_modules

Dockerfile

@@ -0,0 +1,20 @@
+FROM node:16-alpine
+RUN apk add python3 libpcap libpcap-dev util-linux
+COPY software/libnethogs.so.0.8.5-63-g68033bf /usr/local/lib
+COPY software/nethogs-wrapper.py /usr/local/bin 
+RUN chmod +x /usr/local/bin/nethogs-wrapper.py
+#ADD http://pegasus.isi.edu/montage/Montage_v3.3_patched_4.tar.gz /
+#RUN tar zxvf Montage_v3.3_patched_4.tar.gz && \
+#    make -C /Montage_v3.3_patched_4 && \
+#    echo "export PATH=\$PATH:/Montage_v3.3_patched_4/bin" >> /etc/bash.bashrc && \
+#    cd /Montage_v3.3_patched_4 && rm -rf Montage docs grid ../Montage_v3.3_patched_4.tar.gz
+
+ADD software/Montage.tar.gz /
+ENV PATH $PATH:/Montage_v3.3_patched_4/bin
+WORKDIR /usr/src/app
+COPY package*.json ./
+RUN npm install
+RUN npm install -g log4js
+COPY . .
+
+CMD [ "npm", "start" ]

README.md

@@ -3,51 +3,59 @@
 ## Dockerizing application
 Requirements: git, nodejs, docker
 
-git clone https://github.com/hyperflow-wms/hyperflow-awsfargate-executor.git
-
-cd hyperflow-awsfargate-executor
+Clone this repository
 
 npm install
 
-copy selected dockerfile from handler_dockerfiles folder (default is enough in most cases)
-
 docker build -t handler .
 
 ## Deploying docker image on AWS Fargate
+Make sure your AWS credentials are set up correctly (`~/.aws/credentials` exists and has proper values)
+
 Requirements: AWS CLI
 
-Create ECR repository for docker image:
-aws ecr create-repository --repository-name handler
+Create ECR repository for docker image through aws console or CLI:
+`aws ecr create-repository --repository-name handler`
+
+You can replace `handler` with other name of your choice (here and in subsequent usages)
 
 Now you need to push docker image onto ECR repository. Log in to the AWS website and navigate to the Repositories tab under the Elastic Container Service. You should able to see the repository you have just created. Next, click on the repository and click on View Push Commands to get a list of commands that you need to run to be able to push your image to ECR. Follow the steps as they are given.
 
 Example steps:
 
-aws ecr get-login --no-include-email --region eu-west-1
+`aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com`
 
-sudo docker login -u AWS -p <string> https://123456789.dkr.ecr.eu-west-1.amazonaws.com
 
-sudo docker tag handler:latest 123456789.dkr.ecr.eu-west-1.amazonaws.com/handler:latest
+`sudo docker tag handler:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/handler:latest`
 
-sudo docker push 123456789.dkr.ecr.eu-west-1.amazonaws.com/handler:latest
+`sudo docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/handler:latest`
 
 ## Create Fargate application
-First, go to this link https://eu-west-1.console.aws.amazon.com/ecs/home?region=eu-west-1#/getStarted to get started with creating a new Fargate Application.
+
+Create a Fargate cluster and EFS Volume, then add a new task definition:
+
 
 Click get started, then select custom container and configure it. Give it some name, image URI can be found in ECR repository, for example:
 
 123456789.dkr.ecr.eu-west-1.amazonaws.com/handler:latest
 
-Click edit on Task definition - set some task definition name, and create proper task execution role which enables using S3. Click save and next.
+Click edit on Task definition - set some task definition name, and create proper task execution role which enables using S3 and EFS. 
 
-Again next (we don't define load balancer).
+Under Storage add Volume of type EFS for one you created, and a mount point under `/mnt/data`
 
-Set cluster name and next and create. 
-
-On the next page, you should be able to see the status of the service you have just created. Wait for the steps to complete and then click on View Service. 
+You don't need to define anything else for now and can finish the creation. 
 
 ## Running Hyperflow with AWS Fargate
 Update your workflow so it uses awsFargateCommand function. 
-Remember to update config file.
+Remember to update config file - tasks_mapping should have a value corresponding to arn of task you just created.
+
+Afterwards you can run your workflow using `hflow run .`
 
 For more details check hyperflow page: https://github.com/hyperflow-wms/hyperflow. 
+
+
+## Automated setup
+
+To create the environment for you, you can run
+`cd setup && npm install && node index.js`
+which will install the required modules and setup the infrastructure for you.

data-provider/Dockerfile

@@ -0,0 +1,3 @@
+FROM alpine
+WORKDIR /
+CMD ["sh", "-c", "cp -rf /data/* /mnt/data/ && ls -l /mnt/data/"]