ARTYINS architecture

This is the main module for the ARTYINS architecture. ArtyINS

• 

Database

• 

Job Service

• 

Extraction Service

• 

Classifier Service

• 

Saver Service

• 

Monitor Service

• 

---

Table of Contents

• Overview
• Installation
• Running the application
• Using the application

---

Overview

The whole set of application is held by several submodules. Each module has their functions and serves to fulfill specific functions.

Monitor

The monitor serves to monitor the files on the target folder. Upon detecting new files, it would request the JobService to create a new job, at the same time moving the file to processing folder. When a positive reply is received from the JobService, it would move the file to success folder, otherwise, it would be sent to the fail folder.

JobService

The JobService is the main controller of the application. When it receives a new request from the Monitor, it would send a request to the following services in this order ExtractionService, ClassifierService, SaveService. Finally, it would reply to the Monitor on the status of the job.

ExtractionService

The ExtractionService receives requests to extract content from a file. It requires access to the folder where the file is held. Upon extraction of the content, it sends the content back to the caller.

ClassifierService

The ClassifierService receives requests to classify a set of text. Upon classification, it sends the classified Class to the caller.

SaveService

The SaveService receives requests to interact with a storage. Upon saving, it sends a json string which will specify the errors, if any.

Installation

The general idea is as follows

1. Either download the pre-made docker images or build the docker images yourself
2. Clone this project
3. Transfer this project and the docker images to a non-internet connected environment.
4. Run docker-compose up

1a: Downloading pre-made docker images. Internet required.

All docker images are available in quay.io/jax79sg/. You may run the following commands to download the images. After running the commands, you should see docker images such as artyins-database, artyins-jobservice..etc in your local docker repository.

docker pull quay.io/jax79sg/artyins-jobservice
docker tag quay.io/jax79sg/artyins-jobservice artyins-jobservice
docker pull quay.io/jax79sg/artyins-extractionservice
docker tag quay.io/jax79sg/artyins-extractionservice artyins-extractionservice
docker pull quay.io/jax79sg/artyins-classifierservice
docker tag quay.io/jax79sg/artyins-classifierservice artyins-classifierservice
docker pull quay.io/jax79sg/artyins-saveservice
docker tag quay.io/jax79sg/artyins-saveservice artyins-saveservice
docker pull quay.io/jax79sg/artyins-monitor
docker tag quay.io/jax79sg/artyins-monitor artyins-monitor
docker pull mysql:5.7

1b: Rebuilding the images. Internet required

Alternatively, you may rebuild the images from scratch. After running the commands, you should see docker images such as artyins-database, artyins-jobservice..etc in your local docker repository.

git clone https://github.com/jax79sg/artyins-database
git clone https://github.com/jax79sg/artyins-jobservice
git clone https://github.com/jax79sg/artyins-extractionservice
git clone https://github.com/jax79sg/artyins-classifierservice
git clone https://github.com/jax79sg/artyins-saveservice
git clone https://github.com/jax79sg/artyins-monitor

cd /artyins-jobservice
./rebuild.sh
cd ../artyins-extractionservice
./rebuild.sh
cd ../artyins-jobservice
./rebuild.sh
cd ../artyins-classifierservice
./rebuild.sh
cd ../artyins-saveservice
./rebuild.sh
cd ../artyins-monitor
./rebuild.sh

Transfer to offline env

Copying to USB Drive, assuming that the USB disk is on /media/myusbdrive

git clone https://github.com/jax79sg/artyins
cp -r artyin /media/myusbdrive/

docker save mysql:5.7 -o /media/myusbdrive/mysql.tar
docker save artyins-jobservice -o /media/myusbdrive/artyins-jobservice.tar
docker save artyins-extractionservice -o /media/myusbdrive/artyins-extractionservice.tar
docker save artyins-classifierservice -o /media/myusbdrive/artyins-classifierservice.tar
docker save artyins-saveservice -o /media/myusbdrive/artyins-saveservice.tar
docker save artyins-monitor -o /media/myusbdrive/artyins-monitor.tar

Copying to the offline computer, asuming at /home/user/

cp -r /media/myusbdrive/* /home/user/

Running the application

Upon running the following commands, you will see scrolling logs.

cd /home/user/artyins
./runartyins.sh

Using the application

The users can simply copy their raw reports in PDF into the /home/user/artyins/shareddata/new folder.
When the system pick up the files, it will be moved into the /home/user/artyins/shareddata/processing folder
Depending on outcomes, the file will eventually be moved to /home/user/artyins/shareddata/success or /home/user/artyins/shareddata/fail folders.

All results can be accessed via the mysqldb exposed under port 3306, with following parameters. More information on the database schema can be found in http://github.com/jax79sg/artyins-database

username='user'
password='password'
database='reportdb'
tables = 'reports', 'ingests'

---