Pipnize drucker

.gitignore

@@ -106,13 +106,8 @@ ENV/
 # ML Model
 model/
 
-# grpc file
-drucker_pb2.py
-drucker_pb2_grpc.py
-
 # sqlite
-db.sqlite3
-db.test.sqlite3
+*.sqlite3
 
 # Mac OS temporary file
 .DS_Store

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "drucker/grpc"]
+	path = drucker/grpc
+	url = https://github.com/drucker/drucker-grpc-proto.git
+	branch = master

.travis.yml

@@ -0,0 +1,23 @@
+# ref: https://docs.travis-ci.com/user/languages/python
+language: python
+dist: trusty
+sudo: true
+services:
+- docker
+
+matrix:
+  include:
+  - python: 3.4
+    env: TOXENV=py34
+  - python: 3.5
+    env: TOXENV=py35
+  - python: 3.6
+    env: TOXENV=py36
+  - python: 3.6
+    env: TOXENV=coverage,codecov
+
+install:
+- pip install tox
+
+script:
+- tox

README.md

@@ -10,52 +10,83 @@ https://github.com/drucker/drucker-parent
 - [Drucker-client](https://github.com/drucker/drucker-client): SDK for accessing a drucker service.
 - [Drucker-example](https://github.com/drucker/drucker-example): Example of how to use drucker.
 
-## Example
-Example is available [here](https://github.com/drucker/drucker-example).
+## Installation
+From source:
 
-## Procedures
-### Git Submodule Add
 ```
-$ git submodule add https://github.com/drucker/drucker.git drucker
-$ git submodule add https://github.com/drucker/drucker-grpc-proto.git drucker-grpc-proto
-$ cp ./drucker/template/settings.yml .
-$ cp ./drucker/template/predict.py .
-$ cp ./drucker/template/server.py .
-$ cp ./drucker/template/start.sh .
+git clone --recursive https://github.com/drucker/drucker.git
+cd drucker
+python setup.py install
 ```
 
-### When update comes
+From [PyPi](https://pypi.org/project/drucker/) directly:
+
 ```
-$ git submodule update --recursive
+pip install drucker
 ```
 
-Check the files above and if they had updates, merge them to your files.
+## Example
+Example is available [here](https://github.com/drucker/drucker-example).
 
-### Edit settings.yml
-```
-$ vi settings.yml
-```
+### Create settings.yml (Not necessary)
+Write your server configurations. The spec details are [here](./template/settings.yml)
 
-### Edit predict.py
-```
-$ vi predict.py
-```
+### Create app.py
+Implement `Drucker` class.
 
-Write the following methods.
+Necessity methods are following.
 
 #### load_model
-Loading ML model to your ML module. This method is called on the wakeup or switch model.
+ML model loading method.
+
+```python
+def load_model(self) -> None:
+    try:
+        self.predictor = joblib.load(self.model_path)
+    except Exception as e:
+        self.logger.error(str(e))
+        self.logger.error(traceback.format_exc())
+        self.predictor = None
+        if not self.is_first_boot():
+            os._exit(-1)
+```
+
+If you need to load more than two files to your ML module, you need to create a compressed file which includes the files it requires. You can load the file like the below.
+
+```python
+def joblib_load_from_zip(self, zip_name: str, file_name: str):
+    with zipfile.ZipFile(zip_name, 'r') as zf:
+        with zf.open(file_name, 'r') as zipmodel:
+            return joblib.load(io.BufferedReader(io.BytesIO(zipmodel.read())))
+
+def load_model(self) -> None:
+    try:
+        file_name = 'default.model'
+        self.predictor = self.joblib_load_from_zip(self.model_path, file_name)
+    except Exception as e:
+        self.logger.error(str(e))
+        self.logger.error(traceback.format_exc())
+        self.predictor = None
+        if not self.is_first_boot():
+            os._exit(-1)
+```
 
-Argument `model_path` is the path of a ML model. You can load the model like this.
+#### predict
+Predicting/inferring method.
 
+```python
+def predict(self, input: PredictLabel, option: dict = None) -> PredictResult:
+    try:
+        label_predict = self.predictor.predict(
+            np.array([input], dtype='float64')).tolist()
+        return PredictResult(label_predict, [1] * len(label_predict), option={})
+    except Exception as e:
+        self.logger.error(str(e))
+        self.logger.error(traceback.format_exc())
+        raise e
 ```
-self.predictor = joblib.load(model_path)
-```
-
-We recommend the architecture of "1 Drucker loads 1 file" but sometimes your module needs a several files to load. In that case you need to create a compressed file including the files it requires. `model_path` will be your compressed file and you decompress it by yourself.
 
-#### predict
-Predicting or inferencing from the input. The definitions of input or output are described below. `bytes` can be a byte data of a file.
+Input/output specs are below.
 
 ##### Input format
 *V* is the length of feature vector.
@@ -81,7 +112,9 @@ The "option" field needs to be a json format. Any style is Ok but we have some r
 |option |string |Option field. Must be json format. |
 
 #### evaluate (TODO)
-Evaluating the precision, recall and f1value of your ML model. The definitions of input or output are described below.
+Evaluating method.
+
+This method is under construction.
 
 ##### Input format
 |Field |Type |Description |
@@ -99,36 +132,83 @@ Evaluating the precision, recall and f1value of your ML model. The definitions o
 |recall<BR>(required) |double[*N*][*M*] |Recall. |
 |fvalue<BR>(required) |double[*N*][*M*] |F1 value. |
 
-### Edit server.py
-```
-$ vi server.py
-```
+### Create server.py
+Create a boot script.
 
-Since `drucker_pb2_grpc` is automatically generated from `drucker.proto`, you don't need to care about it. You need to implement the interface class of `SystemLoggerInterface` and `ServiceLoggerInterface` if you customize the log output.
+```python
+from concurrent import futures
+import grpc
+import time
 
-### Edit start.sh
-```
-$ vi start.sh
-```
+from drucker import DruckerDashboardServicer, DruckerWorkerServicer
+from drucker.logger import JsonSystemLogger, JsonServiceLogger
+from drucker.protobuf import drucker_pb2_grpc
+from app import MyApp
+
+_ONE_DAY_IN_SECONDS = 60 * 60 * 24
+
+
+def serve():
+    app = MyApp("./settings.yml")
+    system_logger = JsonSystemLogger(app.config)
+    service_logger = JsonServiceLogger(app.config)
+    system_logger.info("Wake-up drucker worker.")
 
-Write the necessity script to boot your Drucker service. Minimum requirement is below.
+    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
 
+    drucker_pb2_grpc.add_DruckerDashboardServicer_to_server(
+        DruckerDashboardServicer(logger=system_logger, app=app), server)
+    drucker_pb2_grpc.add_DruckerWorkerServicer_to_server(
+        DruckerWorkerServicer(logger=service_logger, app=app), server)
+    server.add_insecure_port("[::]:{0}".format(app.config.SERVICE_PORT))
+    server.start()
+    try:
+        while True:
+            time.sleep(_ONE_DAY_IN_SECONDS)
+    except KeyboardInterrupt:
+        system_logger.info("Shutdown drucker worker.")
+        server.stop(0)
+
+
+if __name__ == '__main__':
+    serve()
 ```
-pip install -r ./drucker-grpc-proto/requirements.txt
-sh ./drucker-grpc-proto/run_codegen.sh
 
-python server.py
+### Create logger (Not necessary)
+If you want to use your own format logger, please implement the drucker [logger interface class](./drucker/logger/logger_interface.py).
+
+### Create start.sh
+Create a boot script.
+
+```sh
+#!/usr/bin/env bash
+
+ECHO_PREFIX="[drucker example]: "
+
+set -e
+set -u
+
+echo "$ECHO_PREFIX Start.."
+
+pip install -r requirements.txt
+python ./server.py
+
 ```
 
-### Run it!
+### Run
 ```
 $ sh start.sh
 ```
 
+### Test
+```
+$ python -m unittest drucker/test/test_worker_servicer.py
+```
+
 ## Drucker on Kubernetes
-Drucker dashboard makes it easy to launch Drucker service on Kubernetes.
+Drucker can be run on Kubernetes and can be managed by Drucker dashboard.
 
-You must read followings.
+You must read the followings.
 
 1. https://github.com/drucker/drucker-parent/tree/master/docs/Installation.md
 1. https://github.com/drucker/drucker-dashboard/README.md

drucker/__init__.py

@@ -0,0 +1,20 @@
+# Copyright 2018 The Drucker Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+__project__ = 'drucker'
+__version__ = "0.4.0"
+
+from .drucker_worker import Drucker
+from .drucker_worker_servicer import DruckerInput, DruckerOutput, DruckerWorkerServicer
+from .drucker_dashboard_servicer import DruckerDashboardServicer