config: automatic manifest generation #18

diegodelemos · 2017-02-10T10:03:48Z

(addresses tests: prepare local development test cluster reana#13)
(closes config: make possible to generate manifests automatically #17)

Signed-off-by: Diego Rodriguez [email protected]

tiborsimko · 2017-02-15T08:44:47Z

.gitignore

+
+# Built config
+configuration_manifests/


configuration-manifests as we usually use dashes instead of underscores?

tiborsimko · 2017-02-15T08:46:11Z

reana_resources_k8s/templates/config.yaml

@@ -0,0 +1,126 @@
+---
+# Deployment [local|prod]


This file could be documented (via comments) just a little bit more and then included under "Configuration" section in the documentation. Shall we do it now or later?

Maybe we should do it in a separate PR addressing reanahub/reana#4 so we have the functionality shipped faster, WDYT?

tiborsimko · 2017-02-15T08:50:03Z

scripts/build-manifests.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+


The scripts misses license statement. There are several shell script improvements to do, such as:

do not use $1 throughout the file, rather read the arguments in the beginning into variables and use those;

semicolons at the end of statements are not needed if there is no follow-up statement to execute

see also shellchech recommendations such as cd $somedir || exit

Shall we do these now or via a later cleanup?

I think this can be done now, so we do not have to change it until we create the Python CLI.

tiborsimko · 2017-02-15T08:55:41Z

setup.py

@@ -84,6 +84,7 @@
    extras_require=extras_require,
    setup_requires=setup_requires,
    tests_require=tests_require,
+    scripts=['scripts/build-manifests.sh'],


Do we need to install this script? If it gets installed under /some/prefix/bin/build-manifests.sh, then we should perhaps think of a name like reana-build-manifests... Let's muse IRL.

* (addresses reanahub/reana#13) Signed-off-by: Diego Rodriguez <[email protected]>

Signed-off-by: Diego Rodriguez <[email protected]>

* Since config for namespaces, quotas and workers are stored in a single jinja template, the output was a single file with many yaml files inside. This commit generates a file for each yaml. Signed-off-by: Diego Rodriguez <[email protected]>

* In order to make `build-manifests.sh` accesible when installing the package, both the script and the `templates` directory have been added to it. Signed-off-by: Diego Rodriguez <[email protected]>

Signed-off-by: Diego Rodriguez <[email protected]>

tiborsimko · 2017-02-17T10:05:04Z

docs/gettingstarted.rst

+
+Once we have done it, we should be able to see the directory `configuration-manifests` with the following structure:
+
+.. code-block:: sh


Since you are showing more than just a shell script source, it's better to use code-block:: console here. (And everywhere where you show a console session, i.e. shell commands and their output.)

tiborsimko · 2017-02-17T10:09:01Z

docs/gettingstarted.rst

+
+.. code-block:: sh
+
+   $ python setup.py install


We can be ready towards future release by installation instructions going like:

First, install latest stable release of `reana-resources-k8` package: pip install reana-resources or use development version: git clone ... cd ... pip install . Second, you can build resource manifests by doing... ...

tiborsimko · 2017-02-17T10:12:21Z

scripts/reana-resources-k8s

@@ -0,0 +1,73 @@
+#!/bin/sh
+


Please add copyright header information, e.g. take it from .travis.yml.

tiborsimko · 2017-02-17T10:17:07Z

scripts/reana-resources-k8s

+        template_file_name=$(printf '%s' "$template_file_path" | sed -e "s@$templates_path\/@@g")
+        config_file_name=$(printf '%s' "$template_file_name" | sed -e "s/-template//g")
+        config_file_path="$templates_path/../configuration-manifests/$config_file_name"
+        jinja2 "$template_file_path" "$templates_path/config.yaml" > "$config_file_path"


jinja2-cli is missing in setup.py as a required dependecy. Please add it.

... ditto pyyaml is missing. Please add.

tiborsimko · 2017-02-17T10:20:10Z

docs/gettingstarted.rst

+.. code-block:: sh
+
+   $ python setup.py install
+   $ reana-resources-k8s build-manifests


Please add reana-resources-k8s build-manifests step to the Travis CI building instructions. Otherwise we won't catch errors in shell scripts... (such as missing pyyaml dependency I talk about elsewhere).

BTW shall we use "build-manifests" terminology? Wondering because we usually use "create-instance" etc...

tiborsimko · 2017-02-17T10:28:23Z

docs/configuration.rst

+=============
+
+
+Configuration for the REANA system can be made through the file `config.yaml` under `templates`:


Better use a language iike:

REANA-Resources-K8 default congifuration is defined in templates/config.yaml file. It is printed below for inspiration. You can define your own myconfig.yaml and then use reana-resources-k8 build-manifests myconfig.yaml to fit your computing cloud. (and link to CLI documentation)

Signed-off-by: Diego Rodriguez <[email protected]>

diegodelemos added this to the Initial-Release milestone Feb 10, 2017

diegodelemos added the Status: in review label Feb 10, 2017

diegodelemos force-pushed the 17-automatic-generate-manifests branch 2 times, most recently from 3fcfe4d to 64b5ae0 Compare February 13, 2017 12:53

tiborsimko mentioned this pull request Feb 13, 2017

cephfs: make ceph details (monitors, mounts, user, secret-name) confugurable reanahub/reana-job-controller#29

Open

diegodelemos force-pushed the 17-automatic-generate-manifests branch from 64b5ae0 to 15c822e Compare February 13, 2017 16:02

diegodelemos changed the title ~~WIP config: automatic manifest generation~~ config: automatic manifest generation Feb 13, 2017

diegodelemos requested a review from tiborsimko February 13, 2017 18:12

tiborsimko requested changes Feb 15, 2017

View reviewed changes

tiborsimko added the Need: work label Feb 15, 2017

tiborsimko assigned diegodelemos Feb 15, 2017

diegodelemos force-pushed the 17-automatic-generate-manifests branch from 1d16fac to d48d8db Compare February 15, 2017 14:31

diegodelemos assigned tiborsimko and unassigned diegodelemos Feb 15, 2017

Diego Rodriguez added 10 commits February 15, 2017 17:40

config: automatic build configuration files

7c35dd0

* (addresses reanahub/reana#13) Signed-off-by: Diego Rodriguez <[email protected]>

config: organize configuration as directory tree

c5f8466

Signed-off-by: Diego Rodriguez <[email protected]>

config: split multiple yaml files

ffcb32a

* Since config for namespaces, quotas and workers are stored in a single jinja template, the output was a single file with many yaml files inside. This commit generates a file for each yaml. Signed-off-by: Diego Rodriguez <[email protected]>

config: add script and templates as resources

c89b155

* In order to make `build-manifests.sh` accesible when installing the package, both the script and the `templates` directory have been added to it. Signed-off-by: Diego Rodriguez <[email protected]>

config: amend output config directory

1d9dde6

Signed-off-by: Diego Rodriguez <[email protected]>

docs: add getting started and configuration

43a6dab

Signed-off-by: Diego Rodriguez <[email protected]>

config: amend build-configuration script

67012e9

Signed-off-by: Diego Rodriguez <[email protected]>

config: use sh in build script

002672a

Signed-off-by: Diego Rodriguez <[email protected]>

kwalitee: clean build script

77c316a

Signed-off-by: Diego Rodriguez <[email protected]>

docs: amend build manifests command

221a3b9

Signed-off-by: Diego Rodriguez <[email protected]>

diegodelemos force-pushed the 17-automatic-generate-manifests branch from 38639a3 to 221a3b9 Compare February 15, 2017 16:41

tiborsimko requested changes Feb 17, 2017

View reviewed changes

Diego Rodriguez added 4 commits February 17, 2017 13:32

kwalitee: add license header

a7302c0

Signed-off-by: Diego Rodriguez <[email protected]>

docs: improve getting started and configuration

721df71

Signed-off-by: Diego Rodriguez <[email protected]>

tests: add reana-resources-k8s script to travis

0412a39

Signed-off-by: Diego Rodriguez <[email protected]>

installation: add jinja2-cli and PyYAML dependency

25b5509

Signed-off-by: Diego Rodriguez <[email protected]>

diegodelemos force-pushed the 17-automatic-generate-manifests branch from 2834df2 to 25b5509 Compare February 17, 2017 15:10

config: generate config inside current directory

125e02d

Signed-off-by: Diego Rodriguez <[email protected]>

tiborsimko approved these changes Feb 21, 2017

View reviewed changes

tiborsimko merged commit 125e02d into reanahub:master Feb 21, 2017

tiborsimko removed the Status: in review label Feb 21, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

config: automatic manifest generation #18

config: automatic manifest generation #18

diegodelemos commented Feb 10, 2017

tiborsimko Feb 15, 2017

diegodelemos Feb 15, 2017

tiborsimko Feb 15, 2017

diegodelemos Feb 15, 2017

tiborsimko Feb 15, 2017

diegodelemos Feb 15, 2017

tiborsimko Feb 15, 2017

tiborsimko Feb 17, 2017

tiborsimko Feb 17, 2017

tiborsimko Feb 17, 2017

tiborsimko Feb 17, 2017

tiborsimko Feb 17, 2017

tiborsimko Feb 17, 2017

tiborsimko Feb 17, 2017

tiborsimko Feb 17, 2017


		Once we have done it, we should be able to see the directory `configuration-manifests` with the following structure:

		.. code-block:: sh

		=============


		Configuration for the REANA system can be made through the file `config.yaml` under `templates`:

config: automatic manifest generation #18

config: automatic manifest generation #18

Conversation

diegodelemos commented Feb 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment