Skip to content

Building TAP from Sources

Jump to bottom
Iman Saleh edited this page Dec 22, 2015 · 11 revisions

Wiki Home ▸ Building TAP from Sources

Before you start, make sure that you have the required [ tools ] (Development Environment Setup) installed.

Building Trusted Analytics Platform from sources:

  1. Download versions.yml file from latest release version of platform-appstack. versions.yml file contains a list of names of projects to be built and their release versions accordingly.

  2. Clone the repositories of projects listed in versions.yml file.

_Notes:_
* Names listed in `versions.yml` file represent names under which applications will pushed to Cloud Foundry and may differ from names of their corresponding repositories. The list of differences is presented in the table below.
* Multiple applications may derive from the same sources/repositories (worlds-simplest-service-broker, application-broker). For such applications clone the corresponding repository only once. 


| Name in `versions.yml` |        Repository name          |
|------------------------|:-------------------------------:|
| hdfs-downloader        | downloader                      |
| metadataparser         | metadata-parser                 |
| das                    | data-acquisition                |
| hdfs-uploader          | uploader                        |
| smtp-broker            | worlds-simplest-service-broker  |
| cdh-broker             | worlds-simplest-service-broker  |
| kafka-broker           | worlds-simplest-service-broker  |
| zookeeper-wssb-broker  | worlds-simplest-service-broker  |

* The following projects are written in Go language and should be cloned using command:
  ```
  go get github.com/trustedanalytics/<repository_name>
  ```
    * worlds-simplest-service-broker
    * gateway
    * application-broker
    * app-launcher-helper

  1. Create a new directory where you will keep the sources of Java projects` dependencies:

    mkdir ./trustedanalytics_dependencies

  2. Navigate to the trustedanalytics_dependencies directory and clone trustedanalytics-cloud-parent and spring-project-template:

    git clone [email protected]:trustedanalytics/trustedanalytics-cloud-parent.git
git clone [email protected]:trustedanalytics/spring-project-template.git

Perform the steps described below for each application listed in versions.yml. Note that commits pointing to the version of each application are tagged by the name of the latest release, eg. 'TAP-2015-DEC-1'.

From this point, steps to be taken depend on the language in which each application was written. In case of difficulties, please refer to the particular project's Readme file.

Java projects:

  • Navigate to the project's directory.

  • Checkout to the commit tagged as release:

    git checkout `<release_name>`

  • Open pom.xml file

  • Check version of parent pom (it can be found in project/parent/version field)

  • If you haven't installed this particular version of parent pom:

    • Navigate to the cloned parent pom directory (trustedanalytics_dependencies/trustedanalytics-cloud-parentor trustedanalytics_dependencies/spring-project-template)

    • Find the commit pointing to the desired version of parent pom. It can be identified in the way described in "Useful tips" section at the end of this page ( git grep '<version>version_number_from_pom.xml</version>' $(git rev-list --all) ).

    • Checkout to the selected commit:

      git checkout <commit_sha>

    • Install the desired version of parent pom in local artifacts repository:

      mvn install

  • Navigate back to the current Java project's directory and open pom.xml file

  • Find dependencies from org.trustedanalytics group. They can be identified by groupId field in pom.xml in project/dependencies/dependency/ section. For each such dependency:

  • Navigate to the trustedanalytics_dependencies directory

  • If you haven't cloned the dependency before, clone it:

    git clone [email protected]:trustedanalytics/<dependency_name>

  • Find the commit which points to the desired version of the dependency (version field in project/dependencies/dependency/ section). It can be identified in the way described in "Useful tips" section at the end of this page ( git grep '<version>version_number_from_pom.xml</version>' $(git rev-list --all) ).

  • Checkout to the selected commit:

    git checkout <commit_sha>

  • Install the dependency in local artifacts repository:

    mvn install

  • Now all the required dependencies from org.trustedanalytics group should be available in your local artifacts repository (~/.m2/repositories by default). Navigate to the project`s directory and run tests:

    mvn verify

  • Build the application:

    mvn package

  • Prepare zip artifact:

    zip -r <artifact_name>.zip . -i target/<artifact_name>-<version>.jar manifest.yml

Python projects (Data Catalog)

  • Navigate to the project's directory.

  • Checkout to the commit tagged as release:

    git checkout `<release_name>`

  • Activate virtualenv:

    source .tox/py27/bin/activate

  • Run tests:

    tox -r

  • Download Python dependencies:

    mkdir -p vendor/
pip install --download vendor/ -r requirements-normal.txt
pip install --download vendor/ -r requirements-native.txt --no-use-wheel

  • Deactivate virtualenv:

    deactivate

  • Prepare zip artifact:

    zip -r <artifact_name>.zip vendor/ data_catalog/ manifest.yml requirements.txt runtime.txt

JavaScript/Node.js projects (Console)

  • Navigate to the project's directory.

  • Checkout to the commit tagged as release:

    git checkout `<release_name>`

  • Install Node dependencies:

    npm install

  • If you don't have Firefox installed, install it:

    sudo apt-get install firefox

  • Compile UI and run tests:

    ./node_modules/gulp/bin/gulp.js

  • Clean up unnecessary dependencies:

    rm -rf node_modules bower_components .cfignore

  • Install production dependencies:

    npm install --production

  • Prepare zip artifact:

    zip -r <artifact_name>.zip .

Go projects

  • Navigate to the project's directory.

  • Checkout to the commit tagged as release:

    git checkout `<release_name>`

  • Build project:

    godep go build

  • Run tests (optional) - check particular project's Readme for detailed instructions.

  • Prepare zip artifact:

find . -type f -name '*_test.go' -delete
find . -type f -name '*-mock.go' -delete
zip -r -q <artifact_name>.zip *

Once you have all zipped artifacts, move all of them to one directory and continue with Platform application layer deployment.

Useful tips

Finding a commit which points to a specific application version

For different project languages project version can be found in different files, as described by the table below. The table shows the relation between programming language and: name of file in which application version will be found, expression that we're searching for in the file structure. As an example version 0.4.2 will be used.

Language File name containing app version Expression
Java pom.xml <version>0.4.2</version>
Python manifest.yml VERSION: "0.4.2"
JavaScript/Node.js package.json "version": "0.4.2"
Go manifest.yml VERSION: "0.4.2"

Execute:

git grep '`Expression`' $(git rev-list --all)

For example:

git grep '<version>0.4.2</version>' $(git rev-list --all)

This will print commit SHA that points to the desired version of the application.

If there is more than one commit found:

  • Check which of the commits points to the desired, release version of the application.
  • If there is more than one commit pointing to the same, release version of the application (this should happen rarely) check which one of them passes integration and unit tests.
  • If more than one of them passes the tests - choose the eldest one.

Home | FAQs

Clone this wiki locally