Skip to content

Developer Guide

Jump to bottom
Eric Honer edited this page Jan 19, 2023 · 28 revisions

Table of Contents

Development environment setup

Prerequisites

To develop Brooklin, you need to have the following dependencies installed:

  1. Gradle: https://docs.gradle.org/4.10.3/userguide/installation.html
  2. JDK 8+: https://openjdk.java.net/install/
  3. Git: https://git-scm.com/downloads
  4. Python 2.3+: https://www.python.org/downloads/

IDE

If you do not have a Java IDE set up, you can download and install the latest version of IntelliJ IDEA Community Edition for your platform at https://www.jetbrains.com/idea/download/.

Building and testing Brooklin

  • To clone the Brooklin repository, run the following command from termina/command line: 
    git clone https://github.com/linkedin/brooklin.git

Note: If you are on Windows, please replace gradlew with gradlew.bat

  • To build Brooklin and run all tests:

    ./gradlew clean build

  • To run unit tests:

    ./gradlew test

  • To release the Brooklin binaries into local Maven repository:

    ./gradlew publishToMavenLocal

  • To build the Brooklin tarball:

    ./gradlew releaseTarGz

    You can find the generated tarball at datastream-tools/build/distributions/brooklin.tgz

Attaching a debugger to Brooklin

You can always debug any of Brooklin's end-to-end tests (those extending BaseKafkaZkTest) in an IDE. However, if you ever need to attach a debugger to a deployed instance of Brooklin, here is how to do so:

  1. If you made any code changes to Brooklin, you can run the following command to generate a new Brooklin tarball:

    ./gradlew releaseTarGz

  2. Copy the tarball from datastream-tools/build/distributions/brooklin*.tgz to a convenient location then untar it.

    # You may have a different version of Brooklin
tar -xzf brooklin-1.0.0.tgz
cd brooklin-1.0.0

  3. Modify the script in bin/brooklin-server-start.sh to append the following JVM debugging parameters to the EXTRA_ARGS variable.

    # You can set suspend=n if you don't want Brooklin to pause on startup until a debugger is attached
-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

  4. When you are ready to run Brooklin, run bin/brooklin-server-start.sh.

  5. Configure your IDE to attach a debugger to a running Java process. Make sure to specify the hostname Brooklin is running on (localhost if running locally) as well as the port you used in step 3.

    Here is a SO answer explaining how to do this for IntelliJ IDEA: https://stackoverflow.com/a/30793101

Publishing a new version of Brooklin

Decide the next version

Brooklin adheres to Semantic Versioning, i.e.

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,

MINOR version when you add functionality in a backwards-compatible manner, and

PATCH version when you make backwards-compatible bug fixes.

Create a new release on GitHub

  1. Decide which commit to publish as the new version (in the master branch)

  2. On your local machine, open at terminal window at the root directory of your Brooklin clone. Make sure your HEAD points at the commit you chose in the previous step.

  3. Edit gradle/maven.gradle by setting allprojects.version to the new version.

  4. Run the following command to produce the new tarball

./gradlew releaseTarGz

  1. Go to https://github.com/linkedin/brooklin/releases/ and click "Draft a new release".

  2. Type the new version in both the "Tag version" and "Release title" text boxes.

  3. Set the "Target" select box to the commit you chose in step 1.

  4. Upload the new release tarball located under the ./datastream-tools/build/distributions/ directory, to the new GitHub release.

  5. Publish the new release.

Update maven.gradle

Send a PR with an update to allprojects.version in gradle/maven.gradle setting it to a future SNAPSHOT version. For instance, if you just released version 1.0.0, set allprojects.version to 1.1.0-SNAPSHOT.

Add a new entry to Change Log

Add a new entry to CHANGELOG.md documenting the new version, release date, and all noteworthy changes.

Publish artifacts to JFrog Artifactory

  1. Sign in to Artifactory at at https://linkedin.jfrog.io/ui/login/, or use Okta application JFrog.

  2. After signing in successfully, retrieve an API key that you can use to publish Brooklin artifacts:

    • Click your account name on the upper right corner
    • Click Edit Profile
    • Click API Key on the left navigation menu

  3. Run the following commands in Brooklin's root directory to specify your publishing credentials:

export ARTIFACTORY_USER=<your-bintray-username>
export ARTIFACTORY_API_KEY=<your-API-key>
  1. Publish the new version by running:
./gradlew artifactoryPublish
  1. Go to Brooklin's page on Artifactory, check out the uploaded files, and publish the new version (look for an inline notification message prompting you to publish or discard the new files).

Update the version shield

Update the version shield on the README page. Ideally, we should have been able to use Shields.io's version badge to reflect the new version automatically, but it is flaky.

Contributing to Brooklin

  • Contributions are accepted in the form of Pull Requests.

  • Before submitting a Pull Request, please, make sure your changes are building and passing all tests with the latest code in the master branch of the Brooklin upstream repo. Please, follow the instructions on this page to learn how to sync a fork with an upstream repository.

  • Please, take a moment to read our Contribution Guidelines before submitting patches or opening issues.

Clone this wiki locally