dependency-manager.md

Documentation Home | Metalus Utils Home

Dependency Manager

The dependency manager is a tool used to resolve dependent jars for step libraries. Each step library is allowed to include a file named dependencies.json in the root classpath if there are additional dependent jars (including step libraries) that are needed. The basic structure appears as follows:

{
  "<resolverType>": {
    "libraries": []
  }
}

Here is a working example:

{
  "maven": {
    "libraries": [
      {
        "groupId": "org.mongodb.spark",
        "artifactId": "mongo-spark-connector_2.11",
        "version": "2.4.2"
      },
      {
        "groupId": "org.mongodb",
        "artifactId": "mongo-java-driver",
        "version": "3.12.2",
        "scope": "extraction"
      }
    ]
  }
}

The resolver is set to maven and each library entry provides a structure defined by the Maven dependency resolver. Once the jars have been processed, a classpath will be printed to the console. All jars will be copied to the output-path.

Running

Metalus Utils must be installed before running the script.

The script parameters are:

• --jar-files - A comma separated list of jar files. This should be the full path. Paths starting with http will attempt to download the jars. The authorization parameters will be used if provided unless the no-auth-download is provided.
• --no-auth-download - An optional parameter which when set to true will not use the authorization parameters when downloading jars.
• --output-path - A path to write the jar files.
• --path-prefix - An optional path prefix to add to each jar in the classpath
• --jar-separator - An optional separator character to use when building the classpath
• --include-scopes - An optional parameter that defines additional scopes to include on the classpath in addition to runtime.
• --allow-self-signed-certs - Boolean flag enabling self signed certificates. Default value is false.

Authorization: When pushing metadata to an API or downloading jars, authorization is not used unless the authorization parameters are provided.

Example commands:

Generate the classpath:

bin/dependency-resolver.sh --jar-files /tmp/steps.jar,/tmp/common-steps.jar --output-path /tmp

Generate the classpath with custom separator:

bin/dependency-resolver.sh --jar-files /tmp/steps.jar,/tmp/common-steps.jar --output-path /tmp --jar-separator :

Generate the classpath overriding the maven repo:

bin/dependency-resolver.sh --jar-files /tmp/steps.jar,/tmp/common-steps.jar --output-path /tmp --maven.repo http://localhost/maven2

Generate the classpath overriding the maven repo to use the local:

bin/dependency-resolver.sh --jar-files /tmp/steps.jar,/tmp/common-steps.jar --output-path /tmp --maven.repo file://home/user/.m2/repository

Example Output with a path-prefix of hdfs://acxiom/jars/udl:

hdfs://acxiom/jars/udl/metalus-mongo_2.11-spark_2.4-1.6.0-SNAPSHOT.jar:hdfs://acxiom/jars/udl/mongo-spark-connector_2.11-2.4.2.jar:hdfs://acxiom/jars/udl/mongo-java-driver-3.12.7.jar

Maven Dependency Resolver

Metalus ships with a custom Maven dependency resolver. By default it will resolve dependencies using the central repo. Developers may choose to override the repo within the dependencies.json file by adding the repo attribute like this:

{
  "maven": {
    "repo": "http://localhost/maven2",
    "libraries": [
      {
        "groupId": "org.mongodb.spark",
        "artifactId": "mongo-spark-connector_2.11",
        "version": "2.4.2"
      },
      {
        "groupId": "org.mongodb",
        "artifactId": "mongo-java-driver",
        "version": "3.12.7",
        "scope": "extraction"
      }
    ]
  }
}

Note: the scope attribute is used to differentiate between artifacts that should be included on the classpath. The default is runtime. Marking an artifact with a scope of extraction indicates that the dependency is only used during metadata extraction. Custom scope values may be used and a single dependency may have more than one value separated by commas. If a custom scope is provided, the value of runtime must be included if that dependency should also be available at runtime.

Additionally, the repo may be overridden when calling the script by passing the following command line parameter:

--maven.repo http://localhost/maven2

The local maven cache may also be used by passing in the command line parameter:

--maven.repo file://home/user/.m2/repository

Multiple repos may be provided by separating the paths with a comma:

--maven.repo http://localhost/maven2,file://home/user/.m2/repository