Dynamically creating tasks

This page describes how to avoid a lot of duplication across many similar tasks by dynamically creating tasks. This isn't specific to ml-gradle - this is just showing a Gradle technique. When using ml-gradle, it's often helpful for when you have many instances of MlcpTask that are very similar.

Let's say we have a single MlcpTask that looks like this:

task ingestApples(type: com.marklogic.gradle.task.MlcpTask) {
	classpath = configurations.mlcp
	command = "IMPORT"
	database = "Documents"
	input_file_path = "data/apples"
	input_file_type = "delimited_text"
	delimiter = ","
	generate_uri = "true"
	output_collections = "Apples"
	output_uri_prefix = "/Apples"
	output_permissions = "rest-reader,read,rest-writer,update"
	output_uri_suffix = ".json"
	document_type = "json"
	thread_count = 8
	transform_module = "/some/transform.sjs"
}

Then we need to create similar tasks for ingesting delimited files containing data about bananas, oranges, plums, and many other fruits. But most of our task configuration above is not specific to apples. Instead of duplicating the above task for every fruit - which would quickly bloat our build.gradle file - it'd be nice if we could generate a task based on each file that we need to ingest. Ideally, adding support for another fruit type would be as simple as adding a new property to gradle.properties.

So let's add the following properties to gradle.properties - we'll make up directories for each one that contain one or more files of fruit data:

ingestApples=data/apples
ingestBananas=data/bananas
ingestOranages=data/oranges

And so on. Then, we'll use the following script to dynamically generate a task for every Gradle property that starts with "ingest":

project.properties.each { key, value ->
  if (key.startsWith("ingest") && value) {
    def recordType = key.substring(6)
    task "ingest${recordType}"(type: com.marklogic.gradle.task.MlcpTask) {
      group = "Fruit Ingest"
      classpath = configurations.mlcp
      command = "IMPORT"
      database = "Documents"
      input_file_path = value
      input_file_type = "delimited_text"
      delimiter = ","
      generate_uri = "true"
      output_collections = recordType
      output_uri_prefix = "/" + recordType
      output_permissions = "rest-reader,read,rest-writer,update"
      output_uri_suffix = ".json"
      document_type = "json"
      thread_count = 8
      transform_module = "/some/transform.sjs"
    }
  }
}

We now have tasks named "ingestApples", "ingestBananas", etc available to use. You can of course customize the above script to do whatever you want it to do, this is just one example.

Invoking all of the tasks at once

Note that the tasks also have a Gradle group defined. This makes it easy to define a task that invokes all of the ingest tasks at once:

task ingestAllFruits {
  dependsOn {
    tasks.findAll { task -> "Fruit Ingest".equals(task.group) }
  }
}

Using a subclass of MlcpTask

You may run into a situation where you have additional ingest tasks that don't closely follow the pattern above, but they still share some of the same configuration as the dynamically generated tasks. To avoid any duplication, you can make a custom subclass of MlcpTask that defines all the common stuff - just add this to build.gradle:

class IngestTask extends com.marklogic.gradle.task.MlcpTask {
  IngestTask() {
    command = "IMPORT"
    classpath = project.configurations.mlcp
    input_file_type = "delimited_text"
    generate_uri = "true"
    output_permissions = "rest-reader,read,rest-writer,update"
    output_uri_suffix = ".json"
    document_type = "json"
    delimiter = ","
    thread_count = 8
  }
}

Then just change the script for dynamically generating tasks to use IngestTask instead of com.marklogic.gradle.task.MlcpTask, and use IngestTask for any other tasks you have that use mlcp but don't follow the pattern defined by the script for dynamically generating tasks.