-
Notifications
You must be signed in to change notification settings - Fork 110
Plugins
Drake supports plugins as of version 0.1.4.
Plugins allow the extension of Drake's step protocols and filesystem implementations.
A Drake plugin is a Clojure project that provides a custom implementation of a step Protocol, and/or a custom implementation of a Drake FileSystem.
The step Protocol implementation defines how a custom step protocol should run when called from a Drake workflow.
The FileSystem implementation defines how Drake should interact with an external FileSystem.
If you've created a plugin for Drake, please add it here!
- drake-hive A Hive filesystem plugin for Drake.
- drake-echostep Illustrative example of a simple Drake plugin
- drake-honeyql Supports HoneyQL statements to get data from Factual
To use a plugin that someone else has published, specify it in your plugins.edn
file, located in the same directory as your Drakefile. (Create it if you don't
already have one.)
The plugins.edn file contains a hash-map of plugin configuration. The :plugins
entry specifies an array of Maven repository coordinates for published Drake plugins.
Example plugins.edn file contents:
{:plugins [[dirtyvagabond/drake-echostep "0.1.0"]
[dirtyvagabond/drake-honeyql "0.0.3"]]}By default, Drake will look for the specified plugins in Maven Central and clojars.org.
If you wish to use plugins from repositories other than the deafults, you can
specify additional repos in your plugins.edn file. E.g.:
{:repositories {"jboss" "https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads"
"mycorp" "http://mycorp.com/nexus/content/groups/public"}
:plugins [[mycorp/myprotocol "1.2.3"]
[dirtyvagabond/drake-honeyql "0.0.3"]]}These repos will be merged in with the defaults before trying to resolve dependencies.
You can optionally specify the plugin configuration file you want to use for a
particular Drake workflow using the --plugins option. For example:
bash drake --plugins /my/plugin/conf/my-favorite-plugins.edn
To create a plugin you create and publish a Clojure project that provides an implementation for a step Protocol and/or an implementation for a FileSystem.
The step Protocol implementation defines how a custom step protocol should run when called from a Drake workflow.
The FileSystem implementation defines how Drake should interact with an external FileSystem.
To define a custom step Protocol:
- Define a namespace called
drake.protocol.[PROTOCOL_NAME], e.g.drake.protocol.myprotocol - In that namespace define a zero argument function called [PROTOCOL_NAME], e.g.
myprotocol. This is your protocol reification function. - Have your protocol reificiation function return a reified Protocol -- the implementation you want for your Drake step Protocol.
The implementation returned by your reification function should provide a reified Protocol, as defined in the drake-interface project.
cmds-required?: Returns true if your step Protocol requires commands in
the step. Drake will error out if it encounters a workflow step that calls a
protocol where commands are required, but not provided in the step body.
Most step protocol require commands, but yours doesn't have to if you don't want
it to.
run: Takes a hash-map representing the workflow step and is expected to implement the step. See the Step Data wiki page for more details about the contents of the step hash-map.
Drake will use your Protocol implementation when it encounters workflow steps that refer to it. E.g.:
out.txt <- in.txt [myprotocol]
some commmands
go here
When Drake sees the above step, it will call the run function in your step
Protocol implementation, passing it a hash-map of all the step data, including
the commands in the body. Your run function should then do whatever cool thing
it is you want your custom step Protocol to do!
For an illustrative example of implementing a custom step Protocol, see the echostep protocol.
To define a custom FileSystem:
- Define a namespace called
drake.fs.[FILESYSTEM_NAME], e.g.drake.fs.myfs - In that namespace define a zero argument function called [FILESYSTEM_NAME], e.g.
myfs. This is your FileSystem reification function. - Have your FileSystem reificiation function return a reified FileSystem -- the implementation you want for your FileSystem.
The implementation returned by your reification function should provide a reified FileSystem, as defined in the drake-interface project.
Drake will use your FileSystem implementation when it encounters input or output names prefixed with [FILESYSTEM_NAME]. E.g.:
myfs:/out.txt <- in.txt [shell]
grep something $INPUT
In the above example, Drake will refer to your custom FileSystem implementation
when resolving myfs:/out.txt.
For an illustrative example of implementing a custom FileSystem, see the myfs example in the echostep project.
To share your plugin with the world, publish it to a public Maven repository. A
common and easy approach is to use lein push. But you can also publish your
plugin to a private internal repo if you want to share it with workmates but not
the whole wide world. Or you could use something like lein install to publish
the plugin only for your local development environment.
For an illustrative example of a simple plugin, take a look at drake-echostep. The echostep step protocol simply prints out all the step data to an output file, so it's a handy way to see a real world example of step data.
The drake-honeyql plugin project provides a real-world example of integrating an old Java library with Drake in the form of a custom step protocol.