investigate SparkContext.addFile()

[lbergelson]

JavaSparkContext.addFile seems to be a runtime version of --files. According to this documentation it is used in concert with SparkFIles.get() on the node to lookup the downloaded file location.

It may be possible to persist these files (and those passed with --files) by setting 'spark.yarn.preserve.staging.files true`.

We should investigate incorporating calls to these methods as a way of sharing small-medium sized inputs.

[tomwhite]

We're doing this for the reference and known sites, and it works well: #5127