[SPARK-4186] add binaryFiles and binaryRecords in Python

[davies]

add binaryFiles() and binaryRecords() in Python

binaryFiles(self, path, minPartitions=None):
    :: Developer API ::

    Read a directory of binary files from HDFS, a local file system
    (available on all nodes), or any Hadoop-supported file system URI
    as a byte array. Each file is read as a single record and returned
    in a key-value pair, where the key is the path of each file, the
    value is the content of each file.

    Note: Small files are preferred, large file is also allowable, but
    may cause bad performance.

binaryRecords(self, path, recordLength):
    Load data from a flat binary file, assuming each record is a set of numbers
    with the specified numerical format (see ByteBuffer), and the number of
    bytes per record is constant.

    :param path: Directory to the input data files
    :param recordLength: The length at which to split the records

[davies]

cc @mateiz

[SparkQA]

Test build #22823 has started for PR 3078 at commit bb22442.

• This patch merges cleanly.

[SparkQA]

Test build #22823 has finished for PR 3078 at commit bb22442.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22823/
Test FAILed.

[SparkQA]

Test build #509 has started for PR 3078 at commit bb22442.

• This patch merges cleanly.

[SparkQA]

Test build #509 has finished for PR 3078 at commit bb22442.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #22842 has started for PR 3078 at commit 5ceaa8a.

• This patch merges cleanly.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22841/
Test FAILed.

[SparkQA]

Test build #510 has started for PR 3078 at commit 5ceaa8a.

• This patch merges cleanly.

[SparkQA]

Test build #22842 has finished for PR 3078 at commit 5ceaa8a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22842/
Test PASSed.

[SparkQA]

Test build #510 has finished for PR 3078 at commit 5ceaa8a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #22868 has started for PR 3078 at commit 24e84b6.

• This patch merges cleanly.

[SparkQA]

Test build #22868 timed out for PR 3078 at commit 24e84b6 after a configured wait of 120m.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22868/
Test FAILed.

[SparkQA]

Test build #511 has started for PR 3078 at commit 24e84b6.

• This patch merges cleanly.

[SparkQA]

Test build #511 has finished for PR 3078 at commit 24e84b6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class ExecutorActor(executorId: String) extends Actor with ActorLogReceive with Logging
  • case class GetActorSystemHostPortForExecutor(executorId: String) extends ToBlockManagerMaster
  • case class Params(
  • class VectorUDT(UserDefinedType):
  • class NullType(PrimitiveType):
  • class UserDefinedType(DataType):
  • // in some cases, such as when a class is enclosed in an object (in which case
  • case class ScalaUdfBuilder[T: TypeTag](f: AnyRef)
  • abstract class UserDefinedType[UserType] extends DataType with Serializable
  • public abstract class UserDefinedType<UserType> extends DataType implements Serializable

[mateiz]

This shouldn't be DeveloperAPI I think, it should just be experimental (and same with the binaryRecords method). Maybe the labels are wrong in the Java and Scala API; if so do you mind changing them?

[mateiz]

Looks good, I just noticed one weird thing in the docs (probably an issue in the Java/Scala docs but we might as well fix those too).

[SparkQA]

Test build #22961 has started for PR 3078 at commit cd0bdbd.

• This patch merges cleanly.

[SparkQA]

Test build #22961 has finished for PR 3078 at commit cd0bdbd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22961/
Test PASSed.

[mateiz]

Looks good, thanks!