[SPARK-13367] [Streaming] Refactor KinesusUtils to specify more KCL options

[addisonj]

This patch refactors the KinesisUtils and adds new APIs such that it is
easy to pass more configuration options into the KCL, such as using a
different DynamoDBClient with a different endpoint.

The core of this refactoring/change is to introduce the KinesisConfig
class which is intended to encapsulate all configuration concerns.
Currently, it doesn't do much more than allow for setting the DynamoDB
endpoint, but it is a good place for where other options could be
contained without changing underlying APIs. It could also be inherited
to override behavior for more options without requiring any API changes
(such as overriding buildKCLConfig ti allow for more configuration
changes)

This also introduce a new external API for creating a KinesisRDD, which
takes a KinesisConfig object which may be a more manageable API going
forward.

Docs are still lacking for the new class as well as some basic unit
specs

[AmplabJenkins]

Can one of the admins verify this patch?

[rxin]

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket.

[anuraag19]

Hello addisonj,
Looks like this never made it through Spark - KenisisUtils, is that the right understanding?
Any work around you did to accomplish this.

Thanks

[addisonj]

We didn't end up using it, but I just built my own jar for this subproject and included it.

I tried getting someone to review it but had no luck, if you want to get it through the process lemme know! I think it should probably be about in the same state

[anuraag19]

Hi,

Thanks for getting back. Our intention is to use Kinesalite and Dynalite in test environments and since we plan to use Spark we want to make it happen through Spark. It is similar to what you had in your initial JIRA ticket. I am sure that a more configurable Spark-Kinesis API will be useful in other use cases also.

So yes, I am interested, we can go through the process. Please let me know what needs to be done since it will be doing it for the first time :)