[SPARK-8835][Streaming] Provide pluggable Congestion Strategies for Receiver-based Streams

[huitseeker]

The new (destructive) strategies avoid leaving elements in the TCP buffer, leading to breakages at the sender level when this buffer finally drops them.

[huitseeker]

/cc @tdas and @JoshRosen (the later to issue an 'ok to test' here)

[huitseeker]

Pinging @srowen for an 'ok to test'.
Also, if you're asking yourself why some tests are in "ignore" : the answer is in the commit messages.

[tdas]

ok to test.

[tdas]

I think this requires a good design doc for me to make sense of the overall idea, and discuss the design. Its hard to discuss the design in the PR.

[SparkQA]

Test build #44516 has finished for PR 9200 at commit 2e8e5de.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class BernoulliSampler[T: ClassTag](fraction: Double, rng: Random = RandomSampler.newDefaultRNG) extends RandomSampler[T, T]\n

[SparkQA]

Test build #44519 has finished for PR 9200 at commit bc06de2.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class BernoulliSampler[T: ClassTag](fraction: Double,\n

[SparkQA]

Test build #44520 has finished for PR 9200 at commit b2dd6b8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class BernoulliSampler[T: ClassTag](fraction: Double,\n

[rxin]

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one.